Note: The job is a remote job and is open to candidates in USA. Valiant Harbor International is seeking a Software Development Engineer III – Infrastructure to support the Director’s Office at the Advanced Research Projects Agency for Health (ARPA-H). The role involves managing backend infrastructure, contributing to the development of agentic AI, and ensuring production quality and observability within the GRACE project.
Responsibilities
- Manage end-to-end backend infrastructure for GRACE on Microsoft Azure:
- Azure Functions, Azure API Management, Azure Container Apps, and Azure OpenAI Service
- Manage storage, retrieval pipelines, vector databases, and document indexing that power GRACE's internal knowledge search
- Authentication and identity integration, including ARPA-H Entra ID and application-level access control
- Implement and maintain infrastructure as code for all environments
- Own CI/CD pipelines, deployment automation, and release processes including canary and gradual rollouts
- Be responsible for production system basics (e.g., monitoring, alerting, logging, distributed tracing, SLOs, and incident response runbooks)
- Manage secrets, API keys, and credential rotation across all integrations with external providers
- Monitor for cost-related efficiencies across all LLM providers; track spending, set budgets, build guardrails, and optimize for cost-per-query without sacrificing quality
- Manage the backend implementation of MCP, including MCP server hosting, tool registration, versioning, and lifecycle management on Azure
- Implement and evolve A2A communication patterns to enabling GRACE agents interoperability with internal/external systems
- Design and maintain LLM orchestration, routing, and multi-model switching infrastructure across OpenAI GPT, Anthropic Claude, and Google Gemini families
- Build and operate RAG pipelines; document ingestion, chunking, embedding, and semantic search
- Implement robust fallback, retry, circuit-breaker, and graceful degradation patterns for all AI service dependencies
- Manage tool-calling infrastructure; to include registration, execution, error handling, and observability for all GRACE tools
- Build and maintain end-to-end observability for agentic workflows: latency, throughput, error rates, token usage, and LLM quality metrics
- Implement LLM evaluation pipelines including safety checks, regression monitoring, and grounding assessment
- Define and enforce system-level SLOs for availability, response time, and tool call reliability
- Manage alerting and on-call runbooks
- Establish and improve coding standards, design review processes, and testing practices
- Communicate technical decisions in writing and in conversation to both engineers and non-engineers
- Mentor and guide other engineers
- Think inventively and consider other perspectives; work backward from the user to understand problems before proposing solutions
- Ensure strict privacy, security, and compliance in all systems, integrations, and data handling
Skills
- Bachelor's or Master's in Computer Science, Software Engineering, or related field, or equivalent practical experience
- 7+ years of professional software engineering experience building and operating production systems
- Proven experience in high-velocity environments shipping complex products end-to-end
- Strong proficiency in backed languages (to include Python); familiarity with modern backend frameworks and async patterns
- Solid understanding of distributed systems, APIs, data pipelines, and software design patterns
- Hands-on experience on Microsoft Azure: Azure Functions, API Management, Container Apps, and Azure OpenAI Service
- Experience with containerization, CI/CD, and infrastructure as code
- Strong understanding of authentication and identity systems (OAuth2, OIDC, Azure Entra ID or equivalent)
- Demonstrated experience/ability with production systems (having been on-call, debugged incidents, etc.)
- Excellent communication and team building skills; focused on making others around them better
- Hands-on experience building and operating MCP servers in production, including tool registration, versioning, and hosting on Azure Functions or equivalent serverless infrastructure
- Experience implementing A2A communication patterns and multi-agent orchestration frameworks
- Significant experience building on top of LLMs in production (tool-calling, RAG, multi-step reasoning, multi-model routing, and context window management)
- Ability to demonstrate considerations for cost-per-query, context budgets, and prompt efficiency as first-class engineering concerns
- Experience managing multi-provider LLM integrations, including rate limits, fallback routing, and API versioning
- Experience in security-conscious engineering within regulated or government environments
- Previous track record in startup or early-stage environments (0-to-1 product building, comfort with ambiguity, and a high sense of urgency)
- Experience in big tech building customer-facing platforms or developer infrastructure at scale
- Familiarity with vector databases, embedding pipelines, and semantic search infrastructure
Benefits
- This is a full-time, remote position.
- The candidate must be able to travel within the U.S.
Company Overview