FieldGuide
Senior AI Engineer, Quality
$200K–$250K
San Francisco, CA or Remote (USA) · Hybrid · Senior · Full-time · Posted Apr 20, 2026
- Experience
- Multiple years of production software experience; no specific number of years required.
About the job
Build and own evaluation infrastructure, observability, and feedback systems that measure and improve the reliability of Fieldguide’s AI agents in enterprise audit workflows.
What you'll do
- Build a unified evaluation platform, automated model-testing pipelines, and production feedback loops.
- Develop observability, tracing, comparison frameworks, guardrails, and monitoring for AI agent quality.
- Set evaluation standards and partner with product and ML engineers to integrate evaluations into agent development.
What they ask for
- Experience shipping production software in complex, real-world systems.
- Experience with TypeScript, React, Python, and Postgres.
- Built and deployed LLM-powered features serving production traffic.
- Implemented evaluation frameworks for model outputs and agent behaviors.
Nice to have
- Designed observability or tracing infrastructure for AI/ML systems.
- Experience with vector databases, embedding models, RAG architectures, or evaluation platforms such as LangSmith or Langfuse.
Pay
$200K – $250K (base salary; currency not stated)
Tools
TypeScript, React, Python, Postgres, LangSmith, LangGraph, Langfuse
AI focus
Build evaluation infrastructure and production feedback loops for AI agents; assess models and agent behaviors, improve reliability, and develop observability, guardrails, prompts, retrieval pipelines, and agent orchestration.
Model-written summary · the original posting is authoritative