AI Evaluations Engineer, US Decision Intelligence

What you'll do

• Build and operate evaluations for LLM outputs and agent actions, including rubric-based scoring and real-trace evaluation.
• Instrument workflows and own the platform-wide pass/fail evaluation gate, rerun policy, and variance thresholds.
• Partner with AI engineering, platform, and global evaluation teams to define criteria and improve observability, retries, and logging.

What they ask for

• 5+ years of experience in data and AI-related fields, such as AI engineering, software development, ML engineering, data science, or QA.
• Strong Python skills and proficiency in SQL, plus experience with at least one major data analytics platform.
• Hands-on experience with AI evaluation techniques and Langfuse or similar LLM observability tools.
• B.S. in Computer Science/Engineering, or equivalent work experience.

Nice to have

• Familiarity with embeddings, retrieval algorithms, agents, and vector or graph database data modeling.
• Experience with GenAI and RAG strategies, microservices, recommendation systems, agent evaluation, or distributed systems technologies.

Pay

USD $184,700 to $277,600 base salary

Tools

Python, SQL, OpenAI, Anthropic, Gemini, Pinecone, FAISS, Langfuse

AI focus

Develop and evaluate AI products and agentic workflows, including LLM output quality, task completion, tool correctness, and grounded recommendations; improve accuracy, performance, and latency.