ML Engineer - Automated Evaluation and Adversarial Design

What you'll do

• Design evaluation methods and frameworks for single-turn and multi-turn AI experiences.
• Build adversarial test suites and stress-test pipelines for model and agent failure modes.
• Align automated and human evaluations, integrate evaluation into release workflows, and communicate readiness findings.

What they ask for

• Bachelor’s degree in Computer Science, Machine Learning, Statistics, or a related field.
• 4+ years building or significantly extending ML evaluation systems, including benchmarks or quality frameworks for sequential or multi-step outputs.
• Experience defining evaluation architecture and methodology for AI/ML systems and designing multi-turn adversarial or red-teaming tests.
• Experience with Python and ML frameworks (PyTorch, TensorFlow, or equivalent) in production or near-production settings; track record owning technical direction across multiple features or product areas.

Nice to have

• Experience evaluating user-facing AI features, productivity or creative applications, conversational AI, dialogue systems, or agentic workflows.
• Familiarity with agent orchestration frameworks (LangChain, LangGraph, CrewAI, AutoGen) and observability tooling (LangSmith, Braintrust, Arize).

Pay

$142,300 and $263,300 (base salary); $150,400 and $277,600 (base salary)

Tools

Python, PyTorch, TensorFlow, LangChain, LangGraph, CrewAI, AutoGen, LangSmith

AI focus

Develops and evaluates AI feature quality, including adversarial testing and stress testing of multi-turn conversations and agent workflows.