AI Evaluation Engineer

What you'll do

• Create an assessment framework for design tasks where multiple answers may be valid, and build repeatable tests from realistic tasks and failure cases.
• Work with domain experts, users, and internal teams to assess system performance; develop automated graders and human review processes and validate grader agreement with experts.
• Turn evaluation findings into ranked improvement priorities for the system’s reasoning.

What they ask for

• About five or more years of professional experience in hardware engineering, AI, or both, with depth in one and practical familiarity with the other.
• Substantial hands-on experience evaluating AI systems, including creating test tasks, building automated graders, and benchmarking against human review.
• Solid grasp of autonomous, agent-style AI systems and grounding in a hardware engineering field sufficient to assess design quality.
• Based in the US or Canada and able to travel to an office for one week every 6 weeks.

Nice to have

• Prior experience at a robotics or physical AI company.
• Experience building things from zero in an early-stage startup.

Pay

USD 200,000 to 300,000 per year (base salary)

AI focus

Evaluate an AI system that performs technical design work, including assessing its reasoning through expert-informed frameworks, tests, automated grading, and human review.