Member of Technical Staff, LLM Evaluation Infra

What you'll do

• Build automated evaluation pipelines integrated with model training and deployment workflows.
• Develop evaluation frameworks, benchmarks, and quantitative metrics for generative models.
• Analyze model outputs statistically and collaborate with product and customer-facing teams on evaluation criteria.

What they ask for

• BS/MS/PhD in Computer Science, Machine Learning, Statistics, or a related field (or equivalent experience).
• At least 2 years of experience in ML evaluation, applied ML research, or a related engineering role.
• Proficiency in Python and ML frameworks such as PyTorch; experience designing evaluation metrics and benchmarks for generative models.
• Experience with Git, Docker, and cloud services (AWS, GCP, or Azure); understanding of LLM fundamentals.

Nice to have

• Familiarity with benchmark suites, agentic evaluations, MLOps, CI/CD, or cloud-scale evaluation infrastructure.
• Experience with data engineering, synthetic data, LLM safety evaluation, or human-in-the-loop evaluation systems.

Tools

Python, PyTorch, Git, Docker, AWS, GCP, Azure, Kubernetes

AI focus

Build evaluation systems for LLMs, including measurement of performance, safety, reliability, regressions, failure modes, and biases.