Arena
Machine Learning Scientist - ML Research
SF Bay Area · On-site · Full-time · Posted Dec 18, 2025
- Experience
- Not specified; requires a PhD or equivalent research experience.
- Education
- PhD or equivalent research experience in Machine Learning, Natural Language Processing, Statistics, or a related field.
About the job
Conduct research and build methods to evaluate AI models using human preference data, informing Arena’s leaderboard and tools for model developers.
What you'll do
- Design experiments, metrics, and evaluation protocols for model behavior and performance.
- Analyze large-scale human voting and interaction data to understand model performance and user preferences.
- Collaborate with engineers and product teams to scale research findings into production and communicate results through reports and publications.
What they ask for
- Hands-on experience training large-scale models, including reward and preference models and fine-tuning LLMs with methods such as RLHF, DPO, and contrastive learning.
- Strong foundation in machine learning and statistics, with experience designing training objectives, evaluation schemes, or statistical frameworks.
- Proficiency in Python and ML research libraries such as PyTorch, JAX, or TensorFlow.
- Demonstrated ability to design and analyze experiments rigorously; experience publishing research or contributing to open-source ML, NLP, or AI evaluation projects.
Tools
Python, PyTorch, JAX, TensorFlow, Transformers, diffusion models, RLHF, DPO
AI focus
Research and evaluate AI models, including training reward and preference models and fine-tuning LLMs; analyze human preference signals to improve understanding of model capability, reliability, and alignment.
Model-written summary · the original posting is authoritative