Fireworks AI
Member of Technical Staff, Evals Platform
$210K–$320K
San Mateo · Hybrid · Staff · Full-time · Posted Oct 30, 2025
- Experience
- 1–7 years of software engineering or data science experience; hiring at multiple levels.
About the job
Build and scale evaluation infrastructure, benchmarks, and user-facing post-training workflows so developers can assess and improve models on Fireworks.
What you'll do
- Scale evaluation infrastructure, including coordination, log syncing, and programmatic access.
- Reproduce published benchmark results and design new benchmarks for emerging or domain-specific use cases.
- Build post-training workflows for fine-tuning and work with users to turn their needs into reusable product solutions.
What they ask for
- Strong system design skills for scalable, programmatic systems and infrastructure modernization.
- Hands-on experience with sandbox environments or infrastructure for secure code execution and testing.
- Deep understanding of LLM evaluations and using results to guide model improvement.
- Understanding of the GenAI lifecycle, from prompting and dataset curation through fine-tuning and productionizing agents.
Nice to have
- Experience with the Harbor framework or similar container registry and orchestration tools.
- Interest in publicly writing model analysis, or in GPU constraints and inference optimization.
Pay
$210K – $320K USD base salary
Tools
Harbor, SFT, RFT
AI focus
Develop evaluation and benchmarking systems and post-training workflows, including fine-tuning, to measure and improve model quality.
Model-written summary · the original posting is authoritative