Retool
Software Engineer, Agent Platform
New York, United States +1 · Hybrid · Full-time · Posted Sep 18, 2026
- Location
- New York, United States
San Francisco, United States - Experience
- 6+ years of professional engineering experience.
- Visa sponsorship
- Retool states it is currently set up to employ all roles in the US and specific roles in the UK; consult its careers page for UK-eligible roles.
About the job
Own the reliability, evaluation, and production behavior of AI-powered features across Retool’s products.
What you'll do
- Design prompting, retrieval, routing, tool-use strategies, evaluations, and guardrails for AI features.
- Diagnose and resolve model failures and behavior drift; prevent regressions and define fallbacks.
- Partner with product and infrastructure teams on model choices and shipping quality, cost, latency, and robustness tradeoffs.
What they ask for
- 6+ years of professional engineering experience, with ownership over complex production systems.
- Demonstrated ownership of AI/LLM behavior beyond basic integration, including mitigating variance and failure modes.
- Experience designing or maintaining evaluation frameworks, golden datasets, regression detection, or human-in-the-loop feedback loops.
- Ability to reason about probabilistic systems and tradeoffs; strong product intuition and independence in ambiguous problem spaces.
Nice to have
- Production experience with RAG, agentic systems, or tool-using models.
- Familiarity with vector databases, embeddings, or retrieval pipelines; exposure to fine-tuning, model routing, or post-training techniques; or experience building shared AI infrastructure.
Tools
TypeScript, Node.js, React
AI focus
Own AI behavior in production, including evaluation, reliability, safety, guardrails, and preventing regressions in non-deterministic systems.
Model-written summary · the original posting is authoritative