QA Engineer-AI Native Quality

What you'll do

• Build and maintain eval suites, scoring, and regression tracking for agent behavior changes; run relevant evals in CI and gate releases on results.
• Test agent flows and AI-specific risks, and turn production issues and customer reports into permanent eval cases.
• Design agent-driven QA workflows, automate repeatable checks, and perform exploratory testing across roles, features, and environments.

What they ask for

• 4+ years in QA or test engineering on a complex web product.
• Hands-on experience building evaluations for LLM or agent products, or a clear track record of getting there fast.
• Working knowledge of agent concepts including prompting, skills, tool calling, context, RAG, and orchestration.
• Python (or similar) for eval tooling and data checks; Playwright or similar for end-to-end tests.

Nice to have

• Experience in adtech, martech, or marketing analytics.
• SSO / SAML / OAuth flows, data-connector testing, or eval or observability tooling.

Pay

$115,000-130,000 USD base salary

Tools

Python, Playwright, LLM-as-judge, RAG, SSO, SAML, OAuth, CI

AI focus

Build and validate agent evaluations; test changes to models, prompts, skills, and tools; design guarded AI-agent workflows for QA; probe AI-specific risks such as prompt injection, data leakage, and hallucinated analytics.