Automation Program Manager, Legal and Compliance Operations

What you'll do

• Improve reliability, correctness, precision, and recall by iterating on prompts, rules, and signal thresholds.
• Investigate degradation, drive measured recovery, and document causes and detection gaps.
• Build offline evaluations, regression detection, and internal measurement tools for operational teams.

What they ask for

• 2+ years of relevant experience in data annotation, labeling, quality assurance, or content review operations, including work close to automated decision output.
• Experience with guideline accuracy and inter-rater agreement, including diagnosing reviewer disagreement.
• Working knowledge of Python and SQL, including dataset analysis and calculating precision and recall.
• Experience reasoning about conditional logic and working with large operational datasets in a notebook environment.

Nice to have

• Experience in adversarial or fraud-adjacent domains, or with rules engines, decision tables, routing logic, or classifier thresholds.
• Experience with legal and compliance operations or trust and safety; prompt iteration; responsible AI practices; or AI workflow integration.

Pay

$97,000/year to $139,000/year USD (base salary); bonus and equity amounts not stated

Tools

Python, SQL, LLM

AI focus

Improve LLM-based decisioning through prompt iteration, task decomposition, routing, and abstention and fallback behavior; integrate AI tools into workflows and apply responsible AI practices.