The mission
Build the evaluation, tracing, fallback, and release systems that keep production AI behavior trustworthy.
What you will own
- Own agent orchestration and evaluation infrastructure.
- Build tracing, replay, datasets, and failure analysis.
- Design fallbacks and human-review boundaries.
- Control model, prompt, and tool releases.
- Partner with voice quality and telephony engineering.
What success looks like
- Versioned evaluations connect model behavior to customer outcomes.
- Manual recovery declines and failure categories are clear.
- AI changes have release gates and rollback.
- Production decisions are traceable without leaking sensitive data.
What we are looking for
- Production LLM or agent-system experience.
- Strong software and reliability engineering.
- Evaluation design and statistical judgment.
- Security and privacy awareness.
Helpful, not mandatory
- Realtime voice AI.
- Tool-calling systems.
- Python and TypeScript.
- Model routing and cost optimization.
Compensation
The base salary range is $160,000-$220,000 USD for a US-remote employee. Final compensation will reflect role level, location policy, experience, benefits, variable compensation where applicable, and equity. Contractor arrangements are scoped separately.
How we evaluate
We use the same core evidence for every candidate: comparable outcomes, ownership, customer judgment, role skill, communication, and learning velocity. The process normally includes a short screen, a paid and bounded work sample, structured interviews, and reference checks.
Example paid work sample: Design the detection, prevention, release, and rollback plan for a regression where the agent captures a caller's name but loses the service intent.
Your first 90 days
- First 30 days: Inventory production failure modes and evaluations.
- By 60 days: Establish replay and release discipline.
- By 90 days: Materially reduce the leading manual-recovery category.
How to apply
Email kai@kaicalls.com with the role title in the subject line. Include your resume or LinkedIn profile.