Agent Quality Explained: Reviewing Agent Health and Tracking Fixes Before Callers Notice

Agent Quality is a weekly review queue that checks each agent's recent calls, scores its health, and surfaces the agents that need attention — so you catch a drifting prompt or a silent failure before a caller runs into it. It lives at Dashboard > Agent Quality, and it looks across your whole fleet of agents at once, not just one call at a time.

A prompt that worked perfectly the day you wrote it can quietly stop working. A tool starts failing silently. A required question gets skipped on call after call because a wording edit changed how the agent reads the instruction. None of that shows up unless someone happens to read a transcript. Most businesses don't have time to read transcripts every week, so problems like these sit unnoticed until a customer complains.

This guide covers what Agent Quality actually shows, how it's different from grading a single call, and how to work the findings queue so problems get fixed instead of piling up.

---

Table of Contents

1. What Agent Quality actually is
2. What it surfaces
3. How this is different from per-call grading
4. What it can't do
5. How to use it, step by step
6. What it costs
7. FAQ

---

What Agent Quality actually is

Open Dashboard > Agent Quality and you land on a roll-up across every agent you run, not a deep dive into one. Each evaluated agent gets a health score, summarizing how it has been performing on its recent calls. Agents with no recent problems sit near the top of the list. Agents with repeated or severe findings drop, so the ones that need eyes rise to where you'll actually see them.

The page shows summary cards for healthy agents versus agents needing attention, then an agent table you can scan for health score, recent call volume, last call date, and a suggested next step for each one. Hit Refresh after you've made a prompt change or the agent has taken fresh calls, and the newest status pulls in.

What it surfaces

A finding is something worth a person's eyes — a call that ended without resolving, a silent failure, or behavior that drifts from the prompt you wrote. Findings carry a severity, and the table sorts so the most urgent agents rise to the top. The goal is triage: you shouldn't have to read every call to know where to spend your time.

Beyond the findings themselves, the page shows:

• A health score per evaluated agent, so you can compare agents against each other at a glance
• The latest evaluation run and recent run history, so you know when the score was last refreshed
• A status you can set on each finding — seen, fixing, ready to verify, or ignored — so a team can work the queue together instead of duplicating effort or losing track of what's already been handled

How this is different from per-call grading

KaiCalls already grades individual calls. The seven-check eval scores greeting adherence, required questions, data collection, improvisation, behavior rules, guardrails, and transfer handling on each eligible call, and Call Quality is the per-agent deep dive into that data — latency, cost, and the eval breakdown behind a single agent's numbers.

Agent Quality sits a level above that. Instead of one call or one agent's detail page, it's the fleet view: which agents are healthy and which ones to look at first, across everyone you run. Open an agent from the Agent Quality table when you want the detail behind a low score, and you land in Call Quality to see the per-call evals that produced it. Think of Agent Quality as the triage list and Call Quality as the file you open once you've picked what to work on.

What it can't do

Agent Quality reports on real call activity — it doesn't invent a score from nothing. An agent with no recent calls to evaluate shows no score, because there's nothing yet to judge. It also doesn't rewrite your prompt or your tools for you; it tells you which agent and what kind of problem, and you (or the Learning Agent's suggestions, or a manual edit in Customize Your Agent) make the actual fix. And a score doesn't move the instant you save a prompt change — health is based on real calls, so it updates only after the agent has taken fresh traffic and you refresh.

How to use it, step by step

Review agent health:

1. Go to Dashboard > Agent Quality.
2. Review the summary cards for healthy agents and agents needing attention.
3. Open the agent table to compare health score, recent call volume, last call date, and the suggested next step for each agent.
4. Use Refresh after recent call or prompt changes when you need the newest status.

Work through findings:

1. In the findings table, start with high-severity items and the lowest health scores — they affect the most calls.
2. Open the suggested fix or the agent link when one is available.
3. Update the finding status as your team works on it: seen, fixing, ready to verify, or ignored.
4. After the agent has taken fresh calls, mark fixed items ready to verify so the next run confirms the fix held.

Treat Agent Quality as the weekly review queue for prompt and tool cleanup, and fix the most repeated findings first.

What it costs

Agent Quality is included on every KaiCalls plan — Solo starts at $69/mo, Pro at $199/mo, both with a generous answered-call allowance and no per-minute overage. There's no extra fee to review findings or track fixes; it runs against the calls your plan already covers.

FAQ

Do I need to set anything up to use Agent Quality? No. It works off the calls your agents are already taking. Go to Dashboard > Agent Quality and it shows whatever agents have enough recent activity to score.

Why does one of my agents have no score? It probably hasn't taken enough recent calls to evaluate yet. Scores appear once there's activity to judge.

I fixed the prompt but the score hasn't changed. Why not? Health is based on real calls, not on the prompt text itself. Give the agent fresh traffic, then refresh — the score updates as new calls come in.

A finding isn't actually a real problem. What do I do with it? Mark it ignored. That clears it from the active queue without deleting the record, so it stops cluttering the view your team works from.

Is this the same as the seven-check call grading? No. The seven-check eval grades one call at a time and lives inside Call Quality. Agent Quality is the roll-up across all your agents — it points you at which agent to look at; Call Quality and the per-call evals are where you see why.

---

See your agents' health for yourself. Open Dashboard > Agent Quality, or call the KaiCalls demo line at (417) 386-2898 to talk to Kai first, or visit kaicalls.com.