How to QA AI customer support agents

An AI agent’s mistakes repeat until someone changes the agent, so QA has to run continuously and on every conversation. A human agent who misreads a refund policy affects their own queue. An AI agent with the same gap gives the wrong answer to every customer who asks.

Two more things make AI agents different from human agents:

• They change on a release cycle. A new prompt, model, knowledge article or tool can break behavior that worked last week. Intercom, Sierra and Decagon all build regression testing into their platforms for this reason.⁷⁹¹¹
• The vendor grades its own work. Each AI agent vendor reports its own resolution or automation numbers. Ada, for example, classifies each conversation as Resolved or Not Resolved using its own criteria.¹³ Those numbers are useful for running the agent. They come from the same company that built it.

NIST’s AI Risk Management Framework asks for exactly that check. It says AI systems should be monitored “when in production,” and that “independent assessors” or internal experts who didn’t build the system should take part in regular assessments.¹

If you’re new to the topic, start with what AI agent QA is. This guide is the step-by-step version.

Start by listing the agent’s jobs and its limits, because you can’t grade behavior nobody defined. For each intent the agent handles, write three things:

• The job. What a successful outcome looks like, such as “the customer received the correct return label.”
• The rules. The policy, disclosure and verification steps that apply, such as “verify identity before changing an address.”
• The exits. When the agent must hand off to a person, and what context travels with the customer.

Most of this already exists in your support SOPs. Our guide on turning SOPs into a support QA rubric shows how to convert them into checks.

Use your AI agent vendor’s own testing tools before any change goes live, because they catch regressions before customers see them. As of October 2026:

• Intercom Fin: Evals run simulated conversations built from your own conversations and score each one pass or fail, combining deterministic checks with an LLM judge. Releases let you test changes away from the live version and ramp traffic or run an A/B test.⁷
• Sierra: Simulations run conversations between the agent and mock personas, graded by a separate judge agent, and can gate releases in CI/CD pipelines.⁹
• Decagon: Simulations auto-generate tests, add checkpoints at each stage of a conversation and integrate with CI/CD practices to test every agent version.¹¹
• Ada: the Conversation View shows each conversation’s full transcript, reasoning log and resolution classification, so you can check how configuration changes affect real responses.¹³

Simulations answer one question well: does the agent still behave as designed on the cases you thought of? They can’t tell you what happened in the conversations you didn’t think of. That’s what the next steps cover.

Send the agent’s real conversations to a QA tool that doesn’t sell the agent, so the score doesn’t depend on the vendor’s own definition of success. Look for three things:

• The source. Can it read conversations from your AI agent directly, or from the helpdesk where they land? Check for both, because some AI agent conversations never reach the helpdesk.
• Coverage. Does it score every connected conversation, across chat, email, messaging and voice?
• Privacy. Is personal data removed before conversations are stored or scored?

unitQ Support QA grades AI agents from Sierra, Ada, Decagon and Intercom Fin, and scores 100% of connected conversations across chat, email, messaging, voice and IVR through integrations including Zendesk, Salesforce Service Cloud, Intercom, Freshdesk, Kustomer, Amazon Connect and Five9.³⁵ unitQ removes PII in its data pipeline before conversations are stored or processed, and is SOC 2 Type II and HIPAA compliant.⁶ Our comparison of secure support QA platforms covers the security questions in detail.

Grade the AI agent against the quality standards you already hold your human team to, then add checks for the ways bots fail. Most teams run several rubrics and scorecards, by channel, queue or policy. Shared standards let you compare an AI agent’s resolution, accuracy and policy adherence with your people’s on the same terms.

Shared checks cover what every agent should do:

• Gave correct information that matches your policy.
• Followed required steps, such as identity verification and disclosures.
• Resolved the issue, or handed off correctly.
• Used the right tone for the situation.

Bot-specific checks cover failure modes human agents rarely have:

• False containment. The conversation counted as contained, but the customer gave up.
• Loops. The agent repeated the same answer while the customer rephrased.
• Wrong or invented answers. The agent stated a policy or fact that isn’t true.
• Missed or unnecessary escalations. The agent kept going when it should have handed off, or handed off something it could have solved.

unitQ Support QA runs as many rubrics and scorecards as your policies require: one shared standard for human and AI agents, or bot-specific checks. Its bot checks cover real containment, wrong information, loops, escalations that shouldn’t have happened, inaccurate resolutions and recovery after reaching a person.³⁴ For what each dimension should measure, see how to measure AI agent conversation quality. For handoffs specifically, see how to audit AI agent handoffs.

Check the automated grader against your best human reviewers on real past conversations before its scores go live. Research on LLM-as-a-judge found strong LLM judges can match human preferences at a level similar to agreement between humans, and also documented biases such as position and verbosity bias.² Calibration is how you find out which you’re getting.

A practical calibration loop:

1. Pick a set of past conversations that covers your main intents and a few known failures.
2. Have two or three reviewers score them on the rubric, and settle disagreements.
3. Run the automated grader on the same set and compare, check by check.
4. Rewrite any check where the grader and reviewers disagree, usually by tightening the definition or adding a worked example.
5. Repeat on a fresh set on a regular schedule, and whenever you change the rubric.

In unitQ Support QA, every check has a plain-language definition, fail values and worked examples, and Testbench runs it on your own past conversations next to your reviewers before it goes live.⁴

Once the grader is calibrated, score 100% of the AI agent’s live conversations and put the results next to the vendor’s dashboard. A sample can miss a failure that’s concentrated in one intent, and full coverage shows it.

The comparison is the point. When the vendor reports a conversation as resolved and your rubric says it failed, read the transcript. Common reasons:

• The vendor counts a contained conversation as resolved, but the customer came back on another channel.
• The answer was relevant but contradicted your policy.
• The agent closed the conversation before the customer confirmed the fix.

Every score should link to the line in the conversation that earned it, so reviewers can check the grader quickly. unitQ Support QA attaches transcript evidence to every miss.³

Sort every failure by cause and send it to its owner, because a QA score nobody acts on doesn’t improve the agent. Typical owners:

• Knowledge. The agent was missing an answer or used a stale article.
• Conversation design. The flow looped, asked for the same detail twice or ended too early.
• Policy and compliance. The agent made a commitment your policy doesn’t allow, or skipped a required step.
• Product. The agent answered correctly, but the customer’s problem is a bug or a confusing feature.

unitQ Support QA’s bot audit returns findings with a suggested owner, such as conversation design.⁴ With unitQ Monitor, quality misses can sit beside contact reasons and product issues, so support and product teams work from the same view.³

After each fix, re-run your simulations and watch the live scores for the intents you changed. NIST’s framework also calls for post-deployment monitoring plans that include change management and incident response.¹

Turn each confirmed failure into a new test case in your vendor’s simulation suite, so the same mistake can’t ship twice. Sierra, for example, turns annotated real conversations into regression tests.¹⁰

Intercom, Sierra, Decagon and Ada all ship their own testing and quality tools, and they’re worth using. The table shows what each offers as of October 2026, and what an independent grader adds.

Built-in tools need little setup because they run inside the agent’s own platform. The catch is that the company scoring the agent also built it and decided what success means. Our list of questions to ask an AI support agent vendor covers what to request from them.

AI agent QA tools fall into three groups: the AI agent vendor’s own tools, contact center QA suites, and independent graders. As of October 2026:

• AI agent vendor tools: Fin Monitors, Sierra Monitors, Decagon Watchtower and Ada’s resolution classification and Custom Metrics, covered above.
• Contact center QA suites that also sell AI agents: Zendesk QA scores all interactions, including AI agents and voice, and Zendesk sells AI agents separately.¹⁵ Level AI scores 100% of calls, chats, emails and bot conversations and also sells an AI virtual agent.¹⁶ Observe.AI scores 100% of AI-led conversations for accuracy, empathy, compliance and resolution, and offers a platform to build AI agents.¹⁷¹⁸
• Conversation analytics and independent graders: Rippit (formerly MaestroQA) offers agents that monitor every chatbot conversation for incomplete resolutions, wrong answers and coverage gaps.¹⁹ unitQ Support QA grades human and AI agents on your rubric and doesn’t sell the agent, the platform or the labor it grades.³

For a full side-by-side, see the best AI support QA platforms compared.

If you run one AI agent, inside one vendor’s stack, and you’re comfortable with that vendor’s definition of a resolution, its built-in tools may be all you need. Fin Monitors, Sierra Monitors and Decagon Watchtower all review every conversation against criteria you set.⁸¹⁰¹² If your whole stack is Zendesk, Zendesk QA keeps human and AI agent scores in one place.¹⁵ If voice dominates and you also want the AI agent from the same vendor, Level AI and Observe.AI are built for contact centers.¹⁶¹⁷

Add an independent grader when you run more than one AI agent, when you need to hold bots and people to the same standards, or when someone outside support needs to trust the numbers.