What is auto QA?

• Auto QA uses AI to grade support conversations against your own scorecard.
• It covers the conversations you connect, where manual QA covers a sample.
• Each check returns pass, fail or not applicable, linked to the evidence.
• People still own the scorecard, review the misses and coach the team.
• Judge any auto QA tool by testing it on your own conversations.

Auto QA works in five steps: connect your support conversations, define the checks, grade each conversation, review the evidence and roll the results up into reports. The scorecard is the part your team controls, and it decides how useful everything after it will be.

1. Connect conversations. Support conversations, including chat, email, messaging, voice and IVR (automated phone menus), come in from the platforms your team already uses.
2. Define the checks. Each check has a plain-language definition, a clear fail and worked examples of edge cases.
3. Grade each conversation. The AI applies every check and marks it pass, fail or not applicable, with the part of the conversation that supports the result.
4. Review the evidence. Reviewers open the misses, confirm the pattern and flag checks that need a clearer definition.
5. Roll it up. Scores roll up by check, team, contact reason and agent, so leaders see where quality drops and why.

If you want the bigger picture of the discipline auto QA automates, read what support QA is.

Support teams are looking at auto QA because AI is taking on more of customer service, and a manual sample wasn’t designed to grade AI agents, outsourced teams and in-house agents together. In a Gartner survey of 199 service and support leaders conducted in April and May 2026, spending on AI rose 38% while overall service and support budgets grew 2% ². Looking two years ahead, those leaders predicted GenAI chatbots will become the most valuable customer service channel ².

As more conversations come from bots and vendors, teams need a way to grade them independently of whoever built or staffed them. Gartner’s Daniel O’Sullivan put the priority this way: “The technologies that will create the most value are those that help organizations build, deploy and govern these agents securely and at scale” ².

Auto QA grades the conversations you connect against your scorecard using AI, while manual QA has trained reviewers grade a sample by hand. The two work well together: auto QA provides coverage, and people handle judgment calls, coaching and scorecard design.

Auto QA can check anything you can define clearly enough that two experienced reviewers would agree on the result. Common check groups include:

• Resolution: was the answer correct under your policy, and were next steps clear?
• Process adherence: did the agent follow the required steps, such as an escalation when your criteria are met?
• Compliance: identity verification, privacy handling and required disclosures.
• Communication: tone, clarity and empathy, defined in your own words.
• AI agent behavior: loops, wrong information, missed handoffs and escalations that shouldn’t have happened.

Checks that rely on information outside the conversation, such as whether a refund posted in a billing system, belong in that system’s own reports.

Here’s how auto QA might grade a single billing chat. In this illustrative example, the customer asks for a refund on an annual plan. The agent shares the renewal date before verifying identity, explains the refund policy correctly, and closes with clear next steps.

The cancellation disclosure doesn’t apply, so it’s left out. The process group scores 0 because its only applicable check failed, while resolution and communication score 100. With these illustrative weights, the conversation scores 70: (0.4 × 100) + (0.3 × 0) + (0.3 × 100). The reviewer opens the conversation, sees the renewal date shared before verification, and the miss goes into the agent’s next coaching session.

The weights and math are illustrative, so treat them as a teaching example. For more on designing the groups and weights, see our guide to support QA scorecards.

Auto QA for AI agents grades bot conversations on criteria built for how bots fail: loops, dead ends, wrong information, escalations that shouldn’t happen and handoffs that go wrong. What AI agent QA is covers that discipline in full, and how to measure AI agent conversation quality covers the metrics behind it. Our eight-step playbook covers the rest, including calibrating auto QA for AI agents before you trust its scores.

unitQ Support QA grades human and AI agents side by side in one product, on as many rubrics and scorecards as your policies require, with one shared standard or bot-specific checks, independent of the vendor that built the bot. AI agents get an independent audit either way: is containment real, was the information correct, did the bot loop or dead-end, escalate when it shouldn’t, misunderstand the issue, or resolve it inaccurately, and how often conversations recover after reaching a person ¹. The bot audit is configured for your workspace during setup.

Evaluate auto QA by running it on a set of your own past conversations that your reviewers have already graded, then comparing the results check by check. NIST’s AI Risk Management Framework says accuracy measurements “should always be paired with clearly defined and realistic test sets” that are “representative of conditions of expected use” ³. A vendor’s headline figure, measured on someone else’s conversations, can’t tell you how a tool will grade yours.

1. Build a test set from your own history. Pull past conversations across your main contact reasons, teams and vendors, including the messy ones.
2. Grade it by hand first. Have two experienced reviewers grade the set against the same checks, and settle their disagreements before you compare.
3. Run auto QA on the same set. Compare results check by check as well as on the overall score.
4. Read the disagreements. For each mismatch, open the evidence. If the check’s definition is ambiguous, tighten it so both the AI and your reviewers apply it the same way.
5. Check by segment. NIST notes that accuracy measurements “may include disaggregation of results for different data segments” ³. Compare results by contact reason, team and vendor as well as in aggregate.
6. Test edge cases on purpose. Include conversations where a check shouldn’t apply, so you can confirm not-applicable results are handled correctly.

You can try a check on real past conversations before it goes live, using Testbench in unitQ Support QA, and every scored miss links back to the conversation. Ask to include a side-by-side check on a sample of your own conversations in your evaluation.

Manual QA still fits wherever human judgment is the point. Small teams with low volumes, brand-new programs still working out what a fail looks like, and sensitive cases such as complaints with legal exposure all benefit from a trained reviewer. Teams that adopt auto QA can move their strongest reviewers from grading to refining checks, auditing misses and coaching.

unitQ Support QA is an independent grader for support quality. It grades the support conversations you connect against a rubric your team owns. unitQ Support QA supports all your support channels, including chat, email, messaging, voice and IVR (automated phone menus), through integrations with platforms such as Zendesk, Salesforce Service Cloud, Intercom, Freshdesk, Kustomer, Amazon Connect and Five9 ⁴.

• Buyer-editable rubric. You define each quality check in plain language: what it means, which answers count as a fail, and worked examples that show the edge cases. Your team owns and edits the rubric in Tag Studio, the rubric editor in unitQ Support QA.
• Weighted scores. You can weight rubric groups to match how you score quality. Checks that don’t apply to a conversation are left out of the score.
• One score with context. One quality score built from your rubric, with competency, outcome, operations and compliance views beside it, in the Quality report in unitQ Support QA.
• Breakdowns. Scores break down by contact reason, line of business, BPO vendor vs in-house team, region, manager and agent, all on the same scorecard. Small samples are flagged.
• Outcome context. You can track quality alongside outcome metrics you already measure, such as CSAT, where that data is connected.