What Is an AI QA Agent? Testing, Replays and Limits

An AI QA agent is software that uses an AI model to test an application the way a person would. It opens pages, follows user journeys, judges whether what it sees matches what should happen, and reports problems with evidence. Unlike a scripted test, it is not given every click in advance. It decides its next action from what is on the screen. This guide explains how AI QA agents differ from scripted automation and human testers, gives a method you can use with any tool, and covers the limits. It uses BuildBetter's QA Agent, generally available as of October 23, 2026, as a concrete example. The method does not depend on that product.

What is an AI QA agent?

An AI QA agent is an automated tester that chooses its own actions toward a stated goal, then reports suspected bugs with steps, expected and actual results, and evidence. You give it a starting point and a description of what a journey should accomplish, such as "start at the pricing page and complete a trial signup." The agent works out how to get there. It reads the page, decides what to click or type, and notices when something looks wrong.

The core design problem is trust. An AI model can misread a page, misunderstand what was intended, or hit a one-time network hiccup and call it a bug. Better agents deal with this by trying to reproduce a suspected bug. They start over from a clean state, repeat the steps, and record whether the problem happens again. A bug that shows up every time from a fresh start is far more credible than one seen once during open-ended exploration.

So the useful output of an AI QA agent is not "we tested everything." It is a short list of findings, each with enough evidence for a person to confirm or reject it quickly.

How does an AI QA agent differ from scripted automation and human QA?

Scripted automation checks exactly what you told it to check, human QA applies judgment, and an AI QA agent sits between them: broader than a script, more repeatable than a person, and less reliable than either at its worst. The three are complementary. Most teams that adopt an agent keep their scripted tests and still rely on human review.

The practical takeaway is to keep scripts for the paths where you can write a precise assertion, like "checkout total equals cart total." Point the agent at journeys where the failure could take many forms. Save human attention for reviewing what the agent found and for anything that needs taste or domain knowledge.

What should a good AI QA finding contain?

A good finding contains enough evidence that someone who was not watching the run can confirm or reject the bug without re-testing from scratch. Whatever tool you use, judge its output against this list:

• Steps: the actions taken, in order, from a defined start page.
• Expected result: what should have happened, stated plainly enough to argue with.
• Actual result: what happened instead.
• Reproduction: whether the problem recurred when the steps were repeated from a clean start, and how often.
• Replay or recording: a way to watch what the agent saw, ideally next to a log of the actions it took.

The expected result deserves extra scrutiny. It is where the agent states its assumption about how your product should behave, and that assumption can be wrong. If the expected result describes behavior you never intended, the finding is noise no matter how many times it reproduced.

Verified findings versus findings that need a look

Some tools sort results into confidence tiers. In BuildBetter's QA Agent, a completed run separates Verified bugs from Needs a look. The agent retries suspected bugs from a clean start, and each finding carries a Reproduced value showing how often the problem happened on those fresh attempts. The second bucket holds items flagged for human review instead of being presented as confirmed.

Here is how to treat each tier, in any tool that offers something similar:

• Verified: treat it as likely real, but still watch the replay before filing. Reproduction proves the agent saw the same thing twice. It does not prove the agent's expectation was correct.
• Needs a look: do not dismiss these by default. An intermittent bug that happens one time in three can be worse for users than a consistent one, because it is harder for your team to notice and fix. Spend a short, fixed amount of time on each. Confirm it, close it, or write down why it is unclear.

What is a tool-independent method for using an AI QA agent before releases?

The method has six steps: choose journeys, write expected outcomes, pick triggers, set a budget, review evidence, and route results into existing work. You can follow it with any AI QA tool, or as a checklist when you evaluate tools.

1. Choose a small set of journeys that matter most.
2. Write the expected outcome of each journey in one or two sentences.
3. Decide which journeys run on which trigger (nightly, on release, on demand).
4. Set a usage budget per trigger and check it before manual runs.
5. Review every finding against its replay before acting.
6. Route confirmed bugs into the same queue your team already works from.

1. Choose journeys by consequence, not by coverage

Start with the journeys where a break costs you the most. That usually means signup, login, the main action your product exists for, and anything involving money or data export. Five well-chosen journeys you actually review beat thirty you skim. Each journey needs a clear start page. An agent that starts in the wrong place wastes its effort before it reaches the part you care about.

2. Write expected outcomes like acceptance criteria

"Test the settings page" gives the agent nothing to judge against. "Starting at Settings, a user can change their notification email and see the new address saved after reload" does. The clearer the outcome, the fewer false findings you will get and the easier it is to check the agent's expected result.

3. Match triggers to purpose

A nightly run catches problems introduced during the day while the change is still fresh in someone's mind. A release-triggered run is a last check on the build you are about to ship. On-demand runs are for spot checks, such as after a risky merge. Different triggers can and often should run different sets of journeys.

4. Budget before you run

Agent runs cost money in some form. The next section covers this in detail.

5. Review against the replay

Watch the recording before you decide anything. You are checking three things. Did the agent do what the steps say? Is the expected result actually intended behavior? Does the failure appear in more than one attempt?

6. Route into existing work

A finding that sits in a separate dashboard tends to get ignored. Send confirmed bugs to wherever your team already triages work, with the steps, expected result, actual result, and a link to the evidence attached.

How should you budget credits for AI test runs?

Budget by deciding which journeys run on which trigger, because every run spends usage whether or not it finds a bug. Many AI QA tools charge by usage instead of by seat. BuildBetter, for example, prices on usage-based credits with unlimited seats, publishes its pricing at buildbetter.ai/pricing, and notes in its QA Agent lesson that "each run uses credits." The same planning applies to compute minutes or any other usage unit.

A simple way to plan:

• Count your runs. Nightly runs happen roughly every day of the month. Release runs happen once per release. Estimate on-demand runs separately.
• Multiply by journey count per trigger. A nightly run of three journeys and a release run of eight use usage at different rates. Look up how your vendor meters usage. Do not assume.
• Tier your journeys. Put the most critical journeys in every run. Put slower or lower-risk journeys only in release runs.
• Check limits before manual runs. Look at which journeys are selected and what limit applies before you start an on-demand check, so one curious click does not eat a week of budget.
• Review spend against outcomes monthly. If a journey has run for weeks without a confirmed finding, ask whether the product area is stable enough to drop it to release-only runs. Do not drop it from critical paths just because it is quiet.

Do not let budget anxiety push you into running too little. The point of the tool is to catch problems before customers do. A run that finds nothing on a release day is a good outcome, not wasted spend.

What does a worked release look like? (Hypothetical)

Here is a hypothetical release that shows how the method plays out from setup to triage. The team, product, and findings below are invented for illustration. None of it is a measurement or a customer story.

A small team builds a web app that schedules field technicians. They define five journeys:

1. Sign up and create a workspace (start page: public signup).
2. Create a new job with an address and time window (start page: Jobs).
3. Assign a technician to a job (start page: an open job).
4. Reschedule a job to a later date (start page: Calendar).
5. Export an invoice as a PDF (start page: Invoices).

Nightly runs cover journeys 2, 3, and 4, which change most often. The release trigger runs all five.

The release run returns three items in this hypothetical:

• Verified: Rescheduling a job into the next month saves it to the same day of the previous month. Expected: the job appears on the chosen date. Actual: it appears a month earlier. The agent reproduced it on both clean retries. The engineer watches the replay, confirms the date picker sent the wrong month, and checks that the agent's expectation matches the spec. It does.
• Needs a look: On the Invoices page, the Export button overlaps the totals column at a narrow window width. It was seen during exploration, but the finding shows it did not recur in every retry. The replay shows a late-loading font shifting the layout. It is real, intermittent, and cosmetic.
• Needs a look: During assignment, the agent expected a confirmation toast and saw none. The replay shows the assignment saved correctly. The product manager confirms the toast was removed on purpose two releases ago. The agent's expectation was outdated, so the journey description gets updated.

Triage follows the rules in the next section. The reschedule bug blocks the release because it silently corrupts data customers rely on. The layout overlap gets filed for the next sprint. The toast item is closed, and the team fixes its journey description so the same false finding does not come back. Total human time is spent on reading evidence, not on clicking through five journeys by hand.

How should you triage AI QA findings by risk?

Triage each finding on three questions: how bad is the outcome, how reliably does it happen, and how many users will hit it. Reproduction rate is one input, not the whole answer. A rare bug that loses data outranks a consistent typo.

• Block the release: data loss or corruption, money handled incorrectly, users locked out, or security-relevant behavior, at any reproduction rate.
• Fix before or right after release, with a named owner: a core journey broken for some users, or a consistent failure in a commonly used flow.
• Schedule: cosmetic or layout issues, problems in rarely used paths, intermittent issues with an easy workaround.
• Close with a note: the agent's expectation was wrong, or the behavior is intended. Update the journey description so it does not recur.

One more input is worth adding: whether customers are already reporting it. If support tickets or call notes describe the same symptom, the bug is affecting real users now. That should move it up a tier. This is easier when test findings and customer feedback sit in the same place than when someone has to search across tools.

What are the limits of AI QA agents?

AI QA agents miss bugs, report some non-bugs, and only test what you point them at, so they reduce manual testing rather than replace it. Know these limits before you rely on one:

• Coverage is bounded by your journeys. An agent exploring a signup flow will not notice a broken admin report it never visits.
• Expectations can be wrong. The agent infers intended behavior. Outdated or vague journey descriptions produce confident but false findings.
• Nondeterminism. The same agent on the same build may take slightly different paths. Reproduction from a clean start helps, but it does not remove the variation.
• Environment matters. Test accounts, seed data, feature flags, and third-party services in your test environment affect results. A bug in staging may not exist in production, and the reverse is also true.
• Some quality is hard to judge visually. Performance under load, deep accessibility compliance, and backend correctness that never shows on screen need other kinds of testing.
• Platform scope. Check exactly which kinds of apps a tool supports. Do not assume mobile or desktop apps work because web apps do.
• Data exposure. An agent that records your app sees whatever is on screen. Use test data, and read how any vendor stores and uses recordings. Our review of Granola's privacy policy is an example of reading those terms closely.
• Review time is real. Every finding needs a human decision. An agent that produces many low-quality findings can cost more attention than it saves.

None of these limits are reasons to avoid the category. They are reasons to keep scripted tests on critical paths, keep humans in the decision loop, and treat the agent's output as evidence to weigh, not verdicts to accept.

How does the QA Agent work in BuildBetter?

In BuildBetter, the QA Agent lives under Build > QA Agent and follows the same cycle described above: define journeys, run them on a schedule or release, review reproduced findings with replays, and hand them off. The product lesson sums it up: "You can now have BuildBetter test your app before each release, then turn reproduced bugs into work for your team." According to the University lesson, the feature may need to be turned on for your workspace, and some settings are admin-only.

• Your QA team: AI testers that explore test flows, check layout, and retry suspected bugs from a clean start.
• Test flows: each has a start page and a description of what it tests. Admins use Edit test flows to control which journeys are included.
• Automatic runs: in QA Agent settings, next to Automatic runs, choose Edit, then under When pick a nightly schedule or a GitHub release trigger.
• Run now: an on-demand check. You review the selected test flows and credit limits before starting.
• Recent runs: a completed run shows Verified bugs and Needs a look. Each finding lists steps, expected result, actual result, and Reproduced.
• Replay: watch the exploration, repro 1, and repro 2 alongside the recorded agent actions.
• Hand-off: File to Inbox adds the bug to your team's work in Inbox and Triage, which connect with Linear and Jira. Send to hands the same brief to a coding agent.

Frequently asked questions

These are short answers to the questions teams most often ask when evaluating an AI QA agent.

Can an AI QA agent replace my automated test suite?

No. Scripted tests give precise, repeatable assertions on critical paths, and they run cheaply in CI. An agent adds broader exploration of journeys where failures can take many forms. Use both.

Does a verified finding mean the bug is definitely real?

It means the agent saw the problem again when it repeated the steps from a clean start. That is strong evidence, but the agent's expected result could still be wrong. Watch the replay and confirm the expected behavior before filing.

Should I ignore findings that need a look?

No. Give each one a short, fixed review. Intermittent problems can be more harmful than consistent ones because they are easy to miss. Close the ones that reflect intended behavior, and update the journey description so they do not return.

How many journeys should I start with?

Start with the handful whose failure would hurt most, usually signup, login, your core action, and anything involving payments or exports. Add more once you trust the review routine and understand your usage.

How do I keep credit usage predictable?

Assign journeys to triggers on purpose. Run critical journeys nightly and the full set on release. Check selected flows and limits before any on-demand run, and review spend against confirmed findings each month.

Where should you start?

Start by writing down your three to five most important journeys, each with a start page and a one-sentence expected outcome. That list is useful with or without any tool. It sharpens manual testing, scripted tests, and agent runs alike. When you want to run those journeys automatically before every release, open BuildBetter, go to Build > QA Agent, and walk through the QA Agent lesson to set up your first test flows, choose a nightly or release trigger, and review your first reproduced findings next to the customer feedback they affect.