AI Exploratory Testing vs Scripted E2E Tests: When to Use Each (2026)

Most B2B SaaS teams in 2026 run scripted end-to-end tests, AI exploratory testing, or both, and they still argue about where each one belongs. This guide from the BuildBetter team gives you a way to split your user journeys between the two. It includes a scoring method you can run in a spreadsheet, a decision table, a protocol for verifying AI-found bugs, and a clear rule for when scripts can't be replaced. You don't need budget to start. You need a list of your journeys and an hour of honest scoring.

The argument your team is probably having right now

One engineer asks, "Why are we maintaining 300 Playwright tests when an agent can just click through the app?" Another answers, "Because I can't put a non-deterministic agent in front of the checkout release gate."

Both of them are right, about different parts of the app. The useful question is which flows get which treatment. You can decide that with a spreadsheet.

By the end of this page you will have four things: a scored inventory of your user journeys, a decision table that assigns each journey a primary treatment, a protocol for reviewing agent findings before anyone files a ticket, and a firm line for when scripted tests are required.

What is the difference between AI exploratory testing and scripted E2E tests?

A scripted end-to-end (E2E) test is code that performs a fixed sequence of user actions and asserts a fixed expected result every time it runs. AI exploratory testing gives an AI agent a goal and a starting page; the agent chooses its own path through the app and reports behavior that looks wrong.

The short version: scripted tests answer "Does the thing we already know about still work?" Agent exploration answers "What is broken that we didn't think to check?" The first is regression testing. The second is discovery.

Neither idea is new. Cem Kaner coined the term "exploratory testing" in the 1980s, and James Bach's widely cited definition is "simultaneous learning, test design, and test execution." Michael Bolton and James Bach later separated checking (algorithmic confirmation of known expectations) from testing (evaluation through exploration). Scripted suites are checking. An AI QA agent automates part of testing, specifically the exploring.

The test oracle is the real dividing line

A test oracle is how you decide whether a result is correct. A scripted test carries an explicit oracle: the assertion, such as expect(total).toBe('$108.00'). An agent infers an implicit oracle from the goal description plus general heuristics such as error pages, console exceptions, overlapping elements, and dead buttons. Researchers call the difficulty of knowing the correct expected output the oracle problem (Barr et al., IEEE TSE, 2015). It is the main reason agent findings need human verification.

• Determinism: Scripts take the same path every run. Agents may take a different path each run.
• Cost per run: Scripts cost little more than CI minutes. Agent runs consume model or vendor credits.
• Maintenance: Scripts break when selectors or copy change. Agents adapt to UI changes but need clear goals.
• Bugs found: Scripts catch regressions on known paths. Agents find unknown-unknowns, layout breaks, and odd state combinations.
• Evidence: Scripts give pass or fail against a named assertion. Agents give steps, a guessed expected result, an actual result, and usually a recording.

Why this decision is hard: what breaks on each side

Each approach fails in a predictable way, and the failures are mirror images of each other. Scripts fail by being too narrow and too fragile. Agents fail by being too loose to prove anything specific.

What breaks with scripted suites

• Brittleness. A renamed button or a changed CSS class breaks a test that has nothing to do with a real bug. After enough of these, teams start ignoring red builds.
• Coverage ceiling. Scripts only check paths someone thought of. Bugs off the happy path, such as unusual input orders, back-button states, or elements overlapping at 1,180px wide, go unseen.
• Maintenance drag. Every redesign creates a backlog of test rewrites. On fast-moving features the suite is often wrong before it is useful.
• Flaky end-to-end tests. Google reported in 2016 that about 1.5% of all test runs produced a flaky result and almost 16% of tests showed some flakiness. A 2014 study of open-source projects (Luo et al., ACM FSE) found async waits caused roughly 45% of analyzed flaky tests, followed by concurrency and test-order dependency. Flakiness is a trust problem first. Once people learn to hit "re-run," the suite stops working as a gate.

What breaks with AI agent exploration

• Non-determinism. Two runs can take different paths. That helps discovery and hurts proof that the same thing was checked twice.
• False positives. An agent may flag intended behavior as a bug, or fail because of its own navigation mistake. On the WebArena benchmark at launch in 2023, the best GPT-4 agent completed about 14% of realistic web tasks against about 78% for humans. Scores have climbed since, but agents still trail human reliability on multi-step tasks. Without a reproduction step, you are triaging noise.
• Per-run cost. Running an agent on every commit gets expensive fast.
• Weak oracle on business rules. An agent can see that a page returned a 500 error. It cannot know your proration rule is wrong unless you told it the expected number.
• Audit gaps. A run that differs each time is weak evidence in regulated flows.

The core tension is that you are trading determinism for breadth. The method below decides where each trade is acceptable.

The method: a 6-step split you can run today with a spreadsheet

You can build a hybrid QA strategy without buying anything. Everything below works with a spreadsheet, your existing CI, and a person running manual exploratory sessions if you have no agent yet.

Step 1. Inventory your user journeys

List every journey a user completes end to end: sign up, invite a teammate, check out, upgrade a plan, export a report. Use three columns: journey name, start page, and what success looks like. Aim for 10 to 30 rows. Fewer than 10 means your journeys are too coarse. More than 30 usually means you are listing screens instead of journeys.

Step 2. Score each journey on five criteria (1 to 3 each)

• (a) Risk: revenue or data damage if it breaks.
• (b) Change frequency: how much the flow will change next quarter.
• (c) Audit need: whether you need identical, repeatable proof.
• (d) Oracle clarity: whether you can write the exact expected result.
• (e) Unknown-unknown risk: new, complex, or heavily branched UI.

Step 3. Apply the decision table

Step 4. Set the hybrid default

Most teams end up here: scripts for money paths and anything you must prove, agents for breadth, new features, and layout. The rule of thumb is simple. A script should exist only where you can name the assertion that would block a release. If nobody can say what a failing test would stop, the test is overhead, not coverage.

Step 5. Set cadence by cost

Scripts are cheap per run, so run them on every pull request. Agent exploration costs per run, so run it nightly or on release candidates. Write down a monthly run budget before you start, and review actual spend after two weeks.

Step 6. Install a verification protocol for agent findings

Never file agent findings unreviewed. Use this five-part check:

1. Require reproduction from a clean start. A single unreproduced occurrence is "needs a look," not a bug.
2. Read steps, expected result, and actual result. If the expected result is the agent's guess, check it against the spec or the product owner.
3. Watch the recording or replay before filing.
4. Classify each finding as a real bug, intended behavior (update the journey description), or agent error (refine the start page or goal).
5. Promote any confirmed bug on a money path into a new scripted regression test.

Manual fallback for zero-budget teams

Run a 30 to 60 minute human exploratory session per journey using a written test charter. This is Session-Based Test Management, which Jonathan and James Bach formalized around 2000. Elisabeth Hendrickson's template from Explore It! works well: "Explore <target> with <resources> to discover <information>." Log findings in the same steps/expected/actual format. The split logic stays identical. If writing the scripted half is the bottleneck, our guide to Playwright test generation with AI covers how to write and maintain those tests faster.

Worked example: a SaaS app with 12 scripted checkout tests and nightly agent runs on 8 journeys

This is an illustrative scenario, not a customer case study.

A team with a large, flaky suite usually gains more safety by shrinking it than by adding to it. Picture a B2B SaaS app with self-serve billing. The team has 140 legacy Playwright tests, and about a third fail intermittently. The debate is whether to delete the suite and switch entirely to agents.

They inventory 20 journeys and score them. A sample:

Decision 1: cut the scripted suite from 140 to 12 tests. The survivors cover checkout and billing: card success, card decline, plan upgrade, downgrade proration, coupon applied, tax by region, invoice generated, and similar. They run on every PR and gate releases, because each has a precise assertion and direct revenue impact.

Decision 2: archive about 128 tests covering settings, profile, and admin screens. Brittle selectors made them low-value. Their coverage moves to exploration.

Decision 3: run nightly agent exploration across 8 journeys: onboarding, invite teammate, create project, import CSV, dashboard filters, export report, notification settings, and account deletion request. Each has a high change rate or a wide UI surface, and a miss would be embarrassing rather than costly.

Decision 4: set a budget. One nightly run across 8 journeys is roughly 30 runs per month. The team sets a credit ceiling and reviews spend after two weeks.

One night of triage

A hypothetical night yields five findings:

• Two reproduce from a clean start on every attempt, and the replay confirms them. Filed.
• One reproduces once in three attempts. Marked "needs a look"; a human checks it the next morning.
• One is intended behavior (empty-state copy). The journey description is updated so the agent stops flagging it.
• One is an agent navigation error. The start page is clarified.

One of the confirmed bugs, in CSV import, changes billing seat counts. That touches money, so the team adds a 13th scripted test. Exploration found it; a script now guards it.

The result: the release gate is trusted again because it is small and deterministic, and breadth is covered without maintaining 128 brittle scripts.

Common mistakes (including ones no tool fixes)

Most failed QA programs fail on process and environment, not on the choice of tool. Watch for these:

• Replacing the whole scripted suite with agents. You lose the deterministic release gate and can't prove checkout was tested identically before a release.
• Keeping every legacy script out of fear. A flaky suite nobody trusts is worse than a small one everyone trusts. Quarantine flaky tests and delete what no one would miss.
• Filing agent findings without reproduction or replay review. Engineers learn to ignore QA tickets, and the program dies within weeks.
• Vague journey goals. "Test the dashboard" gives agents and humans nothing to judge against. Each journey needs a start page and a stated expected outcome. This is a spec problem, and no tool fixes it.
• Broken test environments and data. Shared staging data, expired seeded accounts, and third-party sandboxes that go down cause failures in scripts and agents alike. No QA product, ours included, repairs your environment.
• Running agents on every commit. Cost scales with runs, and per-commit exploration rarely finds more than nightly runs.
• Treating E2E as a substitute for unit and integration tests. Per the test pyramid (Mike Cohn, popularized by Martin Fowler), proration math belongs in fast, exact unit tests, not only in a browser test.
• Never promoting findings. A confirmed money-path bug that doesn't become a scripted test can regress silently.
• Assuming platform coverage. Check whether a tool supports your surface (web, native mobile, desktop). Native mobile scripting typically needs something like Appium.

When you need tooling, and when scripts must stay

Manual exploration stops scaling at roughly 8 to 10 journeys changing every sprint, or when you ship more than weekly. One person running charter-based sessions can cover a handful of journeys per release. Past that point, manual exploration either gets skipped or becomes someone's full-time job. That is where automated agent exploration earns its cost.

Regulated flows stay scripted, full stop. Payments under compliance review, healthcare data access, financial calculations, and consent capture need audit-identical, repeatable runs. Agent exploration can supplement them but should never be the evidence of record.

Other signs you need tooling: findings get lost in Slack threads, nobody can show a reproduction, or bugs get fixed without anyone connecting them to the customer complaints that surfaced them.

1. BuildBetter QA Agent — Best for verified AI exploration

BuildBetter QA Agent tests your app before each release and turns reproduced bugs into work for your team. You find it under Build > QA Agent. AI testers explore defined test flows, each with a start page and a description of what it tests, and admins choose which journeys are included. Agents also check layout and retry suspected bugs from a clean start.

• Triggers: nightly, on a GitHub release, or on demand with Run now. You review selected flows and credit limits before starting; each run uses credits.
• Results: completed runs separate Verified bugs from Needs a look. Each finding lists steps, expected result, actual result, and how often it reproduced from a clean start.
• Evidence: watch the replay (exploration, repro 1, repro 2) alongside the recorded agent actions before deciding.
• Handoff: File to Inbox sends the bug into BuildBetter Inbox/Triage, with Linear and Jira; Send to hands the same brief to a coding agent.

This mirrors Step 6 of the method: reproduce, read, watch, classify, act. The other advantage is location. QA findings sit in the same workspace as your customer calls, support tickets, and signals, so a bug and the customer complaints about it are side by side when you prioritize. Teams pairing QA with research can also use Screens, which offers a screen library, captured sessions with replay and friction markers, and flows that synthetic personas can test, alongside AI-moderated user interviews.

Every plan includes unlimited seats, usage-based credits are published at buildbetter.ai/pricing, and the platform is SOC 2 Type II, HIPAA, and GDPR compliant.

2. Playwright — Best for the scripted release gate

Playwright is the default open-source choice for deterministic, cross-browser web E2E tests that gate releases. It gives you explicit assertions, auto-waiting that cuts a share of async flakiness, and traces for debugging failures. Use it for the 10 to 20 money-path tests your decision table marks as scripted. For faster authoring and maintenance, see our Playwright test generation guide.

3. Open-source agent frameworks — Best for building your own harness

Stagehand, browser-use, and Skyvern let you build your own exploration agent. The trade-off is ownership: you write the prompts, the reproduction logic, evidence capture, and the triage workflow yourself. That is reasonable for teams with spare platform engineering time and a specific need.

One caveat applies to every option above: if your product is primarily native mobile, or you need audit-grade evidence, your scripted framework (Playwright for web, Appium for mobile) remains the core investment.

FAQ

Can AI exploratory testing replace scripted E2E tests?

No. AI exploratory testing finds bugs you did not anticipate, but it is non-deterministic, so it cannot prove that the same critical path was checked identically before every release. Keep scripted tests for revenue and compliance paths, and use agents for breadth and new features.

Which flows should stay as scripted tests?

Keep a flow scripted when breaking it costs money or data, when you can write its exact expected result, and when it changes rarely. Regulated flows that need audit-identical runs (payments under compliance review, healthcare data access, financial calculations, consent capture) should always stay scripted.

How do I know if an AI agent's bug report is real?

Treat a finding as verified only if it reproduces from a clean start, its expected result matches the spec, and the recording or replay shows the failure. Anything that reproduces inconsistently is a "needs a look" item, not a confirmed bug.

How often should AI exploratory tests run?

Run them nightly or on release candidates rather than on every commit, because each agent run has a cost. Scripted tests are cheap per run and should run on every pull request.

What is the best split between scripted and AI exploratory testing?

Most teams land on a small, trusted scripted suite for money paths and release gates, plus scheduled agent exploration across the remaining user journeys. Confirmed bugs on critical paths are then promoted into new scripted tests.

Do I need a tool to start exploratory testing?

No. A person running time-boxed sessions with a written charter (goal, start page, what to probe) is doing exploratory testing. Automation becomes worthwhile once more journeys change each sprint than one person can cover per release.

Sources and further reading

• Exploratory testing (Wikipedia)
• Test oracle (Wikipedia)
• Playwright official documentation
• Martin Fowler, "Test Pyramid"
• John Micco, "Flaky Tests at Google and How We Mitigate Them," Google Testing Blog, 2016
• Luo, Hariri, Eloussi, Marinov, "An Empirical Analysis of Flaky Tests," ACM FSE 2014
• Barr, Harman, McMinn, Shahbaz, Yoo, "The Oracle Problem in Software Testing: A Survey," IEEE TSE 2015
• Zhou et al., "WebArena: A Realistic Web Environment for Building Autonomous Agents," 2023 (arXiv 2307.13854)
• Playwright Test Generation with AI: Complete Guide (2026)
• What Are Synthetic Personas?
• How to Run AI-Moderated User Interviews

Make churn optional.

Bugs that reach customers become support tickets, then complaints on calls, then churn risk. BuildBetter puts QA findings in the same workspace as those calls, tickets, and signals, so your team can verify a bug, see who it affects, and ship the fix with full context. Make churn optional. Book a demo.