How to Reproduce Bugs With AI: Clean Repros & Replays (2026)

A customer reports a bug. An engineer tries it once, it works on their machine, and the ticket closes as “cannot reproduce.” Seven days later the same customer reports it again, angrier, and CS escalates. “Cannot reproduce” usually means “not reproduced under my conditions.” It does not mean the bug is absent, and every round trip costs customer trust and engineering hours. This guide from the BuildBetter team gives you a repeatable method for turning a vague report into a verified bug, with a repro rate, a replay, and customer evidence, before anyone is assigned. A spreadsheet and a screen recorder are enough. AI tooling comes in only at the end, for teams that outgrow the manual version.

What Is Bug Reproduction (and What “AI-Assisted” Adds)?

Bug reproduction is the act of making a reported failure happen again, on purpose, from a known starting state, so you can observe it, record it, and prove a fix works. Without a repro you cannot verify a fix. You can only ship a change and hope the customer stops complaining.

Three terms carry the rest of this guide:

• Clean-start repro: each attempt begins from a fresh state. That means a new browser profile or incognito session, cleared cookies and storage (localStorage, sessionStorage, IndexedDB, service workers), a known test account, and a known build. Leftover state from the previous attempt can neither cause the bug nor hide it. Incognito windows opened from the same parent session can share state with each other, so a separate browser profile per attempt is the stricter option.
• Repro rate: the number of clean-start attempts that showed the failure divided by total attempts, for example 2 of 3. This turns “it's flaky” into a number you can triage on, compare before and after a fix, and track over time.
• Replay: a recording of an attempt, covering the screen plus the actions taken (clicks, inputs, and network calls where available). Someone who wasn't there can see exactly what happened. A plain screen recording is the minimum. Richer replays add DOM snapshots, console logs, and network requests.

What AI adds: an AI QA agent can explore a flow, retry a suspected bug several times from a clean start, and record each attempt. The human then reviews the evidence instead of doing the reruns.

The method is identical with or without AI. Steps to reproduce, expected vs actual result, a repro rate, and replays are the same deliverables either way. AI only changes who does the reruns.

Why Bugs Are Hard to Reproduce: What Actually Breaks

Most non-reproducible bugs are reproducible under conditions nobody has matched yet. Research backs this up: in a study of open-source and commercial projects, non-reproducible reports made up roughly 17% of all bug reports and stayed open longer than others, and many were later reproduced and fixed once more information arrived (Joorabchi, Mirzaaghaei and Mesbah, “Works For Me!”, MSR 2014). These are the usual causes:

• Hidden state. Cached data, a logged-in session, a feature flag, or a dismissed onboarding modal makes the engineer's path differ from the customer's. Looks like: the bug only appears for first-time users, and every engineer dismissed that modal months ago.
• Time and locale dependence. Timezone offsets, daylight saving transitions, end-of-month dates, and browser language settings make bugs appear only at certain hours or in certain regions. Looks like: invoices show the wrong month, but only on the last day of the month in Australia.
• Intermittency. Race conditions, slow network responses, and retry logic mean the bug shows up only some of the time. A single attempt is a coin flip. Some of these are true heisenbugs, which change behavior when you attach a debugger because the debugger changes the timing. Looks like: the save button double-submits on a slow connection but never on office Wi-Fi.
• Data dependence. The bug only triggers with the customer's record shape: a long name, an empty field, a large account, or legacy data your test account lacks. Looks like: the export crashes for an account with 40,000 contacts and works fine for one with 12.
• Environment drift. Staging differs from production in config, data volume, third-party integrations, or build version. Looks like: the payment step fails in production because staging uses a sandbox processor with different validation.
• Lossy reports. Customer reports arrive through tickets, calls, and Slack threads, paraphrased two or three times. Exact steps, timestamps, and what the customer expected get lost. Looks like: “the calendar is broken” reaches engineering with no time of day attached.
• Single-attempt culture. Most teams try once, and one failed attempt gets treated as proof of absence. Looks like: “works for me” closes the ticket, and the loop starts.

Single-attempt culture is the root of the problem, and the math shows why. If a bug occurs 30% of the time, one attempt misses it 70% of the time. Three attempts miss it 34% of the time (0.7³), and five miss it about 17% (0.7⁵). Even mature engineering orgs live with this: Google reported that about 16% of its tests showed some flakiness (Google Testing Blog, 2016). One pass is weak evidence.

The Clean-Start Repro Method: 6 Steps You Can Run Today

You need a spreadsheet, a screen recorder (the built-in OS recorder is fine), and 20–40 minutes per bug. No paid tools. Treat each bug as an experiment: hold everything constant, change one variable at a time, and keep the control run.

Step 1: Gather the evidence first

Before you touch the app, collect the original report: the ticket text, the customer's exact words from calls or chats, screenshots, the timestamp, the account, browser and OS, and the region or timezone. Convert the timestamp to the customer's local time. Write down what the customer expected to happen, in their own words. Time-of-day phrases like “after work,” “Monday morning,” or “end of month” are often the whole diagnosis.

Step 2: Write the hypothesis path

Draft numbered steps from a specific start page to the failure point. Then write one line for Expected result and one for Actual result (as reported). If you can't write the expected result, you don't yet understand the bug, so go back to the customer. Developers consistently rate steps to reproduce among the most useful parts of a bug report, while reporters find them the hardest to provide (Bettenburg et al., “What Makes a Good Bug Report?”, FSE 2008). This step closes that gap.

Step 3: Define and reset a clean start

Use a fresh browser profile or incognito window, cleared storage, a test account that matches the customer's plan and data shape, and the same build the customer used. Match timezone and locale to the customer's, using devtools sensor overrides or OS settings. Reset fully before every attempt. A clean start on a machine set to your own timezone is still the wrong experiment.

Step 4: Rerun the failing path at least 3 times, recording each attempt

Run 3 attempts minimum, and 5 if the first 3 disagree. Record the screen for every attempt, including the ones that pass, because a passing replay is evidence too. Log the clock time of each attempt.

Step 5: Record expected vs actual per attempt and count the repro rate

Write one row per attempt. If the rate is 0 of 3 or 0 of 5, change one variable at a time (timezone, data, browser, time of day) before closing. More reruns under the same conditions won't help: by the statistical “rule of three,” 0 failures in 3 attempts only tells you the true rate is probably below about 100%, which tells you nothing. “Cannot reproduce” is allowed only after you've varied the conditions the report suggests.

Step 6: Package the brief, then assign

Bundle the steps, expected vs actual, repro rate, links to every replay (the exploration plus each repro), the original customer evidence, and your best guess at the variable that matters. Only now assign an owner and priority. The engineer receives a verified bug, not a rumor.

The repro spreadsheet

Decision rules

• 3 of 3: deterministic bug. Assign immediately.
• 1–2 of 3: intermittent bug. Assign with the suspected variable and the replays that differ.
• 0 of 3, conditions varied: park it and return to the customer with specific questions.
• Never close on a single attempt.

Rerun from a clean start several times, record expected vs actual, count how often it reproduces, keep a replay of each attempt, attach the customer evidence, and only then assign it.

Worked Example: The Flaky Date-Picker Bug (2 of 3 Repros)

A timezone bug in a date picker shows how three timed, recorded attempts turn a closed ticket into a one-line fix. This is an illustrative example, not a customer case study.

The report

A customer in Seattle files a ticket saying the booking date picker “won't let me pick today” and defaults to tomorrow. A week earlier, an engineer tested at 10:00 a.m., saw today selected correctly, and closed it as cannot reproduce.

Steps 1–3: evidence, path, clean start

• Evidence: the ticket timestamp is 6:12 p.m. Pacific. On a support call the customer says, “it happens when I do bookings after work.” Standard plan, Chrome on macOS.
• Path: start at /bookings/new, open the date picker, observe the default selection. Expected: today's date is selected and selectable. Actual (reported): tomorrow is selected and today is disabled.
• Clean start: a new Chrome profile per attempt, a fresh standard-plan test account, browser timezone set to America/Los_Angeles (UTC−7 in late October, since US daylight saving time ends November 1, 2026), and the current production build.

Steps 4–5: results

Repro rate: 2 of 3.

The replay cracks it

Comparing the passing replay with the two failing ones shows the only difference is the clock. After 5:00 p.m. PDT, the UTC date has already rolled over to tomorrow. The recorded network call shows the “today” value being computed in UTC rather than the user's local timezone. This is a classic JavaScript pattern: new Date().toISOString().slice(0, 10) always returns the UTC date, so it reports tomorrow for anyone west of UTC in the evening (see MDN's Date reference).

Confirm the hypothesis

One extra attempt with the browser timezone set to UTC shows the correct date at any hour. A fixed clock at 11:30 p.m. local reproduces the bug every time. The bug is now deterministic under known conditions: a Bohrbug hiding inside what looked like a heisenbug.

The decision

Priority is high, because the bug blocks same-day bookings for every customer west of UTC in the evening. Assign to frontend with a brief containing the steps, the 2 of 3 repro rate, all three replays, the ticket, the call quote, and the suspected cause (“today” computed in UTC). Add an acceptance check: a test using Playwright's clock API with a fixed time of 11:30 p.m. local. Note that after November 1, Pacific time shifts to UTC−8 and the rollover moves to 4:00 p.m., so a second fixed-clock case after the DST change is cheap insurance.

The lesson: the original 10:00 a.m. attempt was a valid test of the wrong conditions. Three timed, recorded clean-start attempts plus the customer's “after work” remark did the diagnosis.

Common Bug Reproduction Mistakes (Including Ones Tools Won't Fix)

Most failed repros come from process gaps, and several of them survive any amount of automation.

• Closing after one attempt. A single pass is not evidence of absence. Require at least 3 clean-start attempts before “cannot reproduce.”
• Treating 1 of 3 as noise. A low repro rate signals hidden state, timing, or data dependence. Prioritize by impact, not by repro rate.
• A dirty start. Reusing a logged-in session or a test account that accumulates state between attempts makes results unreliable. Empirical work on flaky tests found async waits (~45%), concurrency (~20%), and test order dependency (~12%) to be the top root causes (Luo et al., FSE 2014). Order dependency is leftover state by another name.
• Leaving defaults alone. A clean start still uses your timezone, locale, and screen size unless you deliberately match the customer's. Automated tools have the same blind spot if nobody configures them.
• No expected result. “It's broken” is not a bug report. Without a written expected result, the team will argue about whether it's a bug or a feature request.
• Discarding passing replays. The passing attempt is often the control that reveals the variable, as the date-picker example shows.
• Separating the bug from the customer. When the engineer sees only a ticket ID, not the customer's words, complaint frequency, or account size, priority gets set blind.
• Privacy slips. Pasting raw customer data or PII into the brief or test account. Use redacted or synthetic equivalents.

Limits no tool removes. Some bugs depend on production data you can't copy into a test account. Some race conditions only appear under production load. Some depend on third-party systems you can't reset, and some live in native mobile builds rather than web apps. For these, you need logging, feature-flagged diagnostics, or a call with the customer, not more reruns.

When Manual Repro Stops Working

Manual repro is fine, and faster, for a deterministic crash with a stack trace. Open the trace, follow the line number, fix it. No agent needed. It also fits low volume: a handful of suspected bugs per release on a weekly or slower cadence sits comfortably in a spreadsheet.

These heuristics, not statistics, suggest you've outgrown it:

• Most incoming bugs are intermittent and need 3–5 timed reruns each.
• You ship daily or several times a week and can't re-walk every critical journey before each release.
• Your critical journeys outnumber what one person can walk and record in an afternoon.
• Evidence is scattered across call recordings, tickets, and Slack, and assembling the brief takes longer than the repro.
• Engineers routinely bounce bugs back for missing steps or replays.

1. BuildBetter QA Agent — Best for Verified Bugs Next to Customer Evidence

The BuildBetter QA Agent (Build > QA Agent) tests your app before each release, then turns reproduced bugs into work for your team. A team of AI testers explores test flows, checks layout, and retries suspected bugs from a clean start, which is the six-step method run by agents.

• Test flows: each flow has a start page and a description of what it tests. Admins choose which journeys are included. Runs happen nightly, on a GitHub release trigger, or via Run now. Each run uses credits, so review the selected flows and credit limits before starting.
• Findings: a completed run separates Verified bugs from Needs a look. Each finding lists steps, expected result, actual result, and Reproduced, meaning how often it happened from a clean start. Those are the same columns as the manual spreadsheet.
• Evidence and hand-off: watch the replay (exploration, repro 1, repro 2) alongside the recorded agent actions, and check the evidence before deciding what to fix. File to Inbox adds the bug to your team's triage queue, with Linear and Jira support. Send to hands the same brief to a coding agent.

Why one platform: BuildBetter keeps the QA Agent in the same workspace as your customer calls, tickets, and signals, so a bug and the customer complaints about it can live side by side. That addresses the “separating the bug from the customer” mistake directly. Related Screens features add a screen library, captured sessions with replay and friction markers, and flows that synthetic personas can test.

Every plan includes unlimited seats, with usage-based credits published at buildbetter.ai/pricing. The platform is SOC 2 Type II, HIPAA, and GDPR compliant.

Other Tools for Clean-Start Retries and Replays

2. Playwright (open source)

Isolated browser contexts give you clean starts, the Trace Viewer records DOM snapshots, actions, console, and network for each run, and the clock API pins time-dependent bugs like the date-picker example. To draft tests faster, see our guide to Playwright test generation with AI.

3. browser-use, Stagehand, and Skyvern (open source)

These agent frameworks drive a browser with an LLM. They suit teams willing to build their own retry, recording, and reporting layer on top.

4. rr (open source, Linux)

rr is a record-and-replay debugger for native and backend processes, where browser replays don't apply. Record a failing run once, then replay it deterministically as many times as you need.

Whatever you choose, Step 1 remains a human skill. For structured follow-up with customers who report bugs, see how to run AI-moderated user interviews.

FAQ: Reproducing Bugs With AI

How many times should I try to reproduce a bug before closing it?

At least 3 times from a clean start, and 5 if the first results disagree. Close as “cannot reproduce” only after you've also varied the conditions the report points to, such as timezone, locale, data shape, browser, or time of day. Probability explains why: a bug that occurs 30% of the time is missed by a single attempt 70% of the time.

What is a clean-start reproduction?

A clean-start reproduction is an attempt that begins from a fresh state, meaning a new browser profile or session, cleared storage and cookies, a known test account, and a known build. Leftover state from earlier attempts can't cause or hide the bug.

What does a repro rate like “2 of 3” mean?

The bug appeared in 2 of 3 clean-start attempts. It is real and intermittent, and the difference between the passing and failing attempts usually points to the cause.

Can AI reproduce bugs on its own?

An AI QA agent can explore a flow, retry a suspected bug several times from a clean start, and record replays of each attempt. A human should still review the evidence and decide what to fix and how to prioritize it.

What should a bug report include before it's assigned?

Steps from a specific start page, the expected result, the actual result, a repro rate from clean-start attempts, a replay of each attempt (including passing ones), the clean-start conditions used, the original customer evidence, and your best guess at the variable that matters.

When is manual bug reproduction enough?

Manual reproduction is enough for deterministic crashes with a stack trace and for low-volume, slower release cycles. Tooling pays off when most bugs are intermittent and you ship faster than you can rerun them by hand.

Sources and Further Reading

• Heisenbug (Wikipedia)
• Software bug (Wikipedia)
• Playwright Trace Viewer docs
• Playwright Clock API docs
• MDN JavaScript Date reference
• rr record-and-replay debugger
• Joorabchi, Mirzaaghaei and Mesbah, “Works For Me! Characterizing Non-Reproducible Bug Reports,” MSR 2014
• Bettenburg et al., “What Makes a Good Bug Report?”, FSE 2008
• Luo, Hariri, Eloussi and Marinov, “An Empirical Analysis of Flaky Tests,” FSE 2014
• John Micco, “Flaky Tests at Google and How We Mitigate Them,” Google Testing Blog, 2016
• QA Agent documentation: buildbetter.ai/university/qa-agent and docs.buildbetter.ai

Make Churn Optional

Every “cannot reproduce” that bounces back is a customer deciding whether to stay. BuildBetter puts customer calls, tickets, and Slack threads in the same workspace as AI-verified bug repros, so your team fixes the bugs customers are actually reporting, with the evidence attached.

Make churn optional. Book a demo.