How to Run AI-Moderated User Interviews: A Practical Guide (2026)

You need 40 customer interviews before quarterly planning. You have time to run 6. This guide covers the other 34. It lays out a method for AI-moderated user interviews that a product manager, CS lead or founder can run this week with a recorded call, a shared doc and a spreadsheet. It also covers when tooling such as BuildBetter starts to pay for itself, and when you should skip the AI and talk to people yourself. The method is old: semi-structured interviewing, story-based questions and thematic analysis. What has changed is that software can now ask the follow-up questions for you.

What Is an AI-Moderated User Interview?

An AI-moderated user interview is a one-on-one research session where software asks your prepared questions by voice or on video, listens to the answer, and decides in real time whether to ask a follow-up. It produces a recording and a transcript you analyze afterward.

Structurally, it is a semi-structured interview: a fixed core of open questions, with follow-ups that adapt to each answer. Sessions are asynchronous. Participants take the interview when it suits them, which removes calendar coordination, time-zone conflicts and most no-shows.

How it differs from three similar methods

• Open-text survey: collects whatever the respondent types. It cannot ask “Can you give me an example?” An AI interviewer can. In a peer-reviewed study of roughly 600 participants, Xiao et al. (2020) found chatbot-led conversational surveys produced answers that were more informative, specific and clear than the same questions in a standard web form.
• Unmoderated usability test: records someone completing tasks and measures success, time and errors. It answers “Can they do it?” An interview answers “What happened, and why?”
• Synthetic persona interview: simulates responses from existing data or a language model. No real person takes part, so the output is a hypothesis, not evidence.

Who does what

The human owns four things: the decision the research informs, the questions, who gets recruited, and the judgment about what the answers mean. The AI asks the questions the same way every time, runs many sessions at once, probes thin answers to a depth you set, and asks interview 37 with the same attention as interview 1.

For a public example at scale, see Anthropic's 1,250-interview study, which used an AI interviewer with a general workforce sample plus creative and scientist subsamples.

Why Customer Interviews Break Down (and What AI Moderation Does and Doesn't Fix)

Most teams stop at 5-8 interviews because the time cost of live interviewing grows faster than anyone budgets for. Run the numbers. Forty interviews at 30 minutes each is 20 hours of talking. Add about 15 minutes of scheduling back-and-forth per call (10 hours), rebooking the 15-20% who no-show, writing notes, and rewatching recordings to pull quotes. Synthesis alone often equals the talking time. You end up at 50-60 hours, and a PM with 10 free hours a week needs five or six weeks to get through it. Planning doesn't wait that long.

Four failure modes

• Sample too small. Guest, Bunce and Johnson (2006) found basic meta-themes present by interview 6, with 92% of codes identified by interview 12. Six interviews surface strong themes. They cannot tell you whether a theme belongs to one segment or to all of them. Nielsen's “test with 5 users” rule is often quoted here, but it applies to finding usability problems in a design. It does not apply to understanding needs across segments.
• Interviewer drift. By the tenth live call, questions get reworded, leading follow-ups creep in, and sessions stop being comparable.
• Scheduling bias. The customers who will book a 30-minute call are rarely the ones who churned quietly. Async sessions widen who will take part.
• Synthesis debt. Recordings pile up unwatched, and the “findings” become whatever the PM remembers from the last two calls.

Scale also changes depth. Hennink, Kaiser and Marconi (2017) found that hearing all the issues (code saturation) took about 9 interviews, but understanding them (meaning saturation) took 16-24. Early interviews can sound repetitive while you still don't know why the problems happen.

What AI moderation fixes:

• Calendar time and no-shows
• Question consistency across dozens of sessions
• Reach into customers who won't book a call
• Probing on thin answers, every time

What it leaves to you:

• A clear research question tied to a decision
• A sample that represents the people the decision affects
• Neutral, non-leading questions
• A team willing to change its plan based on what it hears

The Method: How to Run an AI-Moderated Interview Study in 10 Steps (Plus a Live Follow-Up)

A team with a spreadsheet and no budget can run this method today. Every step works with a recorded video call, a shared doc and a spreadsheet. A tool makes the steps faster. It does not change them.

Step 1: Write the decision first

Use one sentence: “We will decide X by date Y; this study tells us whether to do A or B.” Erika Hall's point in Just Enough Research applies: research exists to reduce risk in a decision. If no decision changes based on the answers, stop here.

Step 2: Write 5-7 questions

Put one warm-up first. Ask about the past (“Tell me about the last time you…”) and avoid future hypotheticals (“Would you use…”). This is the core of Rob Fitzpatrick's The Mom Test and Teresa Torres's story-based interviewing. Each question should be open and neutral and should map to the decision. Use this template in your discussion guide:

Step 3: Set probing depth per question

Use three levels:

• None: factual or warm-up questions.
• Light: one clarifying follow-up, only if the answer is under a sentence or vague.
• Deep: up to 2-3 follow-ups asking for a specific example, a sequence of events, or what they did instead (laddering).

Steve Portigal's observation in Interviewing Users is that a first answer is usually a summary and the specifics come in the second and third. Set Deep only on the one or two questions closest to the decision. Unlimited probing tires participants, raises drop-off and pads transcripts with restated answers.

Step 4: Set a time limit

Use 10-15 minutes for 5-7 questions. Shorter sessions get more completions. A 30-minute AI session usually means you have too many questions.

Step 5: Choose audio or video

Step 6: Recruit

You have two sources. Your own customers: pull a list from your CRM or support tool into a spreadsheet, filter to the segment the decision is about, and invite by email or link. A research panel: use one when you need non-customers or lost prospects, and screen with 2-3 qualifying questions. Recruit about 1.5-2× your target, because not everyone completes.

Step 7: Set incentives

Match the reward to the time asked and the audience. Use gift cards for panels. For your own customers, offer a gift card or a charity donation. State the reward and the time commitment in the invite. Incentives matter less for engaged customers than for cold panel participants.

Step 8: Pilot with 3 people

Run the full study with 3 participants, ideally colleagues outside the product team plus one real customer. Read every transcript. Check three things: whether follow-ups probed where you wanted, whether any question was misunderstood, and whether sessions ran over time. Fix the guide, then lock it so later responses stay comparable.

Step 9: Launch and monitor

Send invites in batches. Read the first 5 completed transcripts within a day. Watch the completion rate and drop-off by question. A spike at one question usually means it is confusing or too personal.

Step 10: Read and verify

Read the transcripts, or a summary report if you have one. Draft themes, then check every theme against at least 3 verbatim quotes from different participants. Count how many participants support each theme. Note contradictions and outliers. Braun and Clarke's guidance on thematic analysis is worth remembering: themes don't “emerge.” A person constructs them and is accountable for them.

Closing step: Run 3-5 live interviews on what surprised you

Use the AI study to find the questions worth a human conversation, and spend your 6 hours of live time there.

Copyable checklist

1. Write the decision in one sentence, with a date.
2. Draft one warm-up plus 5-7 past-behavior questions.
3. Set probing depth: None, Light or Deep.
4. Cap the session at 10-15 minutes.
5. Pick audio or video.
6. Recruit 1.5-2× your target from customers or a panel.
7. Set and state the incentive.
8. Pilot with 3 people, fix the guide, then lock it.
9. Launch in batches and read the first 5 transcripts within a day.
10. Verify each theme against 3+ quotes and count supporters.
11. Run 3-5 live interviews on the surprises.

Worked Example: A 40-Interview Study on Onboarding Friction

This is an illustrative walkthrough, not a customer case study. The numbers show the reasoning at each step.

The decision: “Before Q1 planning, decide whether to rebuild the setup wizard or invest in guided templates.”

The questions:

1. Tell me about the last time you set up a new workspace. (warm-up, None)
2. Where did you get stuck? (Deep)
3. What did you do when you got stuck? (Deep)
4. What did you expect to happen on the Connect Data screen? (Light)
5. What made you almost give up? (Light)
6. What would you tell a colleague starting today? (Light)

Format: audio, 12-minute limit.

Recruiting: 70 invites to accounts created in the last 90 days, split between activated and non-activated, plus a panel screener for people who had evaluated similar tools. The target was 40 completes, with a gift card for each completed session. The study got 44 completions. Four were discarded as too short to use.

Pilot: Three sessions showed that question 4, originally “step 3,” was ambiguous, so the team reworded it to name the screen. One Deep question ran long, so they capped its follow-ups at 2.

What the probing caught: One participant said, “The integration part was confusing.” The AI asked which integration and what they expected to happen. The answer showed they didn't know they needed admin permissions in the other tool. A survey text box would have recorded “confusing” and stopped there.

Top themes from the report (illustrative):

• Permission and admin-access blockers: 17 of 40. “I had to wait two days for IT to give me access, and by then I'd forgotten why I signed up.”
• Unclear value before data appears: 14 of 40. “It was an empty dashboard. I couldn't tell what it would look like with my stuff in it.”
• Template expectations borrowed from other tools: 11 of 40. “Every other tool I use starts me with a sample project.”

Verification: The summary also listed “wizard has too many steps” as a theme. When the team checked quotes, it rested on 2 participants who spoke at length. They downgraded it to an outlier, and that changed the decision.

The decision: Don't rebuild the wizard. Add a permissions pre-check and a sample-data template. Then run 4 live interviews with non-activated accounts to test the pre-check concept.

Time spent: about 3 hours writing the guide, 1.5 piloting, 1 monitoring, 5 reading transcripts, 2 verifying themes and 3 on live calls. That's roughly 16 hours of PM time, compared with 50-60 for 40 live calls, and none of it went to filling calendar slots.

Common Mistakes with AI-Moderated Interviews

Most failed AI interview studies fail for the same reasons live studies do, repeated across more sessions.

• Starting without a decision. You get interesting transcripts and no action.
• Asking hypotheticals. “Would you pay for…” produces confident, useless answers. People are poor predictors of their own future behavior, and an AI interviewer doesn't change that.
• Over-probing. Setting every question to Deep makes sessions feel like interrogations and raises drop-off.
• Too many questions. Ten questions in 15 minutes leaves no room for follow-ups.
• Skipping the pilot. Ambiguous wording spreads across 40 sessions instead of 3.
• Editing questions mid-study without tracking versions. Responses before and after the edit can't be compared.
• Trusting the summary without reading quotes. AI summaries, any vendor's and ours included, can overweight articulate participants and smooth over contradictions. No tool removes the need to check themes against verbatim evidence.
• Recruiting only happy, engaged customers. Panel or link recruiting can't fix a badly chosen sampling frame, and no product solves this for you.
• Treating synthetic personas as real participants. They're useful for first reads, not as evidence.
• Never closing the loop. Participants who never hear what changed are less likely to say yes next time.

When AI Moderation Is the Wrong Choice

AI moderation is the wrong choice when the human relationship, or a human's ability to notice something, is part of what the research needs.

• Sensitive topics: health, finances, layoffs, grief, or anything where a participant may need someone to notice distress. Use a trained human moderator.
• Executive and strategic-account interviews: senior buyers expect a person, and the relationship is part of the value.
• When rapport is the point: early design-partner conversations, churn-save calls and co-creation sessions.
• Highly exploratory research: if you don't yet know the questions, run 5-6 live interviews first, then scale with AI.
• Usability tasks that need real-time observation and intervention (see usability testing).

When You Need Tooling (and Which Tools to Use)

A spreadsheet plus recorded calls works up to roughly 8-12 interviews per study. Past about 20, scheduling, transcript reading and quote-matching take more PM time than the research is worth. Past one study a month, you also need versioned questions and a place where findings persist after the slide deck is forgotten.

At that point, tooling should run sessions asynchronously, probe consistently, handle recruiting and rewards, produce transcripts and evidence-linked themes, and connect findings to the rest of your customer data.

1. BuildBetter — Best for AI-Moderated Interviews Connected to Your Customer Data

Interview and survey your customers. AI asks the follow-ups. BuildBetter runs audio interviews in the browser or video interviews with an on-screen AI interviewer. You set the questions, time limit, language and probing depth. It asks real follow-ups instead of reading the next line, and every interview lands as a recording with a transcript and extracted signals.

• Recruiting: invite your own customers by email or link. Or use signal-led targeting: describe who you want in plain language, then approve each matched person, with their evidence, before anything is sent. You can also bring in Prolific, Respondent, User Interviews, CloudResearch Connect, Dynata, or any panel that sends participants to a link. Gift-card rewards run through Tremendous and are set per study.
• Outputs: study reports with an executive summary, key findings, and themes backed by participant quotes. Reports are saved to Knowledge next to your calls, tickets and Slack signals. BoB, BuildBetter's assistant, can draft a study (questions, audience, rewards) for your review. Questions are versioned and immutable once sent. The platform is SOC 2 Type II, HIPAA and GDPR compliant.
• AI Surveys, for when an interview is overkill: send by branded email, public link or in-app widget. A model decides whether one clarifying follow-up is worth asking, and often decides it isn't. Credit pricing is transparent: a completed email response with two follow-ups and three signals costs about 26 credits, roughly $0.026. Previews are never billed, and seats are unlimited. For large open-ended sets, see tools to analyze thousands of open-ended survey responses.
• Synthetic Personas: personas built from your real signals, with citations, for pressure-testing a discussion guide or concept before you recruit. They are for first reads and do not replace real customers. See how ustwo tested AI-moderated interviews and synthetic personas.

“I've never used a tool that actually connects the dots across all my platforms. BuildBetter does that, and it makes me so happy.” — Agnes Abel, Typeform

A caveat: if you need dedicated research-repository workflows such as highlight reels and deep tagging taxonomies, mature repository tools go further there. BuildBetter's strength is connecting interview evidence to calls, tickets and Slack.

2. A DIY Stack — Best for Teams Not Ready to Buy

The full method in this guide works without a dedicated platform. Use recorded video calls, a recruiting panel such as Prolific or User Interviews directly, gift cards through Tremendous, and a spreadsheet for the theme-to-quote matrix: one row per theme, one column per participant, with verbatim quotes in the cells. It's slower, and you'll moderate the sessions yourself, but every step from the decision sentence to the live follow-ups still applies.

FAQ: AI-Moderated User Interviews

How many AI-moderated interviews do I need?

For one segment and one decision, 15-25 completed interviews usually reveal the main themes and much of their meaning. Saturation research (Hennink and Kaiser, 2022) puts the typical range at 9-17 for homogeneous groups. Run 40 or more when comparing segments, such as activated and non-activated accounts. Then run 3-5 live interviews on whatever surprised you.

How long should an AI-moderated interview be?

Aim for 10-15 minutes, with one warm-up plus 5-7 questions. Longer sessions reduce completion without adding proportionally more insight. If you need 30 minutes, you probably have too many questions.

Are AI-moderated interviews as good as human interviews?

They are better at consistency, scale and avoiding interviewer drift. They are worse at rapport, sensitive topics and improvising on unexpected threads. Early academic work (Xiao et al., 2020; Geiecke and Jaravel, 2024) shows AI-led follow-ups outperform open-text surveys and can approach typical human interviewer quality on structured topics. Use AI to find the patterns and humans to go deep on them.

Will participants talk honestly to an AI interviewer?

Often, yes. Research on self-administered modes and virtual interviewers shows people disclose more when they don't feel judged by a person. Candor depends on clear consent, a stated purpose, an honest disclosure that an AI is asking the questions, and a fair incentive.

What's the difference between AI-moderated interviews and synthetic personas?

AI-moderated interviews talk to real people. Synthetic personas simulate likely reactions from existing data. Personas are useful for pressure-testing a discussion guide before you recruit, but only real interviews count as evidence.

How do I make sure the AI summary is accurate?

Check every theme against at least 3 verbatim quotes from different participants. Count how many participants support it, and read a sample of full transcripts yourself. Downgrade any theme that rests on one or two vocal people.

Sources and Further Reading

• Semi-structured interview (Wikipedia)
• Thematic analysis (Wikipedia)
• Usability testing (Wikipedia)
• Anthropic AI Interviewer: 1,250 interviews
• ustwo tested AI-moderated interviews and synthetic personas
• BuildBetter AI Surveys and Interviews
• Guest, Bunce and Johnson (2006), “How Many Interviews Are Enough?”, Field Methods 18(1)
• Hennink and Kaiser (2022), “Sample sizes for saturation in qualitative research,” Social Science and Medicine 292
• Hennink, Kaiser and Marconi (2017), “Code Saturation Versus Meaning Saturation,” Qualitative Health Research 27(4)
• Xiao et al. (2020), “Tell Me About Yourself,” ACM Transactions on Computer-Human Interaction
• Rob Fitzpatrick, The Mom Test; Teresa Torres, Continuous Discovery Habits; Steve Portigal, Interviewing Users; Erika Hall, Just Enough Research

Make churn optional.

Your next planning cycle shouldn't rest on six interviews and a hunch. BuildBetter runs AI-moderated interviews and surveys, links every theme to participant quotes, and stores the findings next to your calls, tickets and Slack threads, so the evidence is still there when you write the PRD. Make churn optional. Book a demo.