AI-Moderated Interviews vs Human: ustwo's 82-Interview Test (2027)

Product studio ustwo let an AI Moderator run 82 customer interviews over one weekend. The team did no recruiting, no scheduling and no interviewing, and the insight still came out shallower than human-led research. This guide covers what ustwo tested, where AI-moderated interviews hold up, where they fail, what they cost, and how lean teams can combine them with human interviews and the customer conversations they already record. We also explain where BuildBetter fits: it pulls depth from verified customer calls, which neither an AI moderator nor a synthetic user can supply. Every finding about ustwo is attributed to its own published post. Everything else is labelled as general practice or our recommendation.

The Story: ustwo Ran 82 AI-Moderated Interviews in a Weekend, and the Insight Was Shallower

ustwo ran 82 AI-moderated interviews in a single weekend and concluded that AI research tools save time and money but can reduce depth and quality of insight. Andy Brown, Senior Strategist at ustwo, published "Putting AI customer research tools to the test" on November 14, 2025. The team tried AI customer research tools on real client work, including the meal-planning service Soph's Plant Kitchen.

The AI Moderator handled the whole run: no recruitment back-and-forth, no scheduling, no team member asking questions. ustwo summed up the trade-off this way:

"In short, AI research tools save time and money, but can reduce depth and quality of insight."

ustwo is an agency, so these findings come from client projects, not an in-house product team.

What ustwo Actually Tested

ustwo tested two categories of AI research tooling: Synthetic Users and AI Moderators, and most of its testing went to AI Moderators. The two categories work in very different ways, and mixing them up is how teams end up trusting the wrong output.

• Synthetic Users are simulated participants. A large language model answers in the role of a customer. No real person is involved.
• AI Moderators are platforms that recruit real people, run the interview with an AI interviewer, and summarise the results. The participants are human. The moderator is not.

ustwo does not name the vendor products it tested. We won't guess, and you should be wary of any article that claims to know which platforms were involved.

Separately, ustwo built its own Synthetic Persona in ChatGPT, grounded in its own interview data. It used that persona only for lower-stakes design decisions, not for deciding who the product was for.

The central comparison was AI-moderated sessions against human-moderated research on the same kinds of client questions. That makes the test useful: it shows how the two approaches behave on real product questions, not on demo prompts. It also limits the test. It reflects ustwo's projects, clients and tools, not a controlled benchmark across the market.

One ground rule for the rest of this article: every ustwo finding comes from that one post. We have added no other figures to ustwo's results. Where we bring in outside concepts, such as social desirability bias, we label them as background.

The Questions ustwo Had to Answer

ustwo's test answers five practical questions that any team considering AI customer research tools will face. Here are the short answers. Each one gets a fuller section below.

Are AI-moderated interviews good enough for real product decisions?

They are useful for spotting patterns. ustwo kept humans in charge of defining target customers and core needs.

Where do AI moderators beat human researchers, and where do they fall short?

They win on speed and volume. They fall short on depth and on reframing the problem.

Should we trust synthetic users?

Only for lower-stakes decisions, and only when the persona is grounded in real interview data.

How do we catch fraudulent participants?

Assume some participants will misrepresent themselves. In ustwo's test, one claimed to be a director at a major energy firm, and the claim was false.

What does AI research tooling cost compared with a researcher?

Per ustwo, a few thousand dollars to implement, then about $20K+ a year, or pay-as-you-go.

AI-Moderated Interviews: Pros and Cons

AI-moderated interviews give you speed and scale. The price is depth. ustwo ran 82 interviews in one weekend without recruiting, scheduling or moderating, and found the tools "can reduce depth and quality of insight."

Pros

• Volume without logistics. The weekend run removed the recruitment back-and-forth, the calendar juggling and the hours of interviewing. For a lean team, that is the difference between running a study and skipping it.
• Possibly more candid feedback. Participants rated features lower with AI Moderators than with human moderators. ustwo suggests people feel less pressure to be polite to an AI.

Background, not a ustwo claim: this fits a known research effect called social desirability bias, where people give answers they think the other person wants to hear. A human interviewer, especially one who looks like they built the product, can pull ratings up.

Cons

• Summaries answer the question as asked. One tool reported which features people preferred. It did not say who the "first best user" was. Ask a narrow question and you get a confident answer to that narrow question.
• The tools don't reframe the problem. A second user group surfaced only through team discussion. In ustwo's words, AI "can surface patterns but can't yet challenge or expand thinking."
• Fraud risk. One interviewee falsely claimed to be a director at a major energy firm. With no human in the session, that kind of claim is easier to miss.

Do this month

Draw a two-column table for your next research round. Put questions where volume matters (reactions to specific features, preference between options) in the left column. Put questions where depth matters (who the user is, why they would switch, what they do today) in the right column. Send only the left column to AI moderation.

AI Interviewer vs Human Moderator: Which Should Run Your Research?

Use human moderators to define who your target customer is and what they need. Use AI interviewers once those are settled and you need breadth. That is the split ustwo recommends for early product development.

In ustwo's projects, AI ran the volume work. The insight that changed direction, the discovery of a second user group, came from humans talking through the findings together. The AI output never flagged it.

The "first best user" gap

Knowing which features people like is different from knowing who to build for first. The first best user is the segment whose need is sharpest and who will adopt early. Finding that person means reading across interviews, noticing that three participants describe the same workaround in different words, and asking what they have in common. ustwo's tool output on feature preferences didn't do that. A summary that averages 82 people's preferences can hide the 12 who matter most.

The honesty trade-off

If participants really are more candid with an AI, then AI-moderated scores run lower for reasons unrelated to the product. Don't put human-moderated and AI-moderated ratings in the same spreadsheet and compare them. Treat each moderation type as its own baseline.

A plan for teams without a research department

1. Run five to eight human interviews to define the user and the core need.
2. Write a one-page hypothesis: who the user is, what they're trying to do, what they use today, and what would make them switch.
3. Run AI-moderated sessions to test that hypothesis at scale.
4. Hold a team synthesis session after the AI study. Read raw transcripts, not only the summary, and ask what the summary leaves out.

Step four is the one teams skip. In ustwo's case, it was the step that found the second user group.

Are Synthetic Users Reliable for User Research?

Synthetic users are not reliable on their own. ustwo used a ChatGPT-based Synthetic Persona grounded in real interview data, limited it to lower-stakes design decisions, and warned that LLMs can hallucinate or be too agreeable.

Definitions

• Synthetic Users are simulated participants generated by a language model.
• A grounded Synthetic Persona is built from your own interview transcripts and notes, so its answers draw on what real customers said.

A grounded persona compresses evidence you already hold. An ungrounded synthetic user is a guess presented as data.

Risk 1: Hallucination

Language models produce fluent, plausible text whether or not it is true. This is known as AI hallucination. A persona may describe a need no real customer ever expressed, and it will sound just as confident as when it is quoting your data.

Risk 2: Agreeableness

LLMs tend to go along with the person asking. Pitch your idea to a synthetic persona and it will often find reasons to like it. Research exists to disprove your assumptions, so a tool that leans toward validation works against that purpose.

Do this month

• Feed the persona only real, recent transcripts. Old research describes customers you may no longer serve.
• Restrict it to questions like copy wording, layout options or the order of onboarding steps.
• Never use it to decide who your target customer is.
• Before acting on any answer, ask the persona which source interview supports it. Then open the transcript and confirm the quote exists. If you can't find it, discard the answer.

AI User Research Tools Tested: What the Categories Can and Can't Do

ustwo found AI Moderators good at running and summarising many real-human interviews quickly, and weaker at identifying the right user or challenging the team's assumptions. That pattern should guide which tools you pick and what you use them for.

Caveat: ustwo did not name the products it tested. The tools below are this article's examples, not ustwo's. Verify features and pricing directly with each vendor.

1. BuildBetter — Best for depth from existing customer conversations

It analyses the customer conversations you already have: sales, CS and research calls, plus support tickets, Slack threads and surveys. It captures new calls with a free local recorder, BB Recorder, which works without a bot joining the meeting. Each piece of feedback is analysed individually for severity and business impact, then turned into PRDs or Linear and Jira tickets.

Honest caveat: it is not an AI moderator that recruits strangers. If you need fresh participants from outside your customer base, for example to test a new market, you still need a recruiting or moderation tool.

2. Outset

An AI-moderated interview platform that runs voice or video interviews with participants and summarises the responses. Check current features and pricing yourself.

3. Listen Labs

An AI interviewer platform built for running qualitative research at scale with real participants. Confirm panel sources and fraud controls before buying.

4. Great Question

A research-ops tool focused on recruiting participants, scheduling and managing a research panel. Useful when the bottleneck is finding people, not running sessions.

Comparison table

Selection checklist

• Can you see raw transcripts, not only summaries?
• Can you flag and exclude suspicious participants?
• Does the tool keep preference data separate from user-definition questions?
• Can you segment results by participant type?

How Much Do AI Interview Platforms Cost?

According to ustwo, AI Moderator platforms charge an implementation fee of a few thousand dollars, then about $20K+ a year, or offer a pay-as-you-go option. Those are the only figures ustwo published, and we won't add vendor price lists or per-interview estimates on top of them. ustwo did not name its vendors, so the numbers aren't tied to any specific product.

What the price buys, and what it doesn't

ustwo's verdict is that the tools save time and money. The 82-interview weekend shows where the savings come from: recruiting, scheduling and moderating hours mostly disappear.

The hidden cost is depth. If an AI study answers "which features" but not "who is this for," you will need follow-up human research to fill the gap. Budget for that from the start, so the tool doesn't look cheaper than it is.

Do this month

1. Estimate the hours your team spends per study: recruiting, scheduling, moderating and synthesis.
2. Multiply by the number of studies you run in a year.
3. Compare that against the annual fee plus the implementation cost.
4. Price in at least one human-led study per quarter, because AI moderation won't replace it.

Annual vs pay-as-you-go

• Pay-as-you-go suits teams running occasional studies, such as one per major release.
• An annual contract only makes sense with steady research volume. If you can't name the next four studies you would run, stay on pay-as-you-go.

How to Validate MVP Features With AI Interviews

Use AI interviews to measure reactions to features after humans have defined the target user. ustwo found the tools report feature preferences well but don't tell you who the first best user is. If you need a refresher on the concept, see the definition of a minimum viable product.

One finding changes how you read the results: in ustwo's work, AI feature ratings came out lower than in human-moderated sessions. Treat AI scores as their own baseline. A 3.4 from an AI session and a 3.9 from a human session do not mean the second group liked the feature more.

Step 1: Write down the target user and core need

Base this on human interviews. One paragraph: who they are, what job they're trying to get done, and what they use today.

Step 2: Screen participants against that definition

Write screener questions that match the user you defined, not a broad demographic. If the MVP is for operations leads at mid-size logistics firms, screen for that.

Step 3: Ask feature questions plus one open question

Ask about each feature, then add at least one open question about context and alternatives, such as "Walk me through the last time you dealt with this problem" or "What do you use today, and what annoys you about it?" Open questions give a hidden segment room to show up.

Step 4: Segment results by participant type

Split responses by role, company size or use case before reading the averages. ustwo's second user group was found through discussion. Segmenting makes that kind of group harder to average away.

Step 5: Challenge the summary as a team

Before deciding anything, hold a 45-minute session. Each person reads five raw transcripts and brings one thing the summary missed. Then decide what to build.

AI User Research Fraud: How to Catch Fake Participants

Expect user interview fraud in AI-moderated studies. In ustwo's test, one interviewee falsely claimed to be a director at a major energy firm. That is one documented example, not a fraud rate, but it is enough to justify building verification into every study.

Why it matters

When no human hosts the session, nobody notices that the "director" can't explain their own team's budget cycle. Senior-buyer claims are the most valuable data in B2B research and the most tempting to fake, because the incentives often pay more for senior profiles. One fake director can skew the insight you rely on for pricing or positioning.

Do this month

These steps are general good practice. ustwo's post does not describe them.

• Screen with role-specific knowledge. Ask questions that only someone in the role can answer well, such as how their team approves a new vendor or which metrics they report upward.
• Cross-check claimed titles where possible, for example against a professional profile.
• Review transcripts behind outlier answers. Vague, generic or contradictory responses are a warning sign.
• Exclude unverifiable high-seniority claims from analysis that drives decisions. Keep them in a separate bucket.

Lower-risk alternative

Use conversations with verified existing customers: sales calls, CS check-ins and support calls. You already know who is speaking, what account they belong to and what they pay. No one has a reason to invent a job title on a renewal call.

What to Copy From ustwo, and What Not to Copy

ustwo's test is a useful field report, not a universal rulebook. Copy the practices it surfaced, and read its limits carefully.

Copy

• Keep human-led research for defining target customers and core needs early in product development.
• Ground synthetic personas in real interview data and limit them to lower-stakes decisions.
• Run a team synthesis after AI studies. That is how ustwo found the second user group.

Don't copy blindly

• ustwo is an agency testing on client projects. Its set of tools, studies and markets may not match yours. A B2B team selling to procurement leads faces different recruiting and fraud problems than a consumer meal-planning service.
• ustwo did not name its vendors. You can't assume any specific platform behaves the way ustwo describes.
• The findings are qualitative observations from ustwo's own projects, not a controlled benchmark.
• The politeness explanation is a suggestion. ustwo offers it as a likely reason for lower AI ratings. It has not been proven as the cause.

Don't over-read the 82

82 interviews in a weekend measures speed. It says nothing about insight quality. ustwo's own conclusion was that the depth dropped.

Where Existing Customer Evidence Fits

ustwo found AI interviews faster but shallower. Recordings of the customer conversations you already have give you depth without recruiting or fraud risk. Most B2B teams sit on hundreds of sales, CS and research calls a quarter that nobody reads.

These conversations address two of the problems ustwo raised:

• Fraud: the speakers are verified customers tied to real accounts.
• Agreeableness: customers on a support call or renewal call are describing real problems in context, often while frustrated. Nobody is being polite to a prototype.

This is the gap BuildBetter fills. To be clear, ustwo did not use or mention it. The free local recorder, BB Recorder, captures calls without a bot joining. The platform then combines those recordings with support tickets, Slack threads and surveys, extracts structured signals with severity and business impact, and produces PRDs or tickets instead of a dashboard. Because it brings together internal team conversations and external customer feedback, the evidence behind a product decision stays traceable to the exact customer quote.

A suggested blend

FAQ

How many interviews did ustwo's AI moderator run?

82 over one weekend, according to ustwo's November 14, 2025 post by Andy Brown. The team did no recruiting, scheduling or moderating.

Did participants answer differently with an AI moderator?

Yes. Participants rated features lower with AI Moderators than with human moderators. ustwo suggests this is because people felt less pressure to be polite, though that explanation is a hypothesis, not a tested cause.

Can AI replace human user researchers?

Not according to ustwo. Its view is that AI "can surface patterns but can't yet challenge or expand thinking." Humans were needed to define the target customer and to spot a second user group.

What do AI moderator platforms cost?

Per ustwo, a few thousand dollars to implement, then about $20K+ per year, or a pay-as-you-go option. Verify current pricing directly with any vendor you evaluate.

Which AI tools did ustwo test?

ustwo did not name the vendors. It tested AI Moderator platforms and separately built its own Synthetic Persona in ChatGPT, grounded in its interview data.

Are synthetic personas safe to use?

Only for lower-stakes decisions and only when grounded in real data, because LLMs can hallucinate or be too agreeable. Always check that a persona's answer traces back to a real transcript.

Sources and Further Reading

• ustwo, Andy Brown, "Putting AI customer research tools to the test" (November 14, 2025)
• Wikipedia, Social-desirability bias
• Wikipedia, Minimum viable product
• Wikipedia, Hallucination (artificial intelligence)
• buildbetter.ai

Make churn optional.

AI moderators give you breadth. Your existing customer calls give you depth from people you've already verified. BuildBetter turns those calls, tickets and Slack threads into PRDs and tickets your team can ship, then closes the loop with the customers who asked.

Make churn optional. Book a demo