Writing · 5 min read

How should you test a product now that AI can help?

Let agents find what’s broken, let people show you what they misunderstand, and let the numbers say how often. AI makes all three cheaper, but it can’t stand in for a person using the thing for real.

A sage lawn seen from straight above, a cream paved walk turning a sharp right angle across it, and one solid black worn path cutting straight across the corner of the grass.

Key idea 1 of 5

An agent can find what’s broken, not what’s misunderstood

When I built Lucy Learns, a dog-training app for my own household, the first version listened for spoken commands. The AI I built it with didn’t see a problem. Neither did I.

Lucy did. Say “next step” brightly at a phone and you’re using the voice she’s being taught is for her. A couple of real sessions in the hallway found it, and listening came out.

That’s the line I’d draw. An agent can walk every path of a build in minutes and tell you where it breaks. It can’t tell you what a person, or a dog, will make of it.

What to ask an agent, and what to ask a person

An agent

  • Every path, on every build
  • Dead ends, broken states, contrast
  • Claims the page can’t back up

A person

  • What they thought would happen
  • Where they hesitated, and why
  • Whether it fits the room they’re in

The agents earn their place on that left column. I gave AI readers my Agent Review case study as screenshots and visible text only, each playing a different reader.

The one playing a skeptical engineering lead checked my claims against the experiment record and found some I couldn’t support. I’d read that page a dozen times and missed every one.

Use agents to check the work, and people to find out what it means.

Key idea 2 of 5

A synthetic user agrees with everything

The tempting next step is to ask an AI to play the customer. Why recruit anyone, if a model can say what a busy nurse would think?

Nielsen Norman Group tried it against studies they’d already run. Asked about online courses, a synthetic user said, “Yes, I completed all the courses I mentioned.” A real participant had finished three of seven.

Shown new concepts, the synthetic users liked nearly all of them. A test where every idea passes isn’t a test.

I still use them, as rehearsal. A model is a good partner for piloting an interview script or learning a field’s vocabulary before a first call. Whatever it says goes down as a question for a real person.

A synthetic user tells you what people usually say, not what this person does.

Key idea 3 of 5

Watch a few people use the real thing, where they’d use it

Lucy’s second version turned the steps on a timer. Then she went to a training boot camp, we loaded its handouts into the app, and ran real sessions.

The timer didn’t hold up. When something goes wrong with a dog, the session pauses, and a clock doesn’t. No review would have found that, because nobody reviewing it was holding a leash.

That’s still the most valuable hour in product design: a few people, real tasks, in the place they’d do them. What AI changes is everything around the hour.

What AI can take off the hour around a session
  1. Beforedrafting the script, rehearsing it, finding people
  2. Duringa transcript, so I watch the person, not my notes
  3. Aftera first pass at grouping what was said

I watch the recordings myself anyway. A summary keeps what people said. It drops the long pause before they said it, and that pause is usually the finding.

AI can shrink the work around a session. It can’t sit in it for you.

The same sage lawn from straight above, the cream walk now running in one long diagonal across it where the shortcut had worn, the old right-angle corner left as a thin dashed outline in the grass, and one small butter-yellow flag planted beside the new walk.
Watch where people actually go, then build there: the walk follows the line they wore, not the one that was drawn.

Key idea 4 of 5

Numbers say how often, and people say why

Qualitative testing tells you what’s wrong. Analytics tell you how often, and for whom. You need both, and AI has made the second much easier to ask.

You can ask a question of the data in a sentence now, or have a model flag the step where people drop off. That makes it easy to watch more numbers than anyone can reason about.

What each kind of feedback is honest about

The numbers

  • How many people finished the task
  • Where they dropped off
  • Whether a change moved it

The people

  • What they were trying to do
  • What they expected instead
  • Whether they’d choose it again

So I pick a few numbers tied to a task, and when one moves I go watch the sessions behind it. For an AI feature, I’d count how often people catch a bad suggestion, not how often they accept one.

A number tells you where to look. A person tells you what you’re looking at.

Key idea 5 of 5

Put each kind of feedback where it’s honest

Put together, it’s a loop that runs at three speeds:

Three speeds of feedback, and what each one is for
  1. Every buildagents walk the paths and check the claims
  2. Every week or twoa few people try real tasks
  3. All the timea few numbers, and a session when one moves

I should say where I stand on my own work. Agent Review is not yet tested with users, and Dispatch Cockpit hasn’t been in front of a working dispatcher. Both pages say what I believe, not what I’ve watched.

The agents have made those pages more honest. Only people can tell me whether they’re right. More in What should a working prototype actually prove?

An agent can tell you the door is locked. Only a person can tell you they didn’t want to go in.