Lesson 8 of 12 · 3 min read

How do you run a design sprint for an AI product?

Keep the five days, and add the questions a classic sprint never had to ask: is the data there, what role does the AI play, where does its authority stop, and what would make a careful person trust it. Those decisions shape the prototype more than any sketch.

Diagram: a five-day sprint with an autonomy boundary, where the AI may draft and suggest and people send and commit, and a Friday test with one answer wrong on purpose.
The sprint adds a line where the AI’s authority stops. On Friday, one answer is wrong on purpose, to see who catches it.

A classic sprint assumes the software does what it’s told

In a usual sprint, the prototype’s behavior is whatever the team drew. With AI in the product, behavior depends on data the team may not have, and it will sometimes be wrong.

Bansi Mehta shared the workbook her team at Koru UX, a design firm, uses with healthcare clients, where a wrong answer has a patient on the other end. It adds eight tools to the week.

The workbook’s eight tools, in the order it lists them
  1. Problema canvas of four questions
  2. Datawhat exists, and whether it’s usable
  3. Rolewhat the AI does, and where its authority stops
  4. Decisionsprompts across six dimensions
  5. Patternsseven interaction cards to sketch from
  6. Trusta sixteen-question checklist
  7. Metricsseven categories, each with a threshold
  8. Feedbackhow use improves the next answer

Draw the boundary before anyone sketches

Of the eight, I’d draw the autonomy boundary on day one, because every sketch depends on it.

In Dispatch Cockpit, my prototype for truck dispatchers on made-up data, the line came before the screen: it recommends a truck and never assigns one. Drivers short of legal hours are ruled out before any ranking.

A caution-outlined card from Dispatch Cockpit: T-114, Sam Bergstrom, can’t take this load. Needs 5.5 hours, has 3.2 left. The hours-of-service limit, not a score.
From my work Dispatch Cockpit. A truck refused on the legal driving limit, before any ranking. See Dispatch Cockpit
What a classic sprint asks, set beside what an AI sprint adds

A classic sprint asks

  • What’s the problem?
  • What could the screen be?
  • Can people use it?

An AI sprint also asks

  • Is the data there to answer it?
  • What may the AI do alone?
  • Do people trust it the right amount?

On Friday, test the wrong answer too

A sprint prototype usually shows only what goes right. So put one wrong recommendation in Friday’s prototype, and don’t tell the participants. The test counts how many catch it.

For the cockpit, I’d plant a top truck that looks right, but its driver’s hours were read 45 minutes ago.

A dispatcher who checks the hours passes on it. One who trusts the screen assigns a driver who may be out of hours.

Count the other direction too. Someone who throws out a good recommendation and redoes the work by hand has also failed the task.

Four things a participant can do with a recommendation

When the AI is right

  • Accepts it: the product worked
  • Rejects it: too little trust, and the work gets done twice

When the AI is wrong

  • Catches it: the product worked
  • Accepts it: too much trust, and the error ships

The cockpit hasn’t been in front of a working dispatcher, so I can’t say how many would catch it. That’s the session I’d run first.

If nobody in the test catches the wrong answer, find out why. The prototype may have taught them to stop looking.

Check yourself

Could your team say, in one sentence, what the AI may do without asking?

If not, that’s the first hour of the sprint. Every sketch after it is guessing at the answer.

Does your test plan include a moment where the AI is wrong on purpose?

Trust shows in what people do with a bad answer. Count who catches it.

The idea: Bansi Mehta, Koru UX, AI design sprint workbook. The example: Dispatch Cockpit, on synthetic data, on this site.