Lesson 7 of 12 · 3 min read

What does a requirements doc need once the product is AI?

Everything a good one already has, plus a plan for judging output that changes every time: the cases it must handle, the ones it must refuse, and the bar it has to clear before launch. The problem still goes on page one. The evaluation plan goes in before anyone writes a prompt.

Diagram: the product problem and success measure feed an evaluation plan of typical, edge, and refusal cases, agreed before prompting.
The test for the answer comes before the prompt, and refusals get as much room as the rest.

An AI feature gives a different answer every time, so “done” needs a test

A button either submits the form or it doesn’t. A summary, a recommendation, or a drafted reply is different on every run, and some runs will be wrong.

Moe Ali of Product Faculty shared a PRD template he credits to an OpenAI product leader. The template runs in four stages and twenty-four sections, with testing from the second stage.

The template’s four stages, and how many sections each one holds
  1. Discoveryis it worth it4 sectionsBusiness, feature, and user value, then a hypothesis for the AI’s part
  2. Designwhat good looks like4 sectionsFlows, screens, a first prompt, and the evaluation plan
  3. Developmake it hold9 sectionsModel, data, the evaluation set, and edge cases
  4. Deploykeep it honest7 sectionsRollout, risk, success metrics, and feedback

The habits of a good PRD still apply, first

This fall I read shared PRD templates from Amazon, Linear, Intercom, Figma, and others. The ones I’d reuse share three habits: the problem on page one, the metric before the feature, and a log of each decision with the options rejected.

What every good PRD asks, and what an AI one adds

Every good PRD

  • The problem, in the customer’s words
  • The metric that moves if it works
  • What it won’t do
  • Whether the number moved

An AI PRD adds

  • Typical, edge, and negative cases
  • What it may read, and how fresh
  • What it does when it’s unsure
  • Whether quality held after launch

Twenty-four sections is a menu, not a form

Commenters called the template waterfall: every decision made before anything is built. I think that’s true only if it’s filled in all at once.

Twenty-four sections, read two ways
  • A requirements card with nine identical fields, every one asked for at once.A form. Every field, filled in at once.
  • A requirements card with three sections, the evaluation plan marked as the one this risk calls for.A menu. The sections the risk calls for.

The best documents I read grow a little at each review, and the detail matches the risk. A two-day fix gets a paragraph; a feature that answers customers gets the evaluation plan in full.

The one section I’d never cut is the evaluation plan, because it writes down “good enough to ship” before anyone is invested in the answer.

An evaluation plan in four lines
  1. CasesTypical ones, edge ones, and ones it should refuse.
  2. RunsEach case several times, since one good answer can be luck.
  3. GradeWhat the customer would receive, not the AI’s rating of itself.
  4. BarWhat share must pass, and which cases can never fail.

Write the test for the answer before you write the prompt, the instructions the AI follows.

Check yourself

For your AI feature, what is one input it should refuse?

If nobody can name one, the evaluation set has no negative cases yet, and the feature will meet its first one in production.

Was the bar for launch written before or after the first demo?

A bar set after a good demo tends to fit the demo. Set it while nobody has seen an answer yet.

The idea: an AI PRD template shared by Moe Ali, Product Faculty. The example: the company PRD templates I read this fall.