Lesson 7 of 12 · 3 min read
What does a requirements doc need once the product is AI?
Everything a good one already has, plus a plan for judging output that changes every time: the cases it must handle, the ones it must refuse, and the bar it has to clear before launch. The problem still goes on page one. The evaluation plan goes in before anyone writes a prompt.
An AI feature gives a different answer every time, so “done” needs a test
A button either submits the form or it doesn’t. A summary, a recommendation, or a drafted reply is different on every run, and some runs will be wrong.
Moe Ali of Product Faculty shared a PRD template he credits to an OpenAI product leader. The template runs in four stages and twenty-four sections, with testing from the second stage.
Discoveryis it worth it4 sectionsBusiness, feature, and user value, then a hypothesis for the AI’s part
Designwhat good looks like4 sectionsFlows, screens, a first prompt, and the evaluation plan
Developmake it hold9 sectionsModel, data, the evaluation set, and edge cases
Deploykeep it honest7 sectionsRollout, risk, success metrics, and feedback
The habits of a good PRD still apply, first
This fall I read shared PRD templates from Amazon, Linear, Intercom, Figma, and others. The ones I’d reuse share three habits: the problem on page one, the metric before the feature, and a log of each decision with the options rejected.
Every good PRD
- The problem, in the customer’s words
- The metric that moves if it works
- What it won’t do
- Whether the number moved
An AI PRD adds
- Typical, edge, and negative cases
- What it may read, and how fresh
- What it does when it’s unsure
- Whether quality held after launch
Twenty-four sections is a menu, not a form
Commenters called the template waterfall: every decision made before anything is built. I think that’s true only if it’s filled in all at once.
A form. Every field, filled in at once.
A menu. The sections the risk calls for.
The best documents I read grow a little at each review, and the detail matches the risk. A two-day fix gets a paragraph; a feature that answers customers gets the evaluation plan in full.
The one section I’d never cut is the evaluation plan, because it writes down “good enough to ship” before anyone is invested in the answer.
- CasesTypical ones, edge ones, and ones it should refuse.
- RunsEach case several times, since one good answer can be luck.
- GradeWhat the customer would receive, not the AI’s rating of itself.
- BarWhat share must pass, and which cases can never fail.
Write the test for the answer before you write the prompt, the instructions the AI follows.
Check yourself
For your AI feature, what is one input it should refuse?
If nobody can name one, the evaluation set has no negative cases yet, and the feature will meet its first one in production.
Was the bar for launch written before or after the first demo?
A bar set after a good demo tends to fit the demo. Set it while nobody has seen an answer yet.
The idea: an AI PRD template shared by Moe Ali, Product Faculty. The example: the company PRD templates I read this fall.