Lesson 2 of 12 · 3 min read
What is UX for when agents build the software?
Three things a factory can’t do for itself: decide what’s worth building, judge whether what it built works for people, and supply the context it builds from. An AI pipeline can plan, build, test, and ship without anyone asking a user, and it will report that everything is fine.
A factory can build the wrong thing perfectly
Tanner Kohler, of the UX research firm Nielsen Norman Group, read the published guides for building software with agents, and found UX reduced to context: guidelines the agents read, then a process that runs without a designer in it.
PlanAn intent, approved by the product owner
DesignA spec and a mockup
BuildA plan, approved by an engineer
TestScreenshots against the mockup, not people
DeployCode review, then a technical lead
MaintainTechnical health, not whether people finish
People approve three stages: plan, build, and deploy. None of them are the people the feature is for. His example: people quit a feature over confusing labels, and the monitoring shows nothing wrong.
A perfect build of the wrong thing is still the wrong thing.
Deciding what’s worth building comes before the first file
At Midwest Couriers, a courier company with 450+ vehicles, I was sure the dispatchers’ new tool needed a map. A morning of watching them ended that: they knew every location code by heart.
A factory would have built my map in an afternoon, passed every test, and shipped it. Speed only delivers a wrong assumption sooner.
The cheaper building gets, the more the decision about what to build is worth.
Judging what got built still takes people
I use agents to test my own builds. They click through every path I script, every time, and flag dead ends. What they can’t tell me is what a person expected to happen.
The factory
- Every path, on every build
- Dead ends, broken states, contrast
- Whether it matches the mockup
A person
- What they thought would happen
- Where they hesitated, and why
- Whether they finished at all
Kohler’s fixes: write the intent around a user’s need, and set acceptance criteria people can fail. Name the changes a designer must review. Watch for people quitting tasks and filing support tickets.
Supplying context, the third thing a factory can’t do alone, is the part the guides already ask of UX. I think it matters least, and the next lesson covers it.
A factory can tell you the build passed. Only people can tell you it worked.
Check yourself
Where in your process would a user’s confusion first show up?
If the answer is a support ticket after launch, the pipeline has no stage that looks for it. That’s the stage to add.
Who writes the acceptance criteria, and could a real person fail them?
Criteria that only a screenshot can fail measure the build. Criteria a person can fail measure the product.
The idea: Tanner Kohler, Nielsen Norman Group, The Role of UX in Agentic Software Factories, shared by Tanner Christensen. The example: the dispatch case study.