Lesson 5 of 12 · 3 min read
What should you never do when building an agent?
Never trust a success the agent reports about itself. A call that returns “OK” may have done nothing, a confident answer may be wrong, and an action nobody can stop will run when it shouldn’t. Many of the rules that prevent this are design decisions.
A status can be accurate and still not tell you enough
In After Send, my bill-paying concept on simulated data, a supplier calls a bakery it invoiced. The supplier’s records say unpaid. The bakery’s screen says Paid.
Both are right. The bakery sent a one-time card, and the supplier hasn’t charged it yet. “Paid” covers three steps, and the screen collapsed them into one word.
IssuedThe buyer sent a one-time card. Done.
ChargedThe supplier runs the card. Not yet.
MatchedThe buyer ties the charge to the invoice. Not yet.
Every rule on the list cost someone an incident
Alistair Greenwood, who builds AI agents, listed 21 things he never does when building agents, each learned from something that broke in live use. One is After Send’s problem: a successful response is not proof that anything happened.
I sort his 21 into four questions a designer can answer:
Controlwho decidesNo irreversible action by default, and no agent choosing its own permissions
Proofwhat really happenedJudge the whole run, not the final answer, and never the model’s own confidence
Memorywhat it carriesEverything it remembers has an owner, an age, and an expiry
Stoppinghow it endsA kill switch and a way back before anything ships
The interface is where these rules become visible
Each rule needs a screen that keeps it. In After Send, the payment record shows when it last checked.
A confidence score only says how sure the system is. Where a fact came from, and how old it is, tells a person what to do.
A run doesn’t only succeed or fail. It can finish some steps, or send a request and hear nothing back, and each ending needs a different next move.
- DoneIt happened, and here is the proof.
- Partly doneWhich steps finished, and which one didn’t.
- UnknownNo answer came back. Check before trying again.
- FailedNothing changed. Safe to try again.
Unknown is the one that hurts. Retrying a payment that may have gone through is how someone pays twice, which is why After Send checks for a late charge first.
Stopping needs the same care: a kill switch nobody can read is just a button. A stop has to work mid-run, from the keyboard, and say what it left half-finished.
Not all 21 are mine to build. Engineers own the retry logic and the tests a release must pass. What counts as passing is a shared call, and I’d want to be in the room.
If the agent can say “done” when nothing happened, the screen has to say what actually did.
Check yourself
Which status in your product could be accurate and still mislead someone?
Look for one word covering several steps. Each step that can be open on its own deserves its own line.
If your agent had to be stopped right now, what would the person see?
If the answer is a spinner that goes away, the stop is designed for the system, not for the person who pressed it.
The idea: Alistair Greenwood, 21 things I never do when building AI agents. The example: After Send, a concept on simulated data, on this site.