Lesson 5 of 12 · 3 min read

What should you never do when building an agent?

Never trust a success the agent reports about itself. A call that returns “OK” may have done nothing, a confident answer may be wrong, and an action nobody can stop will run when it shouldn’t. Many of the rules that prevent this are design decisions.

Diagram: an agent reports an invoice as Paid; checked against records, it was Issued, but not yet Charged or Matched.
The report says paid. The records say only one step of three is done.

A status can be accurate and still not tell you enough

In After Send, my bill-paying concept on simulated data, a supplier calls a bakery it invoiced. The supplier’s records say unpaid. The bakery’s screen says Paid.

Both are right. The bakery sent a one-time card, and the supplier hasn’t charged it yet. “Paid” covers three steps, and the screen collapsed them into one word.

What “Paid” was hiding
  1. IssuedThe buyer sent a one-time card. Done.
  2. ChargedThe supplier runs the card. Not yet.
  3. MatchedThe buyer ties the charge to the invoice. Not yet.

Every rule on the list cost someone an incident

Alistair Greenwood, who builds AI agents, listed 21 things he never does when building agents, each learned from something that broke in live use. One is After Send’s problem: a successful response is not proof that anything happened.

I sort his 21 into four questions a designer can answer:

Greenwood’s 21 rules, in my four groups
  1. Controlwho decidesNo irreversible action by default, and no agent choosing its own permissions
  2. Proofwhat really happenedJudge the whole run, not the final answer, and never the model’s own confidence
  3. Memorywhat it carriesEverything it remembers has an owner, an age, and an expiry
  4. Stoppinghow it endsA kill switch and a way back before anything ships

The interface is where these rules become visible

Each rule needs a screen that keeps it. In After Send, the payment record shows when it last checked.

After Send’s payment record for Hillside Packaging with the card expired and not processed. Replacement options are open, and the first is to check for a late charge.
From my work After Send, on simulated data. The card expired unused, so the screen checks for a late charge before sending a new one. See After Send

A confidence score only says how sure the system is. Where a fact came from, and how old it is, tells a person what to do.

A run doesn’t only succeed or fail. It can finish some steps, or send a request and hear nothing back, and each ending needs a different next move.

Four ways a run can end, and what the screen owes the person
  1. DoneIt happened, and here is the proof.
  2. Partly doneWhich steps finished, and which one didn’t.
  3. UnknownNo answer came back. Check before trying again.
  4. FailedNothing changed. Safe to try again.

Unknown is the one that hurts. Retrying a payment that may have gone through is how someone pays twice, which is why After Send checks for a late charge first.

Stopping needs the same care: a kill switch nobody can read is just a button. A stop has to work mid-run, from the keyboard, and say what it left half-finished.

Not all 21 are mine to build. Engineers own the retry logic and the tests a release must pass. What counts as passing is a shared call, and I’d want to be in the room.

If the agent can say “done” when nothing happened, the screen has to say what actually did.

Check yourself

Which status in your product could be accurate and still mislead someone?

Look for one word covering several steps. Each step that can be open on its own deserves its own line.

If your agent had to be stopped right now, what would the person see?

If the answer is a spinner that goes away, the stop is designed for the system, not for the person who pressed it.

The idea: Alistair Greenwood, 21 things I never do when building AI agents. The example: After Send, a concept on simulated data, on this site.