Who decides when software can act on its own?

Key idea 1 of 6

Designing an interface now means designing a decision

Here’s an illustration. You open your retirement account and an assistant has a suggestion. It noticed you save less than people in similar circumstances, projected your income, and worked out that raising your contribution by 3% would improve your outlook. Under that is a big button: Optimize my future. You press it, and your finances change in one tap.

It’s a very efficient experience. It also hides a question. Did you make an informed decision, or did someone design an interface that made one decision feel inevitable? And if the recommendation was wrong, who answers for it?

Behavioral designers have worked on that question for years. At a Women Who Code summit in October, Zakiya Pope (opens in a new tab), a behavioral design leader at TIAA, laid out the signals that turn hesitation into engagement. People say yes when it feels clear, easy, normal, safe and worth it. Those are real levers. A good default can get someone saving who never would have started. The same levers can push a choice the person wouldn’t make with the facts in front of them.

AI adds two things. It can tune those levers for each person, and it can move from suggesting to doing:

1. 01Answersa person asks, reads and decides
2. 02Recommendsthe system proposes, and a person judges
3. 03Acts on approvala person says yes before anything happens
4. 04Acts, then reportsa person reviews it afterward
5. 05Acts alonenobody looks unless something breaks

Each step down that list is a delegation, and the interface sets its terms. It says what the system may assume, when it acts, when it asks, and when it has to leave the choice alone. Every default, every framing and every button is part of the decision now.

Key idea 2 of 6

A nudge and a push look the same in the metrics

Take two versions of that retirement suggestion.

• One says the change would likely help, shows its assumptions and what it does to take-home pay, and leaves the choice to the person.
• The other shows a large, reassuring button and puts the downsides in small print.

Both raise contributions. On a dashboard they’re the same number. Only one respects the person’s right to decide.

The second version has a name. The FTC’s 2022 report Bringing Dark Patterns to Light (opens in a new tab) documents design meant to trick and trap people. Its examples include ads disguised as something else, fees buried out of sight, subscriptions made hard to cancel, and tricks that get people to share their data. AI can make the same moves personal. A system that knows your habits can know when you’re most likely to say yes.

It sees

• The yes
• How fast the yes came
• How many people said it

It misses

• Whether they understood what they agreed to
• Whether they’d choose it again
• Who quietly reversed it a month later

So the question for a designer is bigger than the conversion rate. It isn’t how well the product gets people to do something. It’s how well it helps them do what they actually want, and that takes measures a funnel doesn’t have.

If a metric can’t tell a nudge from a push, it isn’t measuring the design.

Key idea 3 of 6

Confidence in AI can quietly replace checking it

The risk isn’t only in what the system does. It’s also in what the person stops doing.

In a 2025 study (opens in a new tab), Microsoft researchers surveyed 319 knowledge workers about 936 real examples of using generative AI at work. The more confident people were in the AI, the less critical thinking they reported. The more confident they were in themselves, the more they reported. It’s a self-reported survey, so it shows a pattern, not a cause. It’s still the pattern a designer should worry about: checking fades as confidence grows.

Explanations don’t fix that on their own. An earlier study (opens in a new tab) from the University of Washington and Microsoft Research found that adding explanations raised the odds people would go along with the AI, right or wrong. A fluent reason reads as a good one.

So the interface has to keep the checking alive. Second Look, my dispatch prototype, tries to in three ways:

• It puts five figures from the records under its recommendation, not a score, because figures can be argued with.
• When one of them is stale, the card says how old it is and what to check before assigning.
• When two trucks are level, it says so and declines to pick.

It runs on synthetic data and hasn’t been in front of a working dispatcher yet, so it shows the interaction, not the result. An explanation that sounds right is not evidence that it is right.

Key idea 4 of 6

Aim for appropriate reliance, not maximum trust

The usual goal for an AI feature is more trust. It’s the wrong goal. A system people trust completely is a system nobody checks, and every system is sometimes wrong.

Researchers at Microsoft use a better target: appropriate reliance (opens in a new tab). The aim is for people to take the system’s advice when it’s good and catch it when it’s not. The design’s job is to make both of those possible, understandable and easy.

1. 01Too littlepeople redo what the system got right
2. 02Appropriatethey accept it when it’s right and catch it when it isn’t
3. 03Too muchthey accept what the system got wrong

A few things move people toward the middle:

• Show what a recommendation rests on, drawn from the records rather than generated beside the answer.
• Say how old the evidence is.
• Call a close call a close call.
• Make disagreeing cheap. In Second Look, Assign sits on every row the rules allow, not only the recommended one. The question about why comes afterward and is optional, and Skip weighs the same as Save.

Then measure the decision, not the acceptance. A high acceptance rate can mean the suggestions are good, or that people stopped reading them. There’s more on measuring the decision in Enterprise AI should help people decide, not just answer.

The goal isn’t more trust. It’s trust that matches how good the system really is.

Key idea 5 of 6

An approve button isn’t oversight

When software can act, the usual safeguard is a person who approves. An approval is only oversight if that person can actually say no.

Here’s an illustration from the kind of work I design for. An agent in a dispatch system recommends moving a truck to a different load, and the screen says its confidence is high. The dispatcher approves. But the driver has a limitation that isn’t in the data. The agent’s math wasn’t wrong. The system treated an incomplete picture as enough to act on, and the approval couldn’t catch it, because nothing on the screen said what was missing.

A formality

• A yes or no with nothing to judge
• No word on what’s missing or how old it is
• Saying no takes longer than saying yes

Oversight

• The evidence, and what it doesn’t cover
• The consequence, and whether it can be undone
• A no that costs no more than a yes

The decision should sit with whoever is accountable for the outcome. In Tushar Roy’s case study (opens in a new tab), that’s the person who owns the metric: an agent drafts the analysis, and that person signs off before it’s published. In Second Look the hardest line is a federal rule. The driver’s hours-of-service limit is checked first, before anything is ranked, and no score can overrule it. The close calls stay with the dispatcher.

The standards are catching up. NIST’s AI Risk Management Framework (opens in a new tab) asks organizations to define, assess and document how human oversight works, not just to have some. For a designer that becomes a test: give real people a bad recommendation and count how many catch it. What it takes to supervise a whole run of an agent’s decisions is in What does a person need to supervise an AI agent?

If saying no is harder than saying yes, the approval is a formality.

Key idea 6 of 6

What the system says when it doesn’t know belongs in the design system

A traditional design system defines what an error message looks like. An AI product has a harder question: what does the assistant say when it doesn’t know? Should it:

• Say how unsure it is?
• Ask a clarifying question?
• Hold and wait for a person?
• Refuse to go on?

And what happens after a confident answer turns out to be wrong? Those aren’t copywriting decisions. They’re decisions about the relationship between the system and the person relying on it, and they should be made once, in the system, not by each team on each screen.

Patricia Yu, a senior designer at Adobe, talked on a podcast in April (opens in a new tab) about design systems gaining a conversational layer. It holds the brand voice and the rules of a conversation, kept beside the tokens and components. That’s where the words for uncertainty belong too. A pattern for not knowing reaches every screen the way a color token does.

1. 01What’s missingthe figure it doesn’t have
2. 02How old it isthe reading is 45 minutes old
3. 03It’s a close calltwo answers are defensible, so it doesn’t pick
4. 04Nopast a limit no score can argue with

Second Look has words for the last three, written once. The design system’s job is to make sure the next screen gets the same words, and that the checks can tell when it doesn’t. The rest of what a system needs once AI is in the product is in its own article.

The measures change too. Speed, completion and adoption aren’t enough once a product decides things for people. Measure whether people understood, how often they corrected the system, which outcomes nobody wanted, and whether people still feel in control.

What a system says when it doesn’t know is a pattern, and patterns belong in the design system.