Enterprise AI should help people decide, not just answer

Key idea 1 of 7

A suspected cause is a place to start, not permission to act

AI earns its place in an enterprise workflow when it helps someone make a better decision or finish a real task. An answer is only part of that. The person still has to understand what it rests on, what is still uncertain, and what happens if they act on it.

Take a proposed feature that groups failed orders and suggests a likely cause.

The system notices that twelve rejected orders appear to share a missing customer identifier. It suggests a problem in the rules that translate the partner’s documents into the receiving system’s format.

That is a good place to start an investigation.

It is not yet permission to change the rule or resubmit twelve orders. The design work lives in that distance.

I have built one working version of that distance, for a different job. Second Look is a dispatch prototype. The system recommends a truck for a load, and the dispatcher decides whether it is right.

It runs on synthetic data. The ranking is a transparent weighted rule rather than a model. It has not yet been in front of a working dispatcher. It carries the rest of this article because it is the one I can show rather than describe.

Key idea 2 of 7

Finding, summarizing, suggesting, drafting and acting are five different products

Before choosing between a conversational interface and a recommendation card, the team has to settle what role it is giving the system.

Each role is a different product, with a different cost when it is wrong.

1. Findthe relevant records
2. Summarizethe evidence
3. Suggesta cause
4. Drafta correction
5. Actunattended

Second Look is deliberately the third of the five. It suggests, and nothing on its screen acts without a press. Its own page names the next step: a version that assigns the clear picks itself and holds the close calls for a person. The page calls that step the line between a recommendation and an agent.

Crossing it is a conversation about reliability, permissions, consequences and recovery. It is held with product, engineering and whoever owns the model. A designer should not imply that an action is safe because it was easy to draw as a button.

Part of settling the role is asking whether the task needs AI at all. A known rule deserves a clear deterministic answer, not a generated explanation of itself.

In Second Look the driver’s hours-of-service limit is a federal regulation. The system enforces it before any score is figured. The truck it removes keeps its row and its reason in the one caution color on the screen. There is nothing about that limit to learn or to rank.

Key idea 3 of 7

Put the evidence beside the recommendation

Second Look puts five factors under the recommended truck. Each shows its value, its verdict and where it stands against the rest of the fleet. The same five columns run down the data table beside it.

A score would have been simpler to draw. It is also the one number that cannot be checked against anything. Five figures can be argued with.

The evidence has to be drawn from the records rather than generated alongside the answer, because an explanation that sounds right and cites nothing is worse than no explanation. And uncertainty belongs in the same place.

When two trucks are level, the screen says so. It names the tradeoff, faster pickup against more driver hours, and declines to pick.

When one of the two rests on a stale figure, the card says how old it is and what it left. The hours reading is 45 minutes old. The last one left 30 minutes to spare. Check the logbook before assigning. The doubt does not move the ranking. It changes what the dispatcher does before pressing.

The wording carries the claim. “These orders share a missing field” and “changing this mapping will resolve these orders” are not the same sentence.

Key idea 4 of 7

Give people control that does something

The dispatcher has the phone call the system cannot hear, so disagreeing has to be cheap.

In Second Look, Assign sits at the end of every row the rule allows, not only the recommended one. There is no dialog in the way. The question about why comes afterward. It is optional, and Skip weighs the same as Save.

Feedback needs the same honesty. Dismissing a suggestion might hide it, flag it for review, or feed a later retraining cycle. The interface should say which. It should not imply that the system just learned something.

The override prompt says where its answer goes. Three matching reasons on one customer in a month send a review to the dispatch manager, the one person who can change what the ranking weighs. Whether three is the right number is a guess. The prototype’s page says the reasons on the prompt were guessed too.

A more autonomous feature needs more than this: an authorized scope and stopping conditions that engineering enforces rather than the copy describes. That is the subject of a separate article.

Key idea 5 of 7

Prototype the wrong answer

Second Look was built around four scenarios rather than one happy path:

• A clear pick
• A close call where it declines to choose
• A stale figure under one of the two level trucks
• A rule that removes the truck that would otherwise rank first

The wrong answers are the ones worth building, because a correct suggestion teaches nobody where to look.

What it does not have yet is the watching. The study it was built for puts the fleet table alone beside the table with the recommendation. It does this over a straightforward call, a plausible wrong one, and a load where a figure on screen is stale.

The fourth case in the list below, one that changes while the person is reviewing it, is the one the prototype does not have.

Build

• A correct suggestion
• A plausible but wrong one
• One with incomplete evidence
• A case that changes while the person is reviewing it

Watch for

• Can they detect the mismatch?
• Do they know how to dig further?
• Can they keep working when the system cannot help?
• Do they notice that the case moved?

Simulated output is enough to test the interaction before a model is ready. That holds as long as everyone stays clear that a successful prototype test says nothing about whether the model is accurate. Second Look has no model at all.

Key idea 6 of 7

Measure the decision, not the acceptance

Acceptance rate is the easiest number to collect and the wrong one to call success. A high rate can mean the suggestions are good. It can equally mean people are clicking through them.

The study written for Second Look orders its measures on purpose:

• Decision quality first
• Then missed constraints
• Then checking effort
• Time last

A faster wrong answer is the failure that design could plausibly cause, so it is the one to look for.

The measures are correct decisions, unnecessary interventions, successful recoveries, and whether the underlying problem takes less work to fix. They are read alongside the AI team’s own evaluation of the model.

Key idea 7 of 7

Leave a record a person who was not there can read

The last part of the relationship is the one usability work tends not to reach. People and AI systems develop a relationship over time.

Second Look shows a truck’s record on this kind of load, on time on 38 of the last 40. It puts the two late ones on the card, with what to watch for because of them. A dispatcher who has read “38 of 40” on one lane a dozen times stops reading it. What is left are the two lines saying what to check.

The instinct in a product is to hide where the system is weak until it improves. The better move is to help people see it, so they know when to move fast and when to slow down. Microsoft’s guidelines for human–AI interaction put this under “over time.”

So some of these products should expose their history. Not a technical log of model events. A useful history holds:

• What the system recommended, on which records, at what time
• What happened
• When a person chose differently, and why
• Where this kind of recommendation tends to fail

That is the same record an auditor would ask for. Second Look writes half of it. It saves the override reason with the load. Its page says that what the system recommended, on which figures, at what time, is not written down yet. A decision on it cannot be fully reconstructed by someone who was not there. A production version would need both halves.

The Deploy Console, built after it, writes the whole record with each rollback. That is who decided, when, from what to what, and the figures it was decided on.

The opportunity here is larger than adding a chat window. It is to give someone better evidence, a better starting point and the right amount of help. It is also to keep the consequences of the decision legible to the person who owns it.