Lesson 4 of 12 · 3 min read

What is an AI agent actually made of, besides the model?

A language model is only one part of five. The other four decide what it knows, what it can reach, how it does a familiar job the same way each time, and when it keeps going or stops. Each of the five, not only the model, is a place a design decision lives.

Diagram: Model, Retrieval, Connections, and Skills feed an agent loop of Plan, Act, and Check; at the boundary the run pauses for a person to approve or stop it.
The model is one part of five. At the boundary, a person makes the call.

The model is the part everyone talks about, and one part of five

In a LinkedIn post, Harish Kumar put it plainly: an agent is more than a language model. The model is the brain, and the other four parts decide what that brain can actually do.

Five building blocks, and the design question each one raises
  1. Modelthe brainHow does it say it’s unsure?
  2. Retrievalthe libraryWhat may it read, and how current is it?
  3. Connectionsthe plugsWhat may it touch, and what can’t be undone?
  4. Skillsthe know-howWhich jobs should it do the same way every time?
  5. The loopthe brain in actionWhen does it keep going, and when does it stop?

He adds two cautions: not every agent needs all five, and a connection alone doesn’t make anything an agent.

Three ways software does a job, by how much it decides on its own
  1. Automationone rule, the same result every time
  2. Fixed workflowsteps in an order someone set, with a model inside some
  3. Agentchooses its own next step toward a goal

Use the least of the three that does the job. If the steps never change, an agent only adds ways to be wrong.

What an agent knows changes what it builds

I ran a real experiment. I asked an AI coding agent twice to add bulk actions to a customer data table built from a design system’s parts.

One run read twelve written design rules, like “a destructive action needs a second step or a way back.” The other had only the parts.

The same model, given two different libraries

The parts only

  • The parts, their named values, and examples
  • Stayed inside the system
  • Automated checks found no defect

The parts and the rules

  • The same, plus twelve written rules
  • Stayed inside the system
  • Automated checks found no defect

Both passed: on a job the parts already cover, the rules added nothing. They matter when a job goes beyond the parts, so what an agent may read is a design decision.

The loop is where the person comes in

The fifth block is the loop: plan a step, use a tool, check the result, and go on.

In a simulated run in Agent Review, my prototype of working with coding agents, I give the agent a job: rebuild the billing screens of Relay, a made-up customer app, from the design system’s parts, changing nothing anyone can see.

Agent Review’s record of one decision, marked Decided by you: what the agent did, the designer’s reason in their own words, the accepted trade-off, the remaining risk, a follow-up, and the guidance kept as a rule.
From my work Agent Review, a simulated run. The record of one decision the agent stopped for. See Agent Review

The agent met a cancel link built by hand, quieter than every other action.

The design rules would have made cancel easier to find: a visible change, in a job meant to change nothing. So the agent didn’t guess. It stopped on that one change and asked.

I think that stop is the most designed thing in the run, and the question it raised was mine to answer.

An agent is only as good as the moment it knows to stop.

Check yourself

For an agent on your team, which of the five blocks has nobody designed?

My guess: retrieval or the stop. Someone picked the model; nobody decided what it reads or when it hands back.

What can your agent touch that can’t be undone?

That list is where it should ask first. A connection is a capability, and every capability needs a limit someone chose.

The idea: Harish Kumar, Your AI agent is more than just an LLM. The example: Agent Review, a simulated run, on this site.