Curriculum · Performance

Directing Agents Well

5 lessons · For: Anyone who is responsible for the output of one or more agents

Managing agents is a management discipline, not a prompting technique. The questions are the ones any manager faces — what am I asking for, how will I know if it was done well, and what do I do about it if it was not — asked of a worker that never pushes back and never says it is confused.

That last property is what makes it hard. A human who does not understand the brief usually says so. An agent produces something plausible instead. Everything in this track is downstream of that one fact.

Five lessons. Take Working Inside an AI Organization first if you have not.

Lessons

Write the standard before you write the instruction

You should be able to: Produce a definition of done that someone else could check the output against.

Most bad agent output traces back to a brief that never defined success. The instruction said what to do; it did not say what a correct result looks like. So the agent optimised for something plausible, and there was no standard to catch it.

Invert the order. Write the acceptance criteria first — what must be true of the output for it to be right — then write the instruction that should produce it. If you cannot write the criteria, you are not ready to delegate the work, and that is useful information.

A good test: could a colleague who knows nothing about the task check the output using only your criteria? If not, the criteria are impressions, not standards.

Scope narrowly, then widen on evidence

You should be able to: Set an agent's initial scope and define what earns it a wider one.

The instinct is to give an agent everything it might need. The discipline is to give it the least it can do the job with, and to widen only against a record of it doing that job correctly.

This is exactly how you would onboard a person into a role with real consequences, and for the same reason: standing should be earned by demonstrated performance rather than granted on optimism. The difference is that agents scale instantly, so an over-broad scope granted on day one is not one person's mistake — it is a thousand executions of it.

Decide in advance what would justify widening: a volume of clean executions, a period without exceptions, a category of task handled correctly. Then actually check before you widen.

Source: GDA Group — What authority should an AI agent hold?

Measure the thing, not the activity

You should be able to: Choose metrics for an agent that would survive scrutiny from someone sceptical.

Agents generate enormous quantities of activity data — calls made, tokens spent, tasks closed — almost none of which tells you whether the agent is good. Volume is not performance. An agent producing twice as much wrong output is worse, not better.

The metrics that matter are exception rate (how often did output fail the standard), correction cost (how much human time went into fixing it), and outcome rate (did the work it produced achieve what it was for). All three require the standard from lesson one, which is why that lesson is first.

Track them per agent, over time, at a fixed scope. An agent whose exception rate is rising is telling you something changed — in the inputs, in the environment, or in what you are asking of it.

Source: FlashyOS — Agent optimization

Debug the instruction before you blame the agent

You should be able to: Diagnose a failed run from the record rather than by re-running it.

When output is wrong, there are four candidates: the instruction was ambiguous, the input was wrong, the scope was insufficient, or the agent genuinely underperformed. In practice the fourth is the least common, and it is the one people reach for first.

Work the list in order, from the record. Read what the agent was actually told, then what it was actually given, then what it was actually allowed to do. Most failures resolve before you reach the fourth candidate.

The reason this matters beyond the individual run: if you fix the agent when the problem was the instruction, the failure recurs and you have learned nothing. Instruction defects are systematic. Fixing one fixes every future run.

Know when to stop an agent

You should be able to: State, in advance, the conditions under which an agent gets pulled.

Every agent should have a shutdown condition written before it goes live: the exception rate, the class of error, or the change in environment that means it stops running until someone looks at it. Deciding this under pressure, after something has gone wrong, is how organizations end up defending an agent they should have pulled.

The condition should be checkable without a debate. "If it produces two errors of this class in a week, it pauses" is a condition. "If it seems unreliable" is a conversation.

Pulling an agent is not a failure of the programme. An organization that has never stopped an agent either has not deployed anything consequential or is not checking.

Frequently asked

How do you manage AI agents?

Define the standard before the instruction, scope narrowly and widen on evidence, measure exception rate rather than activity volume, debug the instruction before blaming the agent, and set a shutdown condition in advance. It is a management discipline, not a prompting one.

How do you measure whether an AI agent is performing well?

Exception rate against a pre-set standard, human correction cost, and outcome rate. Activity metrics — tasks closed, tokens spent, calls made — measure how busy the agent was, not whether it was any good.

How much authority should you give an AI agent?

The least it can do the job with, widened only against a record of clean execution at the current scope. Define in advance what evidence would justify a wider scope, and check for it before granting one.

What is the most common cause of bad agent output?

An instruction that never defined what a correct result looks like. The agent optimises for something plausible and there is no standard to catch it — which is why writing acceptance criteria before the instruction is the first lesson rather than a later one.