How do you measure whether an AI agent is working?

Measure task completion against what the user wanted, not uptime or run count. For each run, record whether the agent reached the correct outcome, whether a human had to intervene, and whether it escalated when it should have. An agent that runs perfectly and reaches the wrong answer is a worse problem than one that crashes, because nothing alerts you.

What is an eval set and why does an AI agent need one?

An eval set is a fixed collection of real inputs with known-correct outputs that you score automatically on every change. Without one you cannot tell whether a prompt edit improved the agent or simply moved which cases it fails on, because you only ever check the case in front of you. Twenty to a hundred cases is enough to start, and every production failure should be added to it permanently.

What should you log from an AI agent?

The full input, the full output, the model and version, token counts, latency, every tool call with its arguments and result, and the final outcome. Sampling is not enough early on, because the runs you need are the rare ones. This is also the only way to build a realistic eval set, since real traffic is stranger than anything you would invent.

How often do AI agents fail silently?

Often enough that silent failure, not crashing, is the main risk. A model returns something confident, well-formatted and wrong, nothing errors, and a user acts on it. The defences are a confidence signal the interface respects, schema validation on anything structured, and a rule that an agent which cannot complete a task must escalate rather than improvise.

What does it cost to add evaluation to an existing AI agent?

A basic eval harness with twenty to thirty real cases and automated scoring is one to three days of work. Adding full request logging and a failure-rate alert is another day. Against the cost of discovering a systematic wrong answer from a customer weeks later, it is the cheapest insurance in an AI build.

AI engineering29 September 2026 · 7 min read

How to Tell If Your AI Agent Is Actually Working

The short answer

Uptime and run counts tell you an agent executed, not whether it did the job. To know if it works you need three things: an evaluation set of real cases with known-correct outcomes that you score automatically, task completion rate measured against what the user actually wanted, and full logging of inputs and outputs so a failure can be reproduced. Without an eval set, every prompt change is an unmeasured guess.

A company running AI agent reviews for marketing agencies described the problem in one line: most agencies can see that an agent ran, few can tell whether it did well.

That gap is where AI projects quietly fail. The dashboard is green. Executions are up. Nobody knows whether the thing is doing its job, because nobody defined what doing its job looks like in a way a machine can check.

Uptime is not the metric

Standard monitoring tells you the agent responded. For an LLM agent that is close to meaningless, because the failure mode is not a crash. It is a confident, well-formatted, wrong answer that nothing flags.

Three things to measure instead:

Task completion. Did the agent reach the outcome the user wanted? Not "did it reply", but "did the booking get made, the lead get scored correctly, the question get answered". This requires deciding in advance what the correct outcome is for each kind of request.

Intervention rate. How often did a human have to step in and fix it? This is the number that tracks whether the agent is saving time. An agent with 95 percent task completion that needs a human to check every output has saved nobody anything.

Escalation accuracy. When the agent could not do the job, did it say so? An agent that hands off cleanly at 70 percent completion is more useful than one that improvises at 90, because you can trust the 70.

Build an eval set before you tune anything

An eval set is a fixed collection of real inputs with known-correct outputs that you score automatically on every change.

Without one, every prompt edit is a coin flip you cannot score. You fix the case you were looking at, quietly break two you were not, and because you only check the case in front of you, you never find out.

Twenty to a hundred cases is enough to start. What goes in:

The ordinary middle. Common requests that must keep working. Most of the set.
Known edges. Empty fields, wrong language, enormous inputs, requests that are nearly but not quite the thing.
Adversarial cases. Someone will paste a prompt injection in. Find out now rather than later.
Cases the agent should refuse. Requests outside its remit, where the correct behaviour is escalation. Everyone forgets these and they are the most important group.
Every production failure, permanently. The moment something fails in the wild it becomes a test case. This is how the set gets good, and it costs nothing.

Score with the cheapest honest method. Exact match for structured output. A numeric or string assertion where there is one right answer. A model grading against a written rubric for prose, accepting that the grader is itself fallible and needs spot-checking.

Offline evals and live monitoring do different jobs

Offline evals run against the fixed set before you ship a change. They answer "did this change make things better or worse". Run them in CI so a prompt edit cannot merge without a score.

Live monitoring runs against real traffic after you ship. It answers "is it still working". Live traffic drifts in ways your eval set does not, particularly when a provider updates a model underneath you.

You need both. Teams usually build one and assume it covers the other.

What to log

Log everything, at least until volume makes it genuinely expensive:

The full input and the full output
The model and version, so you can attribute a change in behaviour to a provider update
Token counts and latency, per request
Every tool call, with arguments and results
The final outcome, and whether a human intervened

Sampling is tempting and wrong early on, because the runs you need are the rare ones. You cannot debug a specific customer's complaint from a 5 percent sample.

This log is also where your next eval set comes from. Real traffic is stranger than anything you would invent at a desk.

The alert that matters most

One alert earns its place above the rest: the rate of outputs failing validation.

If your agent returns structured data, validate it against a schema and track how often that fails. A sudden rise is the earliest signal that something upstream changed, usually a model update or a shift in the kind of input arriving. It fires before customers notice, which is the entire point of monitoring.

Add a second on escalation rate. A sharp fall is not good news. It usually means the agent has started answering things it should be refusing.

Guardrails belong in code

A rule written only in the system prompt is a suggestion. The model will follow it most of the time, which is a strange thing to rely on for anything that matters.

Anything that must not happen gets enforced around the model:

Schema validation on structured output, before anything downstream touches it
Hard limits on what tools the agent can call, and with what arguments
A deterministic check on actions that cost money or send something to a customer
A human approval step for anything irreversible

The model proposes. Code decides.

A realistic checklist

An eval set of at least twenty real cases, scored automatically
Evals running in CI, so a change cannot merge unscored
Task completion and intervention rate tracked, not just uptime
Full request logging with tool calls and outcomes
Schema validation on structured output, enforced in code
An alert on validation failure rate
An alert on escalation rate moving sharply in either direction
A written statement of what the agent must never do, enforced outside the prompt

What it costs

Adding this to an existing agent is one to three days for a basic eval harness with twenty to thirty cases and automated scoring, plus about another day for full logging and the failure-rate alert.

Measured against finding out from a customer, weeks later, that your agent has been confidently wrong in a specific situation the whole time, it is the cheapest insurance in an AI build. It is also the part most quotes leave out, so ask for it explicitly.

Hiring for this

Ask any AI developer how they would know their agent had got worse after a prompt change. The answer tells you almost everything. Someone who has shipped agents talks about eval sets and regression scores. Someone who has not talks about testing it manually.

Second question: what happens when the model is wrong? A good answer covers the fallback path, the confidence signal and the escalation route. "We tuned the prompt so it does not happen" is not an answer.

We build and review these as AI agents and automation and AI integration work. The AI Client Qualifier is our own product running this pipeline in production, and autonomous multi-agent operations shows the shape when several agents hand work to each other.

If your agent sits inside an n8n workflow, what an n8n automation costs covers the error handling around it.

Common questions

How do you measure whether an AI agent is working?: Measure task completion against what the user wanted, not uptime or run count. For each run, record whether the agent reached the correct outcome, whether a human had to intervene, and whether it escalated when it should have. An agent that runs perfectly and reaches the wrong answer is a worse problem than one that crashes, because nothing alerts you.
What is an eval set and why does an AI agent need one?: An eval set is a fixed collection of real inputs with known-correct outputs that you score automatically on every change. Without one you cannot tell whether a prompt edit improved the agent or simply moved which cases it fails on, because you only ever check the case in front of you. Twenty to a hundred cases is enough to start, and every production failure should be added to it permanently.
What should you log from an AI agent?: The full input, the full output, the model and version, token counts, latency, every tool call with its arguments and result, and the final outcome. Sampling is not enough early on, because the runs you need are the rare ones. This is also the only way to build a realistic eval set, since real traffic is stranger than anything you would invent.
How often do AI agents fail silently?: Often enough that silent failure, not crashing, is the main risk. A model returns something confident, well-formatted and wrong, nothing errors, and a user acts on it. The defences are a confidence signal the interface respects, schema validation on anything structured, and a rule that an agent which cannot complete a task must escalate rather than improvise.
What does it cost to add evaluation to an existing AI agent?: A basic eval harness with twenty to thirty real cases and automated scoring is one to three days of work. Adding full request logging and a failure-rate alert is another day. Against the cost of discovering a systematic wrong answer from a customer weeks later, it is the cheapest insurance in an AI build.

Work we do on this

AI Agents & Automation AI Integration AI Client Qualifier Autonomous Multi-Agent Operations

Want this looked at properly?

Tell us what you're working with. We'll give you a straight answer in a 30-minute call, free.