AI agent observability

What is AI agent observability?

AI agent observability captures what an agent did, step by step, as structured traces you can query and replay. It exists because agents fail in ways that mimic success, so every conventional monitoring signal stays green while the output quietly goes wrong.

The clearest way to see why it is a separate discipline is to look at what a status code tells you. In a conventional service, a 200 means the request succeeded and there is nothing to investigate. For an agent, a 200 means the model returned text. It says nothing about whether the text was correct, whether the tool it called was the right one, whether it looped six times before answering, or whether the action it took should have required a human.

Latency looks fine. Error rates look fine. Uptime is perfect. And the agent has just closed a case it should have escalated. Traditional monitoring isn't built to catch that. It only tracks uptime and speed, not judgment. So, an agent estate wired only with application performance monitoring looks instrumented, but in the way that matters, it isn't.

What does an agent trace contain?

A tree, not a line. A session holds several agent runs. Each run holds model calls. Each model call can branch into tool calls beneath it. This tree structure lets you trace a wrong answer back to the exact step that caused it.

This shape is now standardized. The OpenTelemetry GenAI semantic conventions model an agent run as a span tree, with named operations covering agent creation and invocation, workflow invocation, tool execution, retrieval, planning, and memory operations.

What sits inside the spans is the part that is genuinely new. A conventional trace carries a route, a duration, and a status. An agent span carries the model and version, input and output token counts, the reason generation stopped, and, when content capture is switched on, the system instructions, the input messages, the output messages, and every tool call with its arguments and results.

Two consequences follow immediately, and they shape everything else on this page. The volume per run is enormous compared with an HTTP trace, because prompts and completions are large text blobs rather than short fields. And the content is often sensitive, because the prompt frequently contains the customer data the agent was reasoning about.

How is agent observability different from traditional observability?

Six differences, and none of them is cosmetic. The unit changes from a request to a trajectory, success stops being a status code, volume rises by orders of magnitude, and reproducing a failure requires capturing inputs you would never have stored before.

Existing tooling extends part of the way and stops. Understanding where it stops tells you what you still have to build or buy.

The reproducing row is the one that catches teams out. Conventional debugging assumes you can re-run the failing request. An agent given the same input twice may take a different path, so unless the trace captured the exact prompt, model version, and parameters at the moment of the failure, the incident is not investigable at all. You cannot go back for that data later.

Why is sampling the decision that matters?

Agent traces are costly enough that nobody keeps them all. The moment you set a sampling rate; you have decided which incidents you won't be able to explain. Sampling by rate throws away exactly the runs you needed.

The economics force the question. A single agent run can produce more telemetry than a thousand HTTP requests, and at production volume storing every trace in full becomes a line item somebody challenges. So teams sample, usually by picking a percentage, which is the approach carried over from APM and the wrong one here.

Random sampling is defensible when every event is equivalent, which is true of web requests and false of agent runs. A ten percent rate means nine out of ten failures have no trace, and the failures are the entire reason you built the system.

• Keep every failed run. Non-negotiable. A failure with no trace is an incident you will close without a cause.
• Keep every run that escalated to a human. These are the boundary cases that tell you where the agent's scope is wrong.
• Keep every run above a cost or step threshold. Expensive runs are where loops and missing stopping conditions show up.
• Keep every run touching regulated data. Decided by policy rather than by budget, and it usually decides itself.
• Keep a small random baseline. Signal-based sampling is biased by construction, so you need an unbiased slice to compute honest aggregate statistics against.

This point matters more than it looks. If you only keep failures and expensive runs, your trace store can't tell you what normal looks like. Every trend you read from it will be wrong. Signal sampling and a random baseline are not alternatives. They are complements.

Is observability the same as an audit trail?

No, and the difference is the sampling decision above. Observability is sampled, mutable, short-lived, and written for engineers. An audit trail is complete, tamper-evident, retained to a schedule, and written for a third party. You cannot sample an audit trail.

This is the most consequential confusion in the subject, because the two look identical while you are building them. Both capture what a system did. Both live in a trace store. A team can build excellent observability and reasonably believe the compliance requirement is covered, and only discover otherwise when somebody external asks a question about a specific run from four months ago.

The practical resolution is not to build two systems. It is to decide, per agent, whether it falls inside a regulatory scope, and route those runs to complete, immutable, long-retention storage while everything else goes to the sampled engineering store. Deciding that per agent is far cheaper than applying audit-grade handling to your entire trace volume, and far safer than discovering the gap during an inquiry. For what an audit-grade record has to contain, and for the wider evidence obligation, see AI governance.

How do you implement agent observability?

Instrument to a shared standard, capture the whole span tree rather than only the model calls, decide content capture and redaction deliberately, and set sampling by signal. Then verify what actually lands in storage rather than trusting the documentation, because the standard is still moving.

The industry has converged on OpenTelemetry as the telemetry layer, and the practical guidance below reflects both what that gives you and what it does not.

1. Instrument to the GenAI semantic conventions

   Emitting standard attribute names means a trace from one framework and a trace from another are readable by the same backend, and you can change backends without rewriting instrumentation. Auto-instrumentation exists for the major providers and frameworks, and several coding agents already emit these traces natively.

2. Capture the tree, not just the model calls

   Instrumenting only the LLM calls is the common half-measure. It tells you what was asked and answered and leaves out the tool calls, retrievals, and handoffs, which is where most agent failures originate. The span tree is the deliverable.

3. Decide content capture deliberately

   Prompts and completions are enormously useful for debugging and frequently contain personal or regulated data. Capturing them is a decision with a privacy consequence, not a default. Decide per agent, redact at the collector rather than at the backend, and record the decision.

4. Sample by signal, with a random baseline

   Configured at the collector, so the policy lives in one place rather than being scattered across services. Everything covered in the previous section applies here.

5. Keep evaluation out of instrumentation

   Record what happened, score it separately. If grading gets built into instrumentation, your trace store becomes tied to today's definition of correct. Then every change to the quality bar forces you to re-instrument. Keep traces as an append-only record and let evaluation evolve on top of it.

6. Verify what actually lands, and pin your versions

   Do not infer the schema from documentation. Emit a trace, read the stored span, and confirm the attribute names. Then pin the framework, the instrumentation package, the SDK, and the exporter, because each carries its own assumptions about the conventions.