AI Agent Observability: Beyond Latency and Tokens

AI Agent Observability: What to Monitor Beyond Latency and Tokens

AI agent observability means recording every step inside an agent run, not only the API around it: each model call with its tokens and cost, each tool call with its arguments and result, each document search, each guardrail verdict, and the start and end time of all of them. Latency and token totals tell you the service answered. A step-level trace tells you which path the agent took and why it said what it said.

When an agent starts calling the wrong tools, looping on a failed retrieval, or drifting from its original intent, latency can stay flat while output quality collapses. Observability for agents has to cover the full execution surface: tool calls, model steps, spend, failure loops, approval queues, guardrail hits, and the configuration that was in force when the run started.

These signals should live next to the controls that change the agent. In CreateOS Studio, traces, usage and the audit log sit in the same Govern stage as guardrails and permissions, so the record of a run and the setting you would change to fix it are in one place.

The AI Agent Monitoring Map: One Question per Signal

AI agent monitoring is easier to organise by the question each signal answers than by data source. The service responding is one question. Whether the agent chose the right path, used the right tools, spent within budget, followed policy, and produced a useful result are the others.

This map is also why AI agent monitoring should connect to the AI agent registry. The registry says what an agent is supposed to do. Observability shows whether the live runtime is still doing it.

Tool-Call Traces Are the New Request Logs

A single agent run can span multiple models, knowledge bases, APIs, and internal services. If you only log the final response, you lose the map of how the agent got there. Tool-call traces capture the sequence of decisions, the arguments passed, the responses received, and the duration of each hop. When a production agent starts returning stale data, the trace usually shows a stale passage in a retrieval step or a malformed parameter that never reached the source.

In Studio, a trace is the step waterfall of one run. Each step is an llm call (input, output and cached tokens, and cost in USD), a tool call (which tool, its arguments and its result), a retrieval (the query and what came back), or a guardrail check (which stage, and the verdict). Every step carries a start time, an end time, an attempt number, and an error type when it failed. Each run also stores a frozen copy of what the agent could do when it started: the model, the knowledge bases, the tools and whose accounts they used, and the guardrail and its mode. Edit the agent tomorrow and last week's trace still shows last week's setup, so you debug against the configuration that actually ran.

Intent Drift and Reasoning Visibility

Agents do not just translate prompts into answers. They plan, re-plan, and sometimes reinterpret the user's goal across multiple turns. Intent drift happens when the agent's execution path gradually shifts away from the original task without triggering an error. The response still looks plausible, but it answers a slightly different question or optimizes for a secondary goal. Standard monitoring misses this because the HTTP status code returns 200 and the token count looks normal.

To catch intent drift, you need visibility into intermediate steps, task classifications, and execution traces. Comparing the stated goal at turn one against the inferred goal at turn five can surface drift before users complain. This kind of monitoring is harder to automate than latency checks. It requires baselines, periodic sampling, and sometimes human review of trace summaries. The payoff is that you catch degradation early, before it compounds across a long agentic workflow.

Model Spend by Agent, Model, and Person

Tokens and latency tell you how expensive a call was. They do not tell you why one agent costs three times more this week than last. In production, spend is a quality signal. A spike usually means the agent is retrying more, using a larger model, or generating long outputs that do not help the task. Without attribution by agent, model, and person, finance and engineering talk past each other about cost.

Studio records four token counts (input, output, cache read, cache write) and a cost figure on every model step, and because every member has their own gateway key, spend is attributable to the person who ran the agent. The Usage page is in preview and shows sample data today; the docs name traces as the authoritative source for real token and cost numbers. Access is split on purpose: trace: read lets a finance or platform ops role see durations, tokens, cost and error types across the organization, while trace: read_payloads is needed to read prompts, tool arguments and results. That makes cost work specific. You switch a step to a smaller model, set a model allowlist, or put the high-spend path behind human approval instead of guessing where the budget went.

Failure Loops, Retry Storms, and Approval Queue Health

Agents fail differently than traditional services. A failed tool call does not always end the run. The agent may retry with a slightly different parameter, call a fallback tool, or loop back to the planning step. Left unchecked, this produces retry storms that hammer downstream APIs and inflate costs while the user waits. In a Studio trace, retries appear as sibling steps with the same name and an increasing attempt number, so you can see how many attempts a flaky tool took, not just that it eventually worked.

Human approval queues add another layer. In Studio Workflows, a Wait for Approval step pauses the run until a named person, group or role decides, and the run shows as Waiting. The builder's Runs view flags a run that has waited more than 12 hours as probably stuck, usually an approval nobody has seen. A clogged queue is a production incident, not a workflow delay. The decision, who made it, and the fields they filled in are written to the run's trace and to the audit log, the same record that maintains audit trails for agents.

Incident Triage Checklist

When an AI agent incident starts, the first hour should not be spent correlating dashboards. Use the observability layer to answer these questions in order:

• Which agent, version, environment, and owner are attached to the failing run?
• Did the failure start after an instruction, model, tool, permission, or data-source change?
• Which tool call first diverged from the expected path?
• Did retries, fallback routing, or approval timeouts amplify the failure?
• Did model spend, output quality, or guardrail verdicts change before user impact appeared?
• Did someone change the agent's wiring or get refused access in the audit log?
• Is there a known rollback target, and what traffic or workflows would it affect?

This checklist turns observability into action. It connects traces to ownership, versioning, approval queues, and rollback instead of leaving teams with a pile of logs and no recovery path.

Output Quality, Schema Validation, and Guardrail Hits

Not all agent failures throw exceptions. Some return perfectly formatted JSON that violates your business rules, or natural language that slips past policy filters. Output quality monitoring means treating schema validation failures, semantic consistency, and guardrail triggers as first-class metrics. A rising rate of schema mismatches usually means the model is drifting from the expected structure, often after a minor instruction change or a model swap.

Guardrail hits act as early warnings. When a Studio guardrail fires on input or output, the run stops at that point, the blocked content is not delivered, and a guardrail step is written into the trace showing which stage fired and why. The guardrail connection and mode are frozen onto each run, so a trace from three months ago still shows which guardrail was in force. Monitoring these hits alongside agent guardrails lets you tune thresholds before production and then watch how they behave under real traffic. The goal is not zero guardrail events. It is a guardrail event rate you can explain.

Rollback Signals and Data Source Access

Observability should tell you when to stop as much as when to keep going. Rollback signals are the subset of metrics that justify reverting a change. They might include a sudden jump in tool-call errors, a drop in schema validation pass rates, or an approval queue backing up because the agent started proposing more risky actions. These signals need to map to a specific version and configuration. Studio Workflows freeze a numbered, read-only version on every publish, and each workflow trace shows the exact published version that ran; for agents, the configuration snapshot on each run does the same job.

Once you know which change caused it, you can roll back production agents with the same trace context you used to detect the problem. Access records complement this by showing what changed around the incident. Studio's audit log records configuration and access decisions, including refusals with their reason, so a new tool binding or a run of denied requests from one account is visible next to the trace.

Studio or Sandbox?

If you are watching agents that act on business systems, the traces, usage and audit log described here are part of Studio. The CreateOS Sandbox is for a different job: running untrusted or agent-written code inside isolated microVMs.

What This Gets You, and What It Costs

Full agent observability is not free instrumentation. Every trace and guardrail event generates data, and high-cardinality dimensions like per-person spend or per-step payloads cost storage and query time. Studio caps the number of steps recorded per run and truncates very large step payloads, recording the original size so you know it happened. There is also a labeling burden. Intent drift and output quality often require sampled human review to establish ground truth, which takes time from product and operations teams.

The tradeoff is scope versus noise. A team shipping its first internal agent probably does not need deep review of every step on day one. It needs tool-call visibility, failure loop detection, and spend per agent. As the agent handles more sensitive actions and longer workflows, deeper observability pays off. The constraint is usually not tooling. It is the discipline to treat observability data as part of the deployment lifecycle, not a post-incident archaeology tool.

If your agent metrics still start and end with latency and tokens, you are monitoring the infrastructure, not the execution.

Book a demo to see Studio traces on your own agents, or read how Studio governs agents.