An agent can return a correct answer and still be unhealthy. It may have used too many model calls, retried a tool several times, waited for an approval, or spent more money than the task justified. Traditional request logs do not show enough of that workflow.

The short answer

Trace every agent run as a workflow. Record model turns, tool calls, retries, approvals, duration, tokens, cost, final status, and user outcome. Alert on abnormal paths, not only server errors.

The minimum trace

Give each run a stable ID. Attach the user request, prompt version, model, tool name, input size, output size, duration, and result status. Record whether a step was planned, retried, cancelled, or blocked by policy.

The trace should connect child tool calls to the parent run. Without that relationship, a billing spike can look like hundreds of unrelated requests instead of one agent looping on a failed action.

Metrics worth watching

Track average and p95 turns per run, tool calls per run, retry rate, approval wait time, abandonment, cost per successful task, and the percentage of runs that reach a limit. Also track output quality through a small reviewed sample or task-specific acceptance signal.

I would run a representative 2-hour test before changing a model or tool description. Include successful work, malformed arguments, provider timeouts, user cancellation, and a denied action. The failure path is where runaway behavior usually appears.

Alerts and safe stops

Set limits for turns, tool calls, wall-clock time, tokens, and money. Alert when a run crosses a normal threshold, but do not wait for the alert to stop it. The agent should stop safely, preserve its trace, and tell the user what remains unfinished.

Writing tools should be idempotent. A retry after a timeout must not create duplicate records or charge twice. Store an operation ID, result state, and provider response separately from the model’s summary.

Make traces useful to humans

A dashboard should show the workflow shape, not only a line chart. Let an engineer open a run, see the decision path, inspect tool inputs and outputs under the correct privacy rules, and identify the first abnormal step. Keep prompt and tool versions beside the trace so a regression can be reproduced.

Sources and limits

OpenAI’s Cerebras partnership shows why latency matters for interactive inference. Its Amazon partnership and scaling update provide context on how serving capacity can be distributed across infrastructure. Those announcements do not define your observability design; the right metrics depend on your workflow and risk.

The goal is not to collect every possible event. It is to make each run explainable enough to control, debug, and improve.

A small implementation can be enough

Start with one trace ID that moves through the request handler, model client, tool executor, and result writer. Emit structured events instead of prose-only logs. A useful event includes timestamp, run ID, parent step, action, duration, status, and a redacted result summary.

Create three dashboards: run health, cost, and failure reasons. Run health shows success and abandonment. Cost shows tokens, calls, and spend by workflow. Failure reasons groups timeouts, invalid arguments, policy blocks, and provider errors. This separation prevents a single average from hiding the cause.

Review traces with an engineer and a product owner. Engineers notice retries and missing state; product owners notice confusing stops and unnecessary approvals. Spend 2 hours reviewing real runs after the first release, then turn repeated findings into tests or limits.

Do not confuse activity with progress

More tool calls, longer traces, and larger token counts do not prove that an agent is becoming better. Pair activity metrics with task success, user correction, and time saved. An agent that completes a job in four well-controlled steps is healthier than one that produces a dramatic trace and needs a human to repair the result.

Measure the outcome users actually value.

Keep that outcome beside the infrastructure metrics.