What does observability look like for agents?
- 2 min — traces and spans; required attributes
- 3 min — debugging example from a trace
- 3 min — metrics, slicing, error-analysis loop
- 2 min — volume, sampling, privacy and residency
- 2 min — ownership and build vs buy
Agent observability is distributed tracing for a non-deterministic program. Each run is a trace; each model call and tool call is a span carrying inputs, outputs, token counts, cost, latency, model id, prompt version, tool version and errors. The OpenTelemetry GenAI semantic conventions (gen_ai.* attributes such as request model and input/output token usage) give a vendor-neutral schema, so traces flow into whichever backend you run.
The bar: from a trace you can reconstruct why the agent did what it did — what it saw, what it chose, what came back — not just that the request took 30 s.
IT-operations agent closed the wrong incident. The trace shows: search_incidents("db latency checkout") returned two incidents with near-identical titles; the model chose the older one; close_incident ran without a confirmation step because closing was classed as low risk. Two fixes follow directly — rank search results by recency and service match, and require the incident id to be echoed back in a confirmation for close actions — plus a new regression task. Without span-level payloads this would have been logged as “model confusion”.
Aggregate traces into metrics sliced by model, prompt version and tool version: task success, steps per task, guard-rail exits, tool error rate, cost and latency distributions — so a regression shows up within hours of a change. Sample traces into a weekly error analysis and back into the golden set. Volume is real: payloads dominate storage, so keep 100% of span metadata, 100% of payloads for failed or guard-rail runs, and a 5–10% sample of successful payloads. Traces contain user data and tool output, so apply PII redaction before storage, region-pinned storage where residency applies (EU, India), retention limits (e.g. 30 days hot), and access control on who can read payloads.
Org scope. The platform team owns instrumentation (one SDK, OTel GenAI conventions), the pipeline and sampling; privacy/security owns redaction rules, retention and payload access; product teams own their dashboards and a rotating weekly error-analysis duty. Build vs buy: buy an LLM-tracing backend (Langfuse, LangSmith, Arize Phoenix, Datadog) or route OTel into the existing stack — don't build the UI. Failure mode: logging only the final answer, which makes every agent incident unreproducible.
Assumptions: 1M runs/day × 25 spans × 6 KB payload ≈ 150 GB/day raw. Keep metadata (≈0.5 KB/span → 12.5 GB/day) for all runs, payloads for the 8% failed/guard-rail runs (12 GB/day) and a 5% sample of the rest (≈6.9 GB/day) → ≈ 31 GB/day, ≈ 0.95 TB over 30 days hot — ~5× less than keeping everything.
What the answer sounds like at each level
Log every model and tool call with inputs, outputs, tokens and latency so you can debug runs.
Uses span-level traces with versions, derives sliced metrics, samples payloads by outcome, and feeds error analysis back into evals.
Standardises on OTel GenAI conventions across teams, splits ownership of pipeline, privacy controls and dashboards, and sizes storage and retention by region.
Follow-ups the interviewer will push
1Now 10× runs. Storage bill triples monthly. What do you cut?
Payload sampling, not metadata: keep all spans' attributes, all failed-run payloads, and sample successes at 5%. Compress and tier to object storage after 7 days.
2A regulated tenant forbids storing prompts. Can you still debug?
Store hashed or redacted payloads plus structured metadata (tool names, argument shapes, ids, token counts), and offer tenant-initiated capture for a debugging window with their consent.
3What alert would have caught the wrong-incident closure earlier?
A per-tool anomaly on close_incident rate relative to search_incidents with multiple near-duplicate hits, plus a weekly review of reversed actions (reopened incidents) as a quality signal.
Red flags interviewers listen for
- Logs only final answers or only aggregate latency.
- No prompt/model/tool version on spans, so regressions can't be attributed.
- Stores raw user data in traces with no redaction, retention or access policy.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Instrumentation | Span-level with versions, tokens, cost; standard conventions. |
| Use of data | Sliced metrics, error-analysis loop into evals. |
| Privacy | Redaction, retention, residency, payload access control. |
| Org scope | Platform/privacy/product split; buy the backend. |
Sources
- Semantic conventions for generative AI systems — OpenTelemetry, 2025spec