Observability & traces
Being able to answer new questions about production from its telemetry. For LLM systems: traces per request with prompts, tool calls, tokens, cost and quality.
Observability means you can figure out why your system did something without shipping new code to find out. You get there by recording three kinds of telemetry: logs (events), metrics (numbers over time, like latency) and traces (the path of one request through every step). For an AI system, a trace shows the retrieval query, the chunks found, the prompt sent, the model's answer, tool calls, token counts and cost, so when a user says 'it gave me a wrong answer', you can see exactly why.
A flight data recorder: after an incident you replay what every instrument read and every control the pilot touched, instead of guessing.
Logging prompts and completions in plaintext to the same log pipeline as everything else. It turns your observability stack into the largest unredacted copy of customer data you have, with the weakest access controls.
Instrument with OpenTelemetry and its GenAI semantic conventions (still marked Development as of Oct 2026, so pin the version): spans for model calls carry gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and the provider name; agent and tool steps are child spans. One trace per user request, with retrieval, rerank, generation and each tool call as spans, lets you compute per-request cost and find which step owns the latency. Content (prompts, completions) is high-value and high-risk: capture it as span events or a separate store with PII redaction, tenant scoping and a shorter retention than metrics. Track quality as telemetry, not just ops: attach online eval scores (judge score, citation validity, refusal) to traces so you can slice quality by prompt version, model and tenant. Backends: Langfuse, Arize Phoenix, Honeycomb, Datadog LLM Observability, Grafana Tempo; most ingest OTLP. Use p50 / p95 / p99 latency (p50/p95/p99), never averages, and tie alerts to SLOs rather than raw thresholds.
OpenTelemetry SDK with GenAI semantic conventions (pinned version); OTLP to Langfuse, Phoenix, Honeycomb or Datadog; spans for retrieval, rerank, generation, each tool; attributes for model, prompt version, tokens, cost, tenant id; content capture redacted, 100% of errors and negative feedback + 5% sample; content retention 30 days, metrics 13 months.
An insurance-claims assistant gets a complaint: a claim was marked 'not covered' wrongly. The trace for that request shows: retrieval returned 8 chunks in 120 ms, the policy rider chunk was ranked 11th and cut by top-k=8, generation used 6,200 input tokens and 180 output tokens ($0.021), and the answer cited a general exclusion. The fix (raise rerank depth to 50, keep top 10) is validated by filtering traces for claim.type=water_damage and re-running 300 of them offline.
Day one I'd emit OTel traces with GenAI conventions for every request, retrieval and tool call as child spans, token counts and cost as attributes, and online quality scores attached, with prompt content stored redacted and tenant-scoped. The follow-up: 'how do you keep it affordable?' Sample content capture (100% of errors and thumbs-down, 5–10% of the rest), keep metrics at 100%.
Numbers worth knowing
- Content capture rate
- 100% errors/negative feedback, 5–10% otherwisekeeps storage and PII exposure bounded
- Trace size with content
- ~10–50 KBRAG request with 8 chunks and a 6k-token prompt
- Tracing overhead
- <1 ms per spanasync batch export; synchronous export adds network latency
How to decide
| When | Do this |
|---|---|
| Starting a new LLM service | OTel GenAI spans from day one; retrofitting costs more. |
| Prompts may contain PII or regulated data | Redact before export, separate content store, tenant-scoped access, 30-day retention. |
| Need to slice quality by version | Attach prompt version, model and eval scores as span attributes. |
| Trace volume cost grows faster than traffic | Tail-based sampling: keep slow, failed and flagged traces. |
Cost maths
Storage = requests × capture rate × trace size. 1M requests/day × 10% × 30 KB = 3 GB/day of content; 30-day retention ≈ 90 GB. Hosted LLM-observability tools typically price per trace or span ingested: at an assumed $0.50–2 per 1k traces (illustrative), 100k captured traces/day ≈ $1.5–6k/month. Metrics-only for the other 90% is cheap. Budget observability at roughly 5–10% of model spend.
Operating it
- Can't answer 'which prompt version caused this?' from telemetry alone
- Observability bill >15% of model spend: add tail sampling
- Multiple teams with their own LLM logging formats: standardise on OTel GenAI attributes
- p95 end-to-end latency vs SLO: alert at 2× burn rate over 1 h
- Cost per request p50/p95: alert on +30% week over week
- Input tokens per request by prompt version: alert on +50% after a deploy
- Online quality score (citation validity, judge score): alert on a 2-point drop over 24 h
When it goes wrong
Model spend for a financial-filings assistant rose 3× in one week with flat traffic.
Billing alert; with no per-request token telemetry, it took two days to find the source.
A retrieval change raised top-k from 8 to 40 for one query class, multiplying input tokens; nothing recorded tokens per request or per step.
OTel spans with gen_ai.usage.* per call, cost computed per trace, dashboards by prompt version and query class, and a cost-per-request alert.
Where it shows up
Sources
- Semantic conventions for generative AI systems — OpenTelemetry, 2025spec
- Site Reliability Engineering, ch. 6: Monitoring Distributed Systems — Google, 2016book
Check yourself
Three questions. Every option is explained once you answer.