Monitoring context systems in production
Research Report 7.3: Observability & Debugging
Research Report 7.3: Observability & Debugging
When your web server goes down, your monitoring dashboard lights up red. When your LLM agent starts hallucinating, your dashboard shows green. The response came back. The latency was normal. The status code was 200. Everything looks fine - except the answer is completely wrong.
This is the fundamental observability challenge for LLM systems: failures don't look like failures. They look like normal outputs that happen to be wrong. Traditional monitoring (uptime, latency, error rates) is necessary but nowhere near sufficient. You need instrumentation that can distinguish between a model producing correct output quickly and a model producing wrong output quickly.
If you're running LLM pipelines without LLM-specific observability, you're flying blind. This report covers what to monitor, how to monitor it, and how to debug problems in systems that are probabilistic by design.
Why LLM Observability Is Different
Three properties of LLM systems make traditional monitoring insufficient.
Non-determinism is the norm. The same input can produce different outputs across runs. This means you can't write assertions like "input X should produce output Y." You need statistical monitoring that tracks output distributions over time and flags when the distribution shifts.
Failures are semantic, not structural. A hallucinated response has the same HTTP status code, the same response format, and similar latency as a correct response. Detecting it requires evaluating the content of the output, not just its structure.
Context makes everything harder. The same prompt with different context window contents produces wildly different results. Debugging a failure means understanding not just the prompt but the full context assembly - what documents were retrieved, what conversation history was included, what system instructions were active.
The Three Pillars for LLM Systems
Structured Logging
Every LLM call should log:
- Full prompt (or a hash if privacy-sensitive): What exactly was sent to the model
- Full response: What exactly came back
- Token counts: Input tokens, output tokens, total cost
- Timing: Time to first token, total generation time
- Model metadata: Model version, temperature, max tokens, any sampling parameters
- Correlation ID: A trace ID linking this call to the broader workflow
The logging overhead is real - LLM prompts and responses can be large - but without it, debugging is guesswork. Log everything in development, apply sampling in production based on your storage budget.
Structure your logs so they're queryable. You need to answer questions like: "Show me all calls to model X in the last hour where output tokens exceeded 2000 and latency was above 5 seconds." Unstructured log lines make this impossible.
Distributed Tracing
Multi-agent workflows involve chains of LLM calls, tool invocations, and data retrievals. A single user request might trigger 10-50 individual operations across multiple agents. Without distributed tracing, you can't reconstruct the sequence of events that produced a given output.
OpenTelemetry has become the standard for LLM tracing, with spans covering:
- Agent-level spans: Which agent handled what task
- LLM call spans: Individual model invocations with full prompt/response
- Tool spans: External API calls, database queries, file reads
- Retrieval spans: Vector searches, document fetching, context assembly
Each span carries the correlation ID that connects it to the parent trace. When a user reports a bad output, you pull the trace and see exactly what happened: which agent was invoked, what prompt it received, what the model returned, what tool calls were made, and how the final response was assembled.
Metrics Collection
Track metrics at multiple levels:
System-level metrics cover the infrastructure: API availability, latency percentiles (p50, p95, p99), error rates, rate limit headroom, cost per request.
Quality metrics evaluate output correctness: hallucination detection rates, semantic similarity to expected outputs, user satisfaction signals (thumbs up/down, edits, regeneration requests), factual accuracy on verifiable claims.
Pipeline metrics monitor workflow health: end-to-end completion rates, retry frequencies, fallback activation rates, agent utilization, task queue depths.
| Metric Category | What to Track | Alert Threshold |
|---|---|---|
| Latency | p50, p95, p99 per model/agent | p95 > 2x baseline |
| Error rate | API errors, validation failures | > 5% sustained |
| Quality score | Semantic similarity, user ratings | Rolling average drops 10%+ |
| Cost | Tokens per request, cost per completion | > 2x budget projection |
| Throughput | Requests per second, completions per minute | < 50% of capacity |
Debugging Non-Deterministic Systems
Traditional debugging relies on reproducibility: reproduce the bug, inspect the state, fix the code. LLM systems break this assumption. The same inputs might produce correct output 9 out of 10 times and wrong output once.
Capture Everything at the Boundary
Since you can't reproduce LLM calls deterministically, your best strategy is capturing complete state at every boundary. Log the exact prompt, the exact response, and the full context at the time of the call. When a bug is reported, you have the raw data to analyze rather than trying to reproduce it.
Replay Testing
Build infrastructure to replay captured inputs through the pipeline. Take a real prompt that produced a bad output and run it through the system again. If the problem reproduces, you have a reproducible case. If it doesn't, you know the issue is related to model non-determinism or transient conditions (rate limiting, timeout, different model checkpoint).
Differential Analysis
When outputs change unexpectedly, compare the inputs. Did the prompt change? Did the retrieved context change? Did the model version change? Did the system instructions change? Systematic differencing narrows the search space from "something went wrong" to "this specific change caused the regression."
Prompt Debugging
Many LLM failures trace back to prompts that work in most cases but fail in edge cases. Debug prompts by:
- Testing with adversarial inputs that probe edge cases
- Checking whether the prompt is sensitive to the order of examples
- Verifying that the prompt works with both short and long contexts
- Testing with inputs that are similar to known failure cases
Alert Design
LLM systems require thoughtful alert design to avoid alert fatigue while catching real problems.
Anomaly-based alerts trigger when metrics deviate from historical baselines. These catch novel failure modes that threshold-based alerts miss. The tradeoff is higher false positive rates, especially during legitimate usage pattern changes.
Quality-based alerts trigger when output evaluation scores drop below acceptable thresholds. These require automated quality evaluation (LLM-as-judge, embedding similarity, or structured output validation) running on a sample of production traffic.
Budget alerts trigger when cost or token consumption exceeds projections. Runaway token usage is often the first visible signal of a deeper problem - an agent stuck in a retry loop or a prompt injection causing verbose outputs.
The key principle: alert on actionable conditions. An alert that fires when a human can't do anything about it creates noise. Pair every alert with a documented response procedure.
Visualization and Dashboards
Effective LLM dashboards show three things at a glance:
System health: Are all components operational? Are latencies normal? Are error rates within bounds? This is the traditional ops dashboard, adapted for LLM infrastructure.
Quality trends: Are output scores stable, improving, or degrading? Quality drift is the silent killer of LLM systems - gradual degradation that nobody notices until it's severe.
Cost and usage: How much are you spending? What's driving the cost? Which agents or pipelines are the most expensive? Cost anomalies often signal behavioral problems before quality metrics catch them.
Trace visualization - showing the full call graph for individual requests - is essential for debugging but too detailed for dashboards. Make it accessible from the dashboard with one click: see the summary view, then drill into specific traces when something looks wrong.
Common Pitfalls
Monitoring only infrastructure. Uptime and latency don't tell you whether the model is producing good outputs. Add quality monitoring or you're only catching half the failure modes.
Logging too little in production. Storage costs for full prompt/response logging are real, but the cost of debugging without logs is higher. At minimum, log hashes of prompts and responses so you can identify patterns, with full content logging for a random sample.
Alert fatigue. Too many alerts with too low thresholds means the team ignores them all. Start with a small number of high-confidence alerts and add more only when you have proven response procedures.
No baseline. You can't detect anomalies without a baseline. Spend the first weeks of any deployment collecting metrics to establish what "normal" looks like before setting alert thresholds.
Treating observability as optional. Teams often add observability after problems occur in production. By then, you have no historical data to compare against. Build observability into the pipeline from day one.
Connections
Observability is the foundation for the Error Propagation and Resilience patterns - you can't contain errors you can't see. The circuit breakers, bulkheads, and fallback mechanisms described there depend on the monitoring signals described here.
The Context Management Best Practices guide covers how context assembly affects output quality - a major source of the quality drift that observability must detect. For multi-agent systems, the Agent Communication Protocols report describes the message flows that distributed tracing needs to capture.