Production LLM agents need an incident budget: OpenTelemetry tracing, eval gates, and span limits

Your agent doesn’t need to be smarter. It needs an incident budget.

If you can’t answer “what happened?” during an incident without either guessing or opening a 200-span trace full of prompt blobs, you don’t have an agent system — you have an unbounded side-effect machine.

The failure mode: trace firehoses and unactionable detail

The common pattern looks like this:

• You add “agent tracing”. It’s helpful for a week.
• Then you ship tool-calling, retries, multi-step plans, retrieval, eval checks, fallbacks.
• Suddenly every user request creates a distributed trace that is both expensive and hard to read.
• On-call starts turning off instrumentation or sampling “until we fix it”, and you’re back to debugging by log grep.

This is not a tooling problem so much as a budgeting problem.

In production, agent observability has two opposing requirements:

• You need end-to-end causality (the chain from request → agent decision → tool call → side effect).
• You need bounded telemetry (hard ceilings on volume, cardinality, and sensitive payload capture).

The playbook below is the smallest set of practices I’ve found that keeps agentic automation operable as it scales.

Principle 1: OTel-first, not dashboard-first

If you make “an LLM observability platform” the centre of your architecture, you usually end up with parallel tracing systems:

• your existing services (API, workers, DB, queues) are in OpenTelemetry
• your agent traces are in a separate UI, with different IDs, different sampling, different retention

That split is exactly what makes incidents painful.

An OTel-first approach means:

• The agent run is just another trace in the same distributed tracing fabric.
• Tool calls are child spans.
• The model call is a span.
• Anything that touches the outside world (payments, email, file writes) is a span with explicit attributes.

OpenTelemetry’s core spec gives you the portability you want here: instrument once, ship over OTLP to whatever backend you standardise on. OpenTelemetry also defines semantic conventions and SDK behaviour (including what it means for a span to be “recording” versus effectively dropped). (opentelemetry.io)

If you are using an LLM-focused product, treat it as an OTLP consumer or producer rather than the source of truth. Several agent/LLM observability tools explicitly support OpenTelemetry ingestion/export now (for example, Langfuse supports exporting via OTLP and documents OpenTelemetry compatibility, and LangSmith documents tracing with OpenTelemetry clients). (langfuse.com)

What “OTel-first” buys you in practice

• One trace ID across API gateway → agent → tools → DB.
• One place to set sampling, PII policies, retention, and cost controls (typically at the Collector).
• A migration path: you can switch backends without rewriting instrumentation.

When not to do this

Don’t start with full OTel if you’re still proving the product and can’t justify any observability spend. If you’re in that phase, use minimal structured logs and manual correlation IDs.

But once the agent triggers real side-effects, you’ve crossed into an on-call world. At that point, the cost of not having operable traces is higher than the cost of setting up basic OTel plumbing.

Principle 2: Model the agent as spans you can reason about

If every step is “just a span with arbitrary JSON”, your traces will be noisy and impossible to query.

You need a small, opinionated span taxonomy. You can implement this yourself, or adopt existing conventions designed for AI traces.

One option is OpenInference semantic conventions, which define attributes such as openinference.span.kind with values like LLM, TOOL, AGENT, RETRIEVER, EVALUATOR, and describe how to represent prompts/templates and LLM calls in OpenTelemetry-friendly attributes. (github.com)

Whatever you choose, make it consistent enough that you can answer questions like:

• “show me tool spans that returned non-2xx and were retried”
• “show me agent runs where a guardrail failed closed”
• “show me model calls above X latency correlated with queue backlog”

A minimal span hierarchy that works

• Root span: agent.run (or similar)
• Child spans:
• llm.call (one per provider call)
• tool.call.<tool_name> (one per tool invocation)
• retrieval.query / rerank (if applicable)
• guardrail.check / eval.check (if you run quality gates inline)

Make the span names boring. Put richness in attributes.

Principle 3: Put a hard cap on span volume (span limits + sampling)

You need two layers of volume control:

1. Span limits: keep any single span from becoming a data dumpster.
2. Sampling: keep the overall trace volume bounded.

These are different problems.

Span limits: stop “just add it to the span attributes”

OpenTelemetry supports limiting attributes/events/links per span via SDK configuration (often surfaced as “SpanLimits” in language SDKs, and as config types like event_attribute_count_limit and link_attribute_count_limit). (opentelemetry.io)

For agents, span limits matter because payloads can explode:

• prompt + system prompt + tool schema + retrieved docs
• tool outputs (HTML pages, long JSON)
• multi-turn scratchpads

If you don’t cap this, you’ll either:

• blow up your tracing bill
• hit backend ingestion limits
• or quietly drop spans and lose the very evidence you needed

Practical approach:

• Default to capturing metadata, not full content.
• Store full prompts/responses in your own storage only when you have an explicit need (debug mode, sampled subset, or an incident capture path).
• Put explicit truncation rules in code (so behaviour is stable across SDK/backend changes).

Sampling: head sampling for cost, tail sampling for incidents

Head-based sampling decides at the start of a trace whether it will be sampled. Tail-based sampling evaluates after the trace completes and can make decisions using outcomes like latency or errors. (opentelemetry.netlify.app)

For agent systems, tail sampling is often the difference between:

• keeping “boring success” traces cheap
• while always retaining “we almost charged the customer twice” traces

OpenTelemetry Collector supports tail sampling with policies such as sampling by status_code and latency, and it evaluates completed traces against configured policies. (opentelemetry.io)

Trade-off you must accept:

• tail sampling requires holding traces until the decision is made, which increases collector memory/latency complexity
• head sampling is simpler but will miss rare failures unless your sample rate is high

A sane default strategy:

• Head sample at a low, stable rate across all traffic.
• Tail sample 100% of:
• traces with ERROR status
• traces above a latency threshold
• traces with specific “incident markers” (more on this below)

Principle 4: Design “incident markers” into your spans

Agents fail in ways traditional services don’t:

• tool output is semantically wrong but syntactically fine
• the model returns a plausible plan that violates policy
• a retry loop amplifies cost
• the agent thrashes between tools

If you rely on “HTTP 500” as your only signal, you will miss most incidents.

Add explicit attributes that make tail sampling and alerting possible:

• agent.outcome: success | blocked | partial | failed
• agent.block_reason: enum you control
• agent.retry_count
• agent.tool_call_count
• agent.cost_bucket: coarse (not exact billing)
• agent.user_visible_error: boolean

Then you can drive policies such as:

• retain traces where agent.outcome != success
• retain traces where agent.tool_call_count > N
• retain traces where agent.retry_count > 0 (or above threshold)

This is the “incident budget” idea: you decide upfront which classes of weirdness are worth keeping at high fidelity.

Principle 5: Regression evals are release gates, not dashboards

The fastest way to ship a broken agent is to treat evals as a report you read “sometimes”.

In a production team, evals should behave like tests:

• they run on every meaningful change (prompt edits, tool schema changes, model/provider changes)
• they fail the build when you regress beyond an agreed threshold
• they produce artefacts you can inspect (inputs, outputs, rationale, tool traces)

OpenAI’s own writing on evaluation practice stresses that harness choice and validity checks are part of the evaluation result — i.e. you don’t get to claim quality without being explicit about how you measured it. (openai.com)

What to gate on (and what not to)

Gate on things that correlate with user harm or operational pain:

• policy compliance (PII leakage, disallowed actions)
• tool correctness on a fixed suite (did it call the right tool with the right arguments?)
• stable JSON/tool-call schema adherence
• “must not do” behaviours (e.g. sending emails without confirmation)

Do not gate on a single fuzzy “quality score” unless you’ve proven it is stable and resistant to prompt drift.

The operable workflow: promote incidents into eval cases

The flywheel you want:

1. Incident happens (or near-miss).
2. You capture the trace ID and the minimal artefacts needed to reproduce.
3. You turn that into an eval case.
4. The eval becomes a permanent regression guard.

This is how you prevent the same class of failure from paging you again in three weeks.

A concrete rollout plan (2–4 weeks, not 6 months)

1. Instrument one critical agent path end-to-end with OpenTelemetry (root span, tool spans, model spans).
2. Decide your span taxonomy and incident markers.
3. Add span limits and truncation in code (not just in the backend).
4. Deploy an OpenTelemetry Collector and implement sampling: - low-rate head sample for baseline visibility - tail sampling policies for errors/latency/outcomes (opentelemetry.io)
5. Stand up a small eval suite: - 20–50 high-leverage cases - run in CI - fail closed on schema/policy violations
6. Add an “incident to eval” playbook to your on-call runbook.

Where codeversols fits (briefly, honestly)

If you have a working agent prototype but you’re about to expose it to real users or real side-effects, this is the point where engineering rigour matters: instrumentation, sampling, and eval gates are not glamour work, but they’re what keep a small team from getting buried.

At codeversols we build production software across web, mobile, AI, cloud and design. When we help with agentic systems, we typically focus on the unsexy parts first: OTel instrumentation that matches existing services, collector configuration that keeps costs bounded, and CI eval gates that make releases predictable.

Close

Production agents are not “apps with a model call”. They’re distributed systems that happen to include a model.

Treat observability and evaluation as an incident budget: define what you will retain, what you will gate, and what you will refuse to ship without. Once you do, you can iterate on intelligence without turning on-call into archaeology.