LLM Request Tracing and Cost Observability in Production: OpenTelemetry, Token Attribution, and Failure Replay

Once a generative-AI feature moves beyond a prototype, “the model returned an answer” is no longer enough to explain whether the system is working well. Production teams need to know which prompt version produced a response, how many input and output tokens it consumed, which tools it called, how long each step took, what it cost, and why a request failed. LLM request tracing and cost observability provide that operational view without requiring engineers to read every prompt and completion.

The challenge is that an LLM request is rarely a single operation. A user request may pass through authentication, retrieval, prompt construction, a primary model call, tool execution, a second model call, moderation, and response streaming. Traditional HTTP logs usually show only the endpoint and status code. That is not enough to troubleshoot quality, latency, or unexpected spending.

Why ordinary API logs are insufficient

Most applications begin with logs containing a request ID, HTTP status, elapsed time, and perhaps an error message. This is useful for detecting outages, but it does not explain the behavior of an AI workflow.

• A request can return HTTP 200 while producing an empty, unsafe, or unusable answer.
• A single user interaction can generate several model calls with different models and prices.
• Latency may come from retrieval, tool execution, model queue time, or time spent streaming tokens.
• Retries can silently double or triple the cost of an otherwise successful request.
• Prompt changes can increase input tokens even when application code has not changed.

LLM observability therefore needs three connected views: distributed traces for execution flow, metrics for trends and alerts, and structured events for detailed investigation. The goal is not to store everything. The goal is to retain enough context to answer operational questions quickly.

Use OpenTelemetry spans as the execution backbone

OpenTelemetry is a practical foundation because it already defines a portable model for traces, metrics, and logs. An LLM workflow should create one root span for the user-facing operation and child spans for meaningful stages.

ai.request
├── auth.check
├── retrieval.search
├── prompt.build
├── llm.generate
├── tool.database_query
├── llm.generate
└── output.validate

The root span should represent the business operation, such as support.answer or document.summarize, rather than merely an HTTP route. Child spans should represent units that can independently affect latency, reliability, or cost.

For an LLM span, record standardized attributes where possible and keep application-specific attributes namespaced. Useful fields include:

• gen_ai.system: the provider or gateway used by the application.
• gen_ai.request.model: the requested model identifier.
• gen_ai.response.id: the provider response ID, when available.
• gen_ai.request.temperature and related generation parameters.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.
• gen_ai.response.finish_reasons: stop, length, tool call, or provider-specific outcomes.
• llm.prompt_version: an immutable application prompt or template version.
• llm.operation: chat, extraction, classification, embedding, or reranking.

Do not create a span for every token. Token-level observability means attributing token usage and cost to the correct request, tenant, feature, and workflow step—not emitting one telemetry record per generated token. Per-token spans would create enormous overhead and make the trace system more difficult to use.

Propagate one correlation identity across the workflow

Every request should have a correlation identity that follows it across internal services, queues, retrieval systems, and model providers where propagation is supported. The OpenTelemetry trace ID is useful for infrastructure-level correlation, while application fields provide business context.

Recommended fields include a privacy-safe tenant identifier, user or account classification, feature name, environment, deployment version, and request outcome. Avoid placing email addresses, raw user prompts, access tokens, or document contents in span attributes. If a support engineer needs to locate a request, use a short internal request ID that maps to protected data in a controlled store.

For asynchronous jobs, inject the trace context into the queue message and create a linked consumer span when the job is processed. If a workflow intentionally starts a separate trace, use a span link rather than losing the relationship entirely. This is especially important for background summarization, batch evaluation, and delayed tool execution.

Attribute cost at the model-call boundary

Cost should be calculated at the smallest useful unit: the individual model invocation. A workflow-level estimate alone hides which step is responsible for spending. For each invocation, capture the model, provider, input tokens, output tokens, cached tokens if reported, and any billed special tokens.

A simple cost calculation is:

request cost = input tokens × input price + output tokens × output price + cached tokens × cached-token price

Prices change, providers expose usage differently, and some models have separate rates for cached input, reasoning tokens, tool calls, or batch processing. Do not hard-code pricing logic throughout the application. Store a versioned pricing table or cost configuration with an effective date, currency, provider, model, token category, and unit price.

Record both the provider-reported usage and the calculated estimate. The reported values are the billing source for reconciliation, while the estimate helps teams compare requests consistently. If usage is unavailable because a provider streams incomplete metadata, mark the cost as provisional rather than silently recording zero.

Aggregate costs by dimensions that help decisions:

• Feature or product workflow
• Model and provider
• Tenant, plan, or internal cost center
• Prompt version and deployment version
• Successful, failed, timed-out, and retried requests
• Interactive versus batch traffic

Be careful with high-cardinality dimensions. A unique user ID, prompt hash, or trace ID is useful in traces but can make metrics expensive and slow. Keep metrics dimensions bounded, and use traces or an event store for request-level investigation.

Separate metrics from detailed trace data

Metrics should answer whether a problem is occurring and whether it is getting worse. Useful production metrics include request count, success rate, model error rate, timeout rate, p50 and p95 latency, input tokens, output tokens, cost per request, cost per successful answer, and retry rate.

Histograms are better than averages for latency and token distributions. A model can have a normal average latency while a significant group of users experiences timeouts. Track both total cost and cost per completed business operation. A failed request that consumed many tokens is still a cost event.

Traces should answer why a particular request behaved as it did. Store prompt and completion content only under an explicit privacy policy. In many systems, a redacted prompt summary, content hash, prompt template ID, and token counts are enough for routine troubleshooting. Full content can be stored separately with access controls, retention limits, encryption, and audit logging.

Prevent observability from becoming a data leak

LLM telemetry can contain sensitive information even when the application does not intend it. Prompts may include customer records, source documents, internal instructions, or secrets pasted by users. Treat traces as sensitive production data.

• Redact credentials, payment data, access tokens, and regulated identifiers before export.
• Prefer hashes, classifications, and references over raw content.
• Apply separate retention periods to metrics, traces, and replay artifacts.
• Restrict access to raw prompt and completion data by role.
• Encrypt telemetry in transit and at rest.
• Document whether provider-side request logging is enabled.

Sampling is another important control. Use head sampling for ordinary traffic, but add tail-sampling rules that retain errors, timeouts, unusually expensive requests, and slow traces. You can also retain a small percentage of successful traces for baseline analysis. Sampling decisions should be made after enough span data exists to identify interesting requests.

Design failure replay as a controlled experiment

Failure replay is valuable when a trace contains enough information to reproduce the workflow without exposing production secrets. A replay record might include the prompt version, model parameters, retrieved document IDs, tool input schemas, model response metadata, feature flags, application version, and sanitized request input.

Do not blindly replay a production request against live tools. A tool call that created a ticket, sent an email, or changed a database could execute again. Replay tools should default to dry-run mode, use a sandbox, or replace side effects with recorded responses. Mark replay requests clearly so they do not enter normal billing, analytics, or customer-visible workflows.

For deterministic comparisons, record the exact model identifier and generation parameters. Even then, hosted models can change behavior over time. A replay should be described as “reproduced under these conditions,” not guaranteed to produce byte-for-byte identical text. For structured outputs, compare schema validity, required fields, tool selection, citations, and business assertions—not only string equality.

Instrument retries, fallbacks, and streaming correctly

Retries deserve their own spans or explicit attempt attributes. A root request may succeed after two failed attempts, so the final status alone is misleading. Record the attempt number, retry reason, backoff duration, provider error class, and whether the attempt incurred billable usage.

Fallbacks should also be visible. If a primary model times out and a smaller model answers the request, the trace should show both model calls and the reason for switching. This makes it possible to distinguish resilience from accidental double spending.

For streaming responses, close the model span when the provider has finished delivering the response, not when the first token arrives. Capture time to first token separately from total generation time. Useful measurements include time to first token, inter-token delay, total stream duration, cancellation rate, and tokens emitted before cancellation.

Build a practical rollout plan

Start with one high-value workflow rather than instrumenting every endpoint at once. Add a root business-operation span, model-call child spans, usage capture, error status, and a bounded set of metrics. Verify that the calculated cost reconciles with provider invoices over several days.

Next, add tail sampling for failures and expensive traces, prompt version tracking, retry visibility, and protected replay records. Create dashboards for latency, error rate, tokens, and cost by feature and model. Finally, establish alert thresholds based on business impact: unexpected cost per successful answer, rising timeout rates, sudden output-token growth, or a model fallback rate above normal.

Good LLM observability is not a transcript archive. It is a structured explanation of how an AI request moved through the system, what resources it consumed, and whether the result was useful. OpenTelemetry spans provide the execution map, token-level attribution turns usage into accountable cost, and controlled replay turns vague failures into testable engineering problems. With bounded metrics, privacy-aware traces, and targeted sampling, teams can gain that visibility without drowning operators in logs or exposing the data their AI systems are meant to protect.