LLM Request Tracing and Cost Observability in Production: OpenTelemetry, Token Attribution, and Failure Replay
LLM Request Tracing and Cost Observability in Production: OpenTelemetry, Token Attribution, and Failure Replay Once a generative-AI feature moves beyond a prototype, “the model returned an answer” is no longer enough to explain whether the system is working well. Production teams need to know which prompt version produced a response, how many input and output tokens it consumed, which tools it called, how long each step took, what it cost, and why a request failed. LLM request tracing and cost observability provide that operational view without requiring engineers to read every prompt and completion. The challenge is that an LLM request is rarely a single operation. A user request may pass through authentication, retrieval, prompt construction, a primary model call, tool execution, a second model call, moderation, and response streaming. Traditional HTTP logs usually show only the endpoint and status code. That is not enough to troubleshoot quality, latency, or unexpected spendin...