Idempotency Keys for Production LLM Agent Tool Calls in October 2026: Stop Duplicate Charges, Emails, and Writes When Retries Fire

LLM agents retry. Gateways retry. Clients retry after timeouts. Those retries are often correct for model inference, but they are dangerous for tool side effects. Charge a card twice, send the same email three times, or open duplicate tickets, and you have a production incident that offline evals never caught. In October 2026, the practical fix is the same pattern payment APIs have used for years: idempotency keys that make tool calls safe under replay.

This guide covers why agent retries duplicate side effects, where to attach keys, how to design key scope and TTL, concrete store and middleware patterns, common failure modes, and a production checklist you can apply without rewriting your whole agent stack.

Why retries duplicate tool side effects

Agent loops are not single HTTP requests. A typical production path looks like this: user message → planner/model → tool call → tool HTTP/RPC → model again → maybe more tools. Failures can happen at every hop:

• Gateway timeouts — the model or tool took longer than the proxy deadline. The client or gateway reissues the same agent turn.
• Transport retries — load balancers and HTTP clients retry on 502/503/504 or connection reset. The first attempt may already have succeeded on the tool side.
• Agent framework retries — the orchestrator treats a tool error as transient and calls the tool again with the same arguments.
• Human-in-the-loop resume — a paused workflow is resumed and re-enters the tool step with cached arguments.
• Duplicate user clicks — the UI submits the same “confirm purchase” action twice while the first request is still in flight.

Inference is usually safe to retry: you pay extra tokens, but you do not invent a second bank transfer. Tools are different. Payments, emails, ticket creation, inventory decrements, and CRM writes are not naturally idempotent unless you design them that way.

The failure mode is classic at-least-once delivery colliding with non-idempotent handlers. You asked for “try again if unsure.” The tool heard “do it again.”

What an idempotency key is (and where it lives)

An idempotency key is a client-chosen identifier for a single logical operation. The first request with that key performs the side effect and stores the outcome. Later requests with the same key return the stored outcome instead of repeating the side effect.

For LLM agents, attach the key along the whole path:

1. Client / session — generate a key when the user confirms an irreversible action (checkout, send email, create ticket).
2. Agent runtime — pass the key into the tool-call arguments or metadata so retries of the same step reuse it.
3. Tool gateway / middleware — enforce lookup-before-execute against a shared store.
4. Downstream tool — prefer native idempotency (Stripe-style Idempotency-Key, provider request IDs) when the vendor supports it.

If only the outermost HTTP client retries with a new key each time, you still get duplicates. The key must be stable for the logical operation, not for each network attempt.

Key design: scope, TTL, payload hash, response cache

A usable key strategy answers four questions.

1. Scope

Scope the key so two unrelated operations cannot collide, but retries of the same operation match. A common pattern:

idempotency_key = hash(
  tenant_id + "|" +
  conversation_id + "|" +
  tool_name + "|" +
  logical_op_id
)

logical_op_id should be minted once when the agent decides to call the tool (or when the user confirms), then reused on every retry of that decision. Do not regenerate it inside the HTTP client retry loop.

For multi-tenant systems, always include tenant (or account) identity. Without it, two customers can collide on short keys or shared UUIDs if your generators ever misbehave.

2. TTL

Store records long enough to cover the longest realistic retry window: gateway timeout × max attempts, plus human resume delays if workflows can pause. Twenty-four hours is a common default for payment-like tools; shorter TTLs (minutes) may be enough for read-like or chatty tools. Too short, and a late retry executes again. Too long, and you retain PII-heavy response payloads longer than policy allows—store a redacted result or a pointer when needed.

3. Payload hash

Store a hash of the canonical request body (tool name + normalized arguments) next to the key. On reuse:

• Same key + same payload hash → return cached response (or in-flight wait).
• Same key + different payload hash → reject with a clear conflict error. Do not silently run the new payload.

This catches a dangerous bug: the agent retries “the same step” but with mutated arguments (different amount, different recipient) while accidentally reusing the old key.

4. Response caching

Cache the successful tool response (status, body, downstream IDs) so retries can return an identical result to the model. Also cache a short-lived “in progress” marker so concurrent duplicates wait or get a 409 instead of starting a second charge. On permanent failure, decide whether to cache the error (so retries do not hammer a bad request) or allow a new attempt with a new key after human review.

Concrete patterns you can ship

HTTP Idempotency-Key header

If your tools are HTTP services, accept an Idempotency-Key header (or a JSON field for RPC). Normalize: trim whitespace, enforce max length, require uniqueness per tenant. Many payment and email providers already document this pattern; mirror it in your internal tools so agents have one habit everywhere.

Example contract for an internal charge endpoint:

POST /v1/charges
Idempotency-Key: conv_9f3a:charge:op_4412
Content-Type: application/json

{"amount_cents": 2500, "currency": "usd", "customer_id": "cus_123"}

First call creates the charge and stores {status, charge_id, body_hash}. Second call with the same key returns the same charge_id without creating another payment intent.

Redis or DB-backed store

Use a store both the tool service and the agent gateway can reach:

• Redis — SET key with NX + TTL for the in-progress lock; then SET the final result. Fast and TTL-native. Good when all tool traffic shares a cluster.
• Postgres (or similar) — unique index on (tenant_id, idempotency_key), columns for request hash, status, response JSON, created_at. Stronger audit trail; easier to join to billing and support tickets.

Pseudocode for the critical section:

record = store.get(tenant, key)
if record and record.hash == request_hash:
  if record.status == "done":
    return record.response
  if record.status == "inflight":
    wait_or_409()
if record and record.hash != request_hash:
  raise Conflict("idempotency key reuse with different payload")
store.put_inflight(tenant, key, request_hash, ttl)
try:
  response = execute_tool(request)
  store.put_done(tenant, key, request_hash, response, ttl)
  return response
except Exception as e:
  store.clear_or_put_failed(...)
  raise

Make the “put inflight → execute → put done” path crash-safe. If the process dies after the side effect but before put_done, a pure NX lock that simply expires can still double-execute. Prefer writing a durable “attempt started” row first, and on recovery reconcile with the downstream system’s own idempotent APIs (for example, retrieve charge by metadata) before starting a new attempt.

Tool-wrapper middleware in the agent runtime

Not every tool speaks HTTP headers. Wrap tool invocation in the agent framework:

1. Before calling any side-effecting tool, require an idempotency key in tool metadata.
2. If missing, mint one from conversation_id + tool_call_id (the model’s tool_call_id is useful when the framework retries the same call object).
3. Run the store check, then call the tool, then store the result.
4. Classify tools: read (no key required), side_effect (key required), unsafe (key required + human confirmation).

Keep the wrapper outside the model’s control. The model proposes arguments; your runtime supplies and enforces the key. Letting the model invent keys every turn defeats the purpose.

Failure modes to design for

• Key reuse with different args — treat as conflict, not as a new operation. Alert: this usually means a bug in key minting or argument mutation on retry.
• TTL too short — late retries after workflow resume create duplicates. Align TTL with your longest pause + retry policy.
• TTL too long with sensitive payloads — cached responses may include emails, card last-fours, or ticket bodies. Redact or store references.
• Non-idempotent downstream tools — if the vendor has no idempotency API, create an outbox row first (your UUID), then call the vendor, then mark the outbox sent. Retries resume from the outbox state instead of firing a new vendor call blindly.
• Partial success — email API accepted the message but your process crashed before caching. Prefer vendor message IDs stored in the outbox; on retry, look up by your key before sending again.
• Fan-out tools — one agent step that charges and emails needs either one key for the whole saga with compensating actions, or separate keys per side effect with a clear state machine. A single key that only covers the charge still duplicates the email.
• Clock skew and multi-region stores — eventual consistency between Redis regions can allow two inflight executes. Use a strongly consistent primary for financial tools, or fence with a DB unique constraint.

Practical checklist for production agents

1. Inventory every tool: mark read-only vs side-effecting. Require keys only where needed, but do not guess—payments, messaging, tickets, and writes are in by default.
2. Mint keys once per logical operation; never inside low-level HTTP retry helpers.
3. Propagate keys through gateway → agent → tool; log them next to trace_id and conversation_id.
4. Persist request hash + response (or redacted response) with an explicit TTL policy.
5. Return cached successes identically so the model does not “fix” a duplicate because the second response looked different.
6. Reject key/payload mismatches loudly; page on sudden spikes of those conflicts.
7. Prefer vendor-native idempotency headers when available; wrap everything else with your store.
8. Load-test timeout-and-retry scenarios, not only happy-path tool calls. Kill the tool mid-request and confirm a single side effect.
9. Document how support looks up a charge/email/ticket by idempotency key when a user reports a duplicate.
10. Keep human confirmation for high-impact tools even with keys—idempotency stops duplicates; it does not stop a wrong-but-unique operation.

Wrap-up

Retries are mandatory for unreliable networks and slow model/tool stacks. Duplicate side effects are optional—and preventable. Idempotency keys turn “at least once” delivery into “at most once” effects for the operations that matter: money movement, outbound messages, and durable writes. Scope keys carefully, store payload hashes, cache responses, wrap tools in middleware the model cannot bypass, and test the timeout paths you already know will happen in production. Do that, and agent retries stop being a source of duplicate charges and start being what they should be: boring recovery.