Writing Application Caching Code With LLMs in November 2026: Decide How Stale Is Acceptable, Invalidate on Every Write Path, and Stop Cache Stampedes

Ask an LLM to "add caching" to a slow endpoint and you will get working code within seconds. Usually it looks like this: check Redis for a key, return the value if it exists, otherwise query the database, store the result, and return it. It passes the tests, the endpoint gets fast, and the change gets merged.

The problems show up later. A user updates their profile and still sees the old name. A price change takes an hour to appear. The cache server restarts at peak traffic and the database falls over because every request misses at once. None of these are exotic bugs. They are the predictable result of caching code written without three decisions: how stale the data is allowed to be, which writes must clear it, and what happens when many requests miss at the same moment.

This post covers how to give an LLM those decisions up front, what to look for in the code it returns, and how to test it.

Decide what "stale" means before you prompt

Every cache serves old data some of the time. That is the trade you are making for speed. The model cannot know how much staleness your product can tolerate, so you have to tell it. Go through the data you want to cache and put each item in one of these buckets:

  • Can be minutes or hours old. Public product listings, blog post pages, exchange-rate snapshots shown for information only, aggregated dashboard counts.
  • Can be seconds old. Inventory counts shown on a product page (as long as checkout re-checks the real number), follower counts, search results.
  • Must be fresh for the person who just changed it. A user's own profile, settings, or cart. Other people seeing a slightly old version is usually fine; the person who just saved it seeing the old version looks like a bug.
  • Should not be cached at all. Account balances used to authorize payments, permission checks after a role change, one-time tokens, anything where serving an old answer causes a wrong action rather than a slightly outdated display.

Write those answers into the prompt. A good prompt reads something like:

Add a read-through cache for getProduct(id) using our existing Redis client.
- Product data may be up to 5 minutes stale for anonymous visitors.
- Admin edits go through updateProduct() and deleteProduct(); both must
  clear the cached entry for that id.
- Never cache null results for longer than 30 seconds.
- If Redis is unavailable, fall back to the database and log a warning;
  do not fail the request.
- Show me the key format and the TTL as named constants.

Compare that with "cache this function." The second version leaves every important decision to the model, and it will fill the gaps with whatever is most common in its training data, which is often a cache with no expiry and no invalidation.

Shelves in a pantry stocked with food

A cache is a pantry: fast to reach, but you still need to know how long each thing on the shelf stays good. Image: Vorratskammer by ChristianSW, CC BY-SA 3.0, via Wikimedia Commons.

Give every key a TTL and a clear name

The first thing to check in generated caching code is whether every write to the cache sets an expiry. Code like cache.set(key, value) with no TTL means a single missed invalidation keeps wrong data around forever, or until someone flushes the cache by hand. Even when you invalidate on writes, a TTL is your safety net for the write path you forgot.

Things to look for:

  • A TTL on every set. In Redis that means SET key value EX 300 or the client library's equivalent, not a separate EXPIRE call that can be skipped if the process dies between the two commands.
  • Named constants, not magic numbers. PRODUCT_CACHE_TTL_SECONDS = 300 is easy to review and change. A bare 300 scattered across five files is not.
  • A little random jitter. If you warm thousands of keys at once with the same TTL, they all expire at once. Adding a small random amount, for example 10% of the TTL, spreads the expiries out.
  • Versioned, namespaced keys. A format such as product:v2:{id} lets you change the shape of the cached object by bumping the version instead of hunting down every old entry. It also makes it obvious in a key browser what a key belongs to.
  • Every input in the key. If the response depends on the user's language, currency, or permissions, those values must be part of the key. A model that caches product:{id} for a response that includes a localized price will serve one visitor's currency to everyone. This is the most common serious bug in generated caching code, so read the key construction line by line.

Be careful with per-user data in shared caches

Caching anything that is specific to a logged-in user deserves extra review. If the key leaves out the user ID, one user can see another user's data. The same risk applies to HTTP caching: a response that varies by session must not be stored by a shared cache or CDN. Ask the model to set Cache-Control: private or no-store on authenticated responses, and check that it did not add a public cache header to a route that reads the session.

Invalidate on every write path

The phrase "cache invalidation is hard" mostly means "it is hard to find every place that changes the data." The model only sees the files you give it, so it will happily clear the cache in updateProduct() and miss the bulk import script, the admin batch edit, the background job that syncs prices from a supplier, and the database migration that backfills a column.

Before you ask for invalidation code, list the write paths yourself. A quick search for the table name or the repository method across the codebase usually finds them. Then paste that list into the prompt and ask the model to confirm it handled each one.

When you review the result, check these points:

  1. Delete after the database write commits, not before. If you delete the key first, a concurrent read can miss, load the old value from the database, and put it straight back into the cache before your write lands. Deleting after commit narrows that window a lot.
  2. Prefer delete over update. Writing the new value into the cache from the write path seems efficient, but two concurrent writes can finish in one order in the database and the opposite order in the cache. Deleting the key and letting the next read repopulate it avoids that race in most cases.
  3. Clear related keys too. If a product appears in a cached category list and a cached search result, changing the product may need to clear those as well, or those caches need short TTLs. Ask the model to list every key that contains this data.
  4. Handle deletes. Deleting a record must clear its cache entry. Generated code often only handles updates.

For the "fresh for the person who changed it" bucket, a simple pattern is to skip the cache for that user's own reads for a short time after they write, or to read their own record straight from the primary database. Ask for this explicitly; models rarely add it on their own.

Frederic Remington painting of a cattle stampede at night

When a hot key expires, every request rushes the database at once unless you stop it. Image: The Stampede by Frederic Remington by Frederic Remington, Public domain, via Wikimedia Commons.

Stop cache stampedes

A cache stampede happens when a popular key expires, or the cache is emptied, and many requests miss at the same moment. Each one runs the expensive query and writes the same result back. If the query takes two seconds and you receive 500 requests in those two seconds, the database runs it 500 times. This is how a cache restart turns into a database outage.

Generated read-through code almost never protects against this unless you ask. There are a few standard fixes, and you can ask the model for whichever fits:

  • Request coalescing in the process. Keep a map of in-flight loads keyed by cache key. The first request starts the load; the others wait on the same promise or future. This only protects one process, but it is simple and often enough.
  • A short lock in the shared cache. The first request to miss sets a lock key with SET lock:product:42 1 NX EX 10. Whoever gets the lock loads the data; others wait briefly and retry the cache, or serve a stale copy. The lock must have its own expiry so a crashed worker does not block the key forever.
  • Serve stale while refreshing. Store the value with a "soft" expiry inside the payload and a longer real TTL. When a request sees the soft expiry has passed, it triggers one background refresh and keeps serving the old value in the meantime. This keeps latency flat, as long as the data is in a bucket where a little extra staleness is acceptable.

Also ask what happens when the loader fails. A good implementation does not cache the error, does not retry in a tight loop, and, where acceptable, keeps serving the last known good value.

Cache misses for things that do not exist

If someone requests /product/999999 repeatedly and the product does not exist, a naive cache misses every time and hits the database every time. Caching a "not found" marker for a short TTL fixes this. Keep that TTL short, because the record may be created soon, and make sure creating the record clears the marker.

Plan for the cache being down

Ask the model directly: "What happens if Redis is slow or unreachable?" Common answers in generated code are an unhandled exception that fails the whole request, or a client with a default timeout of several seconds that makes every request slow. Neither is what you want for a cache, which should be an optimization rather than a dependency.

Check for a short connection and command timeout, a fallback to the source of truth on error, and a log line or metric so you notice. If the database cannot handle full traffic without the cache, say so in the prompt; then the fallback needs rate limiting or load shedding too, and that is a design decision you should make, not the model.

Test the cache, not just the function

Most generated tests check that the function returns the right value. That proves almost nothing about the cache. Ask for tests that cover:

  • A second call does not hit the database (use a counter or a mock on the loader).
  • After each write path you listed, the next read returns the new value.
  • Two users with different languages, currencies, or permissions get different cached entries.
  • When the cache client throws or times out, the function still returns the right value from the source.
  • Fifty concurrent calls on a cold key cause one load, not fifty.
  • A not-found result is cached briefly and cleared when the record is created.

In production, add hit and miss counters per key prefix. A hit rate near zero means the key is wrong or the TTL is too short; a sudden drop usually means a deploy changed the key format or the cache was flushed.

A short review checklist

  1. Is every cached item in a staleness bucket you chose on purpose?
  2. Does every cache write set a TTL, with a little jitter?
  3. Does the key include every input that changes the response, including user, locale, and permissions?
  4. Is every write path, including scripts and background jobs, clearing the right keys after commit?
  5. Is there protection against stampedes on hot keys?
  6. Does the code keep working, quickly, when the cache is down?
  7. Do tests prove the cache is actually used and actually invalidated?

Caching is one of those changes an LLM can write very quickly and you can merge very quickly, which is exactly why it deserves a slower review. Give the model your staleness rules, your write paths, and your failure behavior, and the code it returns will be fast for the right reasons instead of fast and quietly wrong.

Comments

Popular posts from this blog

Grok Bot - a step closer to AGI

Tencent Hy4 Preview: Open 770B MoE Built for Real Work

Structured Outputs and Constrained Decoding for Reliable LLM APIs in Production (September 2026)