Writing Rate Limiting Code for Your Own API With LLMs in November 2026: Key on the Right Identity, Keep Counters Where Every Instance Can See Them, and Return 429 With Retry-After

Sooner or later every public endpoint needs a rate limit. A login form gets hammered by a password-guessing script, a single customer's runaway loop eats your database, or an expensive report endpoint gets called a thousand times a minute. Asking an LLM to "add rate limiting" is a reasonable first move, and it will produce something that looks finished in seconds. The trouble is that the most common generated version counts requests in a dictionary inside one process, keys on whatever IP header shows up first, and returns a bare error with no hint about when to come back.

This post covers how to get an LLM to write server-side rate limiting that actually holds up: what to decide before you prompt, which generated patterns to reject, and how to test the limiter before real traffic does it for you. (If you are on the other side, calling someone else's rate-limited API, that is a different problem: back off and honor their Retry-After.)

A row of subway turnstiles at 86 Street station

A turnstile lets people through one at a time. A rate limiter does the same for requests. Image: 86 Street turnstiles by Fabio Sola Penna, CC BY 2.0, via Wikimedia Commons.

Decide what you are protecting before you prompt

"Add rate limiting" is not a spec. Different goals need different limits, and the model cannot guess yours. Write down, for each endpoint or group of endpoints:

  • The threat or cost. Brute-force login attempts, scraping, accidental client loops, or a slow query you cannot afford to run often.
  • Who is being limited. An anonymous visitor, a logged-in user, an API key, a whole organization, or a target account (for login, you usually want to limit attempts against an account, not just attempts from an IP).
  • The number. For example, 5 failed logins per account per 15 minutes, 100 requests per API key per minute, 10 report exports per user per hour. Pick numbers from your real traffic if you have logs; a limit far below normal usage just breaks your own app.
  • What happens when the limiter's storage is down. Fail open (allow requests) or fail closed (reject them). Most general APIs fail open so a cache outage does not become a full outage; login and other security-sensitive endpoints may reasonably fail closed.

Put that table in the prompt. A model given "limit POST /login to 5 failures per email per 15 minutes and 50 attempts per IP per 15 minutes, fail closed" writes very different code from one given "add rate limiting."

Key on the right identity

The rate limit key is the part LLMs most often get wrong, and the mistake is usually silent.

Do not trust client IP headers blindly

Generated code frequently does something like ip = request.headers.get("X-Forwarded-For", request.remote_addr). Any client can send an X-Forwarded-For header with any value, so an attacker just puts a random IP in every request and gets a fresh bucket each time. Rate limiting is now decorative.

The fix is to only read forwarded headers that your own proxy or load balancer sets, and only take the address your proxy appended. Most frameworks have a setting for this (for example, a trusted-proxy count or list of proxy addresses), and many hosting platforms document which header carries the real client IP. Tell the model exactly what sits in front of your app: "We run behind one load balancer that appends the client IP to X-Forwarded-For; use the framework's trusted proxy setting with one hop, never parse the header by hand."

Prefer authenticated identity when you have it

  • Logged-in traffic: key on the user ID or API key, not the IP. Many users can share one IP (offices, mobile carriers), and one user can switch IPs.
  • Anonymous traffic: IP is often the only option. For IPv6, consider grouping by prefix (commonly a /64), because a single client can easily rotate through many addresses inside its own block.
  • Login and password reset: use more than one key. A per-account limit stops slow guessing spread across many IPs; a per-IP limit stops one source from trying many accounts.

Also normalize the key. If you limit by email address, lowercase and trim it first, or Alice@example.com and alice@example.com get separate buckets.

Keep counters where every instance can see them

The single most common generated limiter stores counts in a module-level dictionary. That works on a laptop and fails quietly in production for three reasons: each server instance or worker process has its own dictionary, so your real limit is the configured limit times the number of processes; counts vanish on every deploy or restart; and the dictionary may never evict old keys, so memory grows forever.

If you run more than one process, the counters need to live in shared storage. Redis is the usual choice because it has fast atomic increments and key expiry. Many frameworks and API gateways also ship rate limiting that already handles this; if yours does, tell the model to use it instead of hand-rolling one.

Watch for the non-atomic increment

A typical generated Redis limiter looks like this:

count = r.incr(key)
if count == 1:
    r.expire(key, 60)

If the process crashes or the connection drops between those two calls, the key never gets an expiry and that client stays blocked until someone deletes the key by hand. Make the increment and expiry atomic. One simple way is a short Lua script, which Redis runs as a single unit:

import redis

r = redis.Redis()

FIXED_WINDOW = r.register_script("""
local current = redis.call("INCR", KEYS[1])
if current == 1 then
  redis.call("EXPIRE", KEYS[1], ARGV[1])
end
local ttl = redis.call("TTL", KEYS[1])
return {current, ttl}
""")

def check_limit(key: str, limit: int, window_seconds: int):
    """Return (allowed, retry_after_seconds)."""
    current, ttl = FIXED_WINDOW(keys=[key], args=[window_seconds])
    if current > limit:
        return False, max(ttl, 1)
    return True, 0

Treat this as a starting point, not a drop-in: you still need to decide the key format (for example rl:login:email:alice@example.com), wrap the call in error handling that implements your fail-open or fail-closed choice, and set a short timeout on the Redis client so a slow cache does not stall every request.

Pick an algorithm on purpose

Ask the model to name the algorithm it used, then check it matches what you need:

  • Fixed window (like the script above): simple and cheap, but a client can send a full limit at the end of one window and another full limit at the start of the next, so short bursts can reach roughly twice the limit.
  • Sliding window log: stores a timestamp per request and counts the ones in the last N seconds. Accurate, but memory grows with the limit.
  • Sliding window counter: blends the current and previous fixed windows to approximate a sliding window with two counters. A good default for most APIs.
  • Token bucket: tokens refill at a steady rate up to a maximum; each request spends one. Allows controlled bursts while enforcing an average rate, which suits API keys well.

For a login form, fixed window is usually fine. For a paid API where customers notice burst behavior, a token bucket or sliding window counter is worth the extra code.

A ramp meter signal controlling on-ramp traffic to Interstate 5 in Downey, California

A ramp meter releases cars at a steady rate, much like a token bucket releases requests. Image: Ramp control meter in California by Eric Polk, CC BY-SA 4.0, via Wikimedia Commons.

Return 429 with Retry-After

When a client is over the limit, the response should tell it so, clearly and consistently:

  • Status 429 Too Many Requests. Not 403, not 500, and not 200 with an error body. Clients and monitoring tools treat 429 specially.
  • A Retry-After header with the number of seconds until the client can try again. Well-behaved clients and many HTTP libraries use it to wait the right amount of time instead of retrying immediately.
  • Optional remaining-quota headers. Many APIs send X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset on every response. These are a widespread convention rather than a formal standard (the IETF has been working on standardized RateLimit header fields), so pick one format, document it, and keep it consistent.
  • A short, honest body. "Too many login attempts. Try again in 12 minutes." Do not reveal whether an account exists through different messages or limits for real and fake emails.

Also decide what counts. For login, count failed attempts (and possibly reset the per-account counter on success). For an expensive endpoint, count every call. Make sure rejected requests are cheap: the limit check should run before you hit the database or call another service, otherwise the limiter protects nothing.

A prompt that gets a better first draft

Add server-side rate limiting to this Flask app (code below).

Context:
- Runs as 4 gunicorn workers on 3 instances behind one load balancer.
- The load balancer appends the client IP to X-Forwarded-For.
  Use the framework's trusted-proxy setting with 1 hop; never parse
  the header by hand.
- Redis is available at REDIS_URL.

Limits:
- POST /login: 5 failed attempts per normalized email per 15 min,
  and 50 attempts per client IP per 15 min. Fail closed.
- /api/*: 100 requests per API key per minute (token bucket). Fail open.

Requirements:
- Counters in Redis; increment and expiry must be atomic.
- Over the limit: 429, Retry-After in seconds, JSON error body.
- Check limits before any database work.
- Redis client timeout of 100 ms; log limiter errors.
- Explain which algorithm you used and its known weaknesses.
- Include tests that use a fake clock, not sleep().

Test it before production does

Generated limiters often pass a glance and fail a test. Check these at minimum:

  1. The limit triggers at the right count. Send limit + 1 requests and confirm the last one gets 429:
    for i in $(seq 1 7); do
      curl -s -o /dev/null -w "%{http_code} " \
        -X POST https://staging.example.com/login \
        -d 'email=test@example.com&password=wrong'
    done; echo
    With a limit of 5 failures you should see five normal failure responses followed by 429s.
  2. Retry-After is present and sane. Run curl -i on a limited request and check the header value is a positive number of seconds that matches your window.
  3. The window actually resets. Use a fake clock or a short test window in unit tests rather than sleeping for 15 minutes.
  4. Spoofed headers do not help. Repeat the loop with a different X-Forwarded-For value on each request. You should still get 429s.
  5. It works across instances. In staging with more than one instance or worker, confirm the total allowed is the configured limit, not a multiple of it.
  6. Storage failure behaves as decided. Stop Redis in staging and confirm your fail-open endpoints still serve and your fail-closed endpoints reject.
  7. Keys expire. After a test, inspect Redis and confirm the limiter keys have a TTL (TTL rl:login:email:test@example.com should not return -1).

Review checklist

  • Each limit has a written reason, key, number, and fail-open or fail-closed choice.
  • Client IP comes from trusted proxy configuration, not a raw header.
  • Authenticated traffic is keyed on user or API key; login uses per-account and per-IP limits.
  • Keys are normalized.
  • Counters are in shared storage with atomic increment and expiry.
  • The algorithm is named and its burst behavior is acceptable.
  • Over-limit responses are 429 with Retry-After.
  • Limit checks run before expensive work.
  • Tests cover the threshold, reset, spoofed headers, multiple instances, and storage outage.

An LLM can write the plumbing for any framework or store you name. What it cannot know is who you are limiting, how much is too much, and what sits in front of your app. Supply those, reject the in-memory dictionary and the trusted-by-default header, and test from the outside, and the generated limiter becomes something you can rely on.

Comments

Popular posts from this blog

Grok Bot - a step closer to AGI

Tencent Hy4 Preview: Open 770B MoE Built for Real Work

Structured Outputs and Constrained Decoding for Reliable LLM APIs in Production (September 2026)