Debugging Flaky Tests With LLMs in November 2026: Capture the Failure Twice, Isolate Shared State, and Never Accept a Sleep as a Fix

A flaky test is one that passes and fails on the same commit. It is green on your laptop, red in CI, then green when you press "Re-run failed jobs." The usual prompt, "this test is flaky, fix it," tends to produce the same three answers: add time.sleep(2), wrap the assertion in a retry, or mark the test xfail. Those make the red bar go away. They do not tell you whether the test is poorly isolated or the product has a race. This guide shows a workflow where you collect evidence of the intermittency first, let the model classify the flake against that evidence, and only keep a change that makes the failure go away for a reason you can explain.

A flaky test is a roll you cannot trust. Capture the failure more than once before you ask the model what went wrong. Image: A throw of two dice as in the game of hazard by JonRichfield, CC BY-SA 4.0, via Wikimedia Commons.

Why "just fix the flake" goes wrong

A model cannot see a failure it was not shown. The source of a test looks fine when the test passes, and most of the useful information lives in how it fails: which assertion broke, on which worker, after which other test, under what clock and random seed. Without that, the model guesses from patterns it has seen before, and the most common pattern for "intermittent" is "wait longer."

The usual failure modes:

Sleeping through the race. A fixed pause hides the timing problem on a quiet machine and still fails on a loaded CI runner. It also makes the suite slower for everyone.
Weakening the assertion. Accepting extra status codes, dropping an order check, or widening a numeric tolerance until the test almost cannot fail.
Skipping or xfailing. The red bar disappears and the underlying issue does not.
Editing production code without a reproduction. The model "fixes" a function that was not involved, and the flake remains.
Treating every flake as a timing problem. Many intermittent failures are leftover database rows, a hardcoded port, an unseeded shuffle, or datetime.now() crossing midnight. A sleep does nothing for those.

The way through is simple to state: capture the failure more than once, run a few isolation checks yourself, and make the model quote the evidence behind every claim.

Step 1: Capture the failure more than once

One red log is not enough. A test can fail for different reasons on different retries, for example a timeout in one run and a unique-constraint error in the next. Collect at least two failing runs before you ask for a diagnosis, and save the full output rather than a screenshot of the last line.

A tight loop around a single test is a good start:

# If pytest-repeat is installed
pytest tests/test_checkout.py::test_creates_invoice --count=50 -q

# Without an extra plugin
for i in $(seq 1 50); do
  echo "run $i"
  pytest tests/test_checkout.py::test_creates_invoice -q || break
done

Also try the surrounding file, not just the single test, and try parallel workers if your project uses them:

pytest tests/test_checkout.py -q
pytest -n auto tests/test_checkout.py -q

For each failure, keep:

The full traceback and the assertion diff, not a trimmed summary.
The exact command and the pytest header, including plugins and the xdist worker name if one appears (gw0, gw1, and so on).
Whether the run was local or in CI, the Python version, and the timezone if you know it.
The name of the test that ran immediately before the failure, when the log includes it.

If the test uses Hypothesis, save the seed or falsifying example it prints. That seed is the closest thing you will get to a deterministic reproduction of a randomized failure.

Step 2: Run a few isolation checks yourself

These experiments often classify the flake before the model says a word, and they give it something concrete to interpret. Write the answers down in a short note.

Does it fail when run alone? Repeat pytest path::test_name. If it only fails as part of the file or the full suite, suspect leftover state or order dependence.
Does it fail only under parallel workers? If pytest -n auto flakes and a single process does not, suspect a shared resource: a fixed port, a shared database, a hardcoded temp path, or a process-wide singleton.
Does the clock matter? Failures near midnight UTC, on the first or last day of a month, or only in a timezone that is not UTC point at datetime.now(), date formatting, or "today" assertions.
Is anything random? Unseeded random or numpy generators, a shuffled fixture, or an expected value built from a fresh uuid that the test then tries to guess.
Is anything on the network? A real HTTP call, a container that is not ready, or a shared staging service that other jobs also hit.

A note like "failed 8 times in 50 under pytest -n auto, never failed in 50 runs alone, error is IntegrityError on email" is worth more to the model than the test source by itself.

Step 3: Give the model the evidence, not just the test

Now the model has something real to work with. A good prompt includes:

The test function and the fixtures it uses, including autouse fixtures from conftest.py.
Two complete failure traces, pasted as text.
The isolation notes from the previous step.
How the suite runs: pytest plugins, whether xdist is on in CI, and whether tests share a database.
Constraints: do not add fixed sleeps, do not weaken assertions, and do not skip or xfail the test.

Then ask a narrow question, for example:

Here is test_creates_invoice, its fixtures, and two failing pytest logs.
Isolation notes: fails under pytest -n auto (8/50), never failed alone (0/50).
Both failures are IntegrityError on users.email; the email in the test is "buyer@example.com".
CI uses pytest-xdist. Tests share one Postgres database.
1. Classify the flake (leaked state, race, time, randomness, environment, external system).
   Quote the log lines and source lines you rely on.
2. Propose at most three hypotheses, ranked.
3. For the top hypothesis, describe a fix that removes the cause.
   Do not suggest time.sleep, retries, skip, or xfail.
4. Tell me exactly what to rerun, and how many times, to gain confidence the flake is gone.

Asking the model to quote the lines it relies on keeps the diagnosis tied to evidence. If it claims a race but cannot point to shared state, a missing wait, or a parallel-only pattern in what you pasted, treat that claim as a guess.

Leftover state is the quiet cause of many flakes. If a previous test left dishes in the sink, the next one should not have to wash them. Image: Sink full of dirty dishes by Historyofspiders, CC0, via Wikimedia Commons.

Step 4: Know the fixes that actually remove flakes

Most real repairs fall into a short list of patterns. When the model proposes something, it helps to recognize which category it belongs to.

Leftover state between tests. One test inserts a user with a fixed email such as buyer@example.com, and another test assumes that email is free. Use a unique value per test, for example f"buyer-{uuid4()}@example.com". Prefer transactional fixtures that roll back, or a finalizer that deletes what it created even when the test fails. Watch for module-level lists, mutable class attributes, and lru_cache on helpers that tests share.
Shared resources under parallel workers. Two xdist workers bind localhost:8001 or write ./tmp/test.db. Bind port 0 and read the assigned port; use pytest's tmp_path, which is unique per test; give each worker its own database. The xdist plugin exposes a worker_id fixture you can append to database names and cache directories.
Races in async or UI tests. The test clicks Submit and immediately asserts on a row the server has not finished writing. A fixed sleep fails on a slow runner and wastes time on a fast one. Wait for a condition with a timeout instead. Playwright's expect(locator).to_have_text(...) polls until the text appears or the timeout fires. In plain Python, a small helper does the same job:

def wait_until(predicate, timeout=5.0, interval=0.05):
    deadline = time.monotonic() + timeout
    last = None
    while time.monotonic() < deadline:
        last = predicate()
        if last:
            return last
        time.sleep(interval)
    raise AssertionError(f"condition not met within {timeout}s; last={last!r}")

The pause inside the loop is not the same as time.sleep(2) before an assertion. The timeout is a budget, the predicate is the real check, and the test still fails loudly if the condition never happens.

Time dependence. Code that calls datetime.now() plus an assertion about "today" breaks around midnight, during daylight saving changes, or when CI runs in UTC and a developer does not. Inject a clock (a now callable the production code uses) or freeze time in the test with a library such as time-machine or freezegun. Do not change the assertion so that yesterday and tomorrow both count as a pass.
Uncontrolled randomness. Seed the generator in the test, or pass the shuffled input in explicitly so the test does not depend on global random state that another test may have altered. If Hypothesis found the failure, keep that example as a regression case.
External systems. Prefer fakes or recorded responses for unit-level tests. If you must talk to a real service or a container, wait on a health check rather than a fixed startup sleep, and fail with that system's error instead of an unrelated assertion later.

Be skeptical of larger moves, such as rewriting the test as an end-to-end scenario or disabling xdist for the whole file, until the simple isolation fixes are tried. Those changes can hide the same leak and make the suite slower.

Step 5: Watch for "fixes" that only hide the problem

Send the model back to the evidence if it offers any of the following as the solution:

A fixed time.sleep before an assertion.
A retry decorator on the test itself as the only change. Markers such as @pytest.mark.flaky(reruns=3) are a temporary CI bandage while you investigate, not a diagnosis.
Catching a broad exception and continuing.
Loosening equality into a containment check, or a numeric tolerance you cannot justify from the requirements.
Deleting the test, skipping it, or marking it xfail.

A useful follow-up is: "What is the underlying race or leaked state? Propose a fix that removes that cause." If the model still cannot name one, you are missing evidence. Run another isolation check rather than accepting a weaker test.

Step 6: Sometimes the test is right

A flake can be a real race in the product: two requests minting the same invoice number, a websocket handler reading a field before it is written, a cache filled by a background thread. In that case the correct change is in the application, and the test should keep failing until the race is gone.

Ask the model to consider this on purpose: "Could this failure be correct? What would a product-level race look like here?" If the answer is yes, try to turn the race into a deterministic reproduction, for example by running two requests concurrently or injecting a delay at the suspicious point, and then fix the code. Do not silence a real bug because CI is noisy.

Step 7: Prove the flake is gone

Apply one change, then rerun the command that used to fail, many times. If it failed 8 times in 50 under pytest -n auto, run at least that many again. Passing once proves nothing. Also run the surrounding file and the full suite once, so unique IDs or cleanup did not collide with another test.

If you added a wait-for-condition, confirm it still fails quickly when the condition is genuinely broken. Temporarily assert on the wrong text, or break the feature on purpose. A wait that always succeeds is just a quieter sleep.

Keep a short log in the pull request:

flake: test_creates_invoice IntegrityError on users.email
isolation: 8/50 fail with -n auto, 0/50 alone
change: unique email per test, worker-scoped database
rerun: 0/50 fail with -n auto; full suite passed

That note saves the next person from rediscovering the same flake, and it stops you from keeping a change you cannot defend.

A quick checklist

Capture at least two full failing logs before you ask the model anything.
Run the test alone, in its file, and under parallel workers; write down what changes.
Paste the test, its fixtures, both traces, and the isolation notes. Ban sleeps, retries-as-fixes, skips, and weaker assertions.
Make the model classify the flake, quote its evidence, and name the shared state or race.
Prefer unique data, transactional cleanup, per-worker resources, injected clocks, and wait-for-condition helpers.
Consider that the product, not the test, may be racy, and keep the failing check if so.
Rerun the previously failing command many times, run the full suite once, and record the counts in the pull request.

Used this way, an LLM is a good partner for flaky tests. It is fast at matching a traceback and a fixture to a known pattern, and it can suggest several plausible causes at once. The isolation runs tell it which pattern is real, the quoted evidence keeps it from inventing a race that is not there, and the rerun counts tell you whether the change actually earned its place.

Search This Blog

Technology On the Net