Generating Unit Tests With LLMs in October 2026: Run Every Test, Prove It Can Fail, and Keep Only the Ones That Earn Their Place

Asking a language model to "write tests for this file" is easy, and the output usually looks convincing: a dozen neatly named test functions, a few mocks, everything green on the first run. The trouble is that green on the first run proves very little. A generated test can pass because it checks nothing meaningful, because it mocks away the code under test, or because it faithfully locks in a bug that already exists. This guide walks through a practical pipeline for generating unit tests with an LLM and then filtering them hard, so the tests you merge actually protect your code.

Why Raw Generated Tests Are a Liability

Before building the pipeline, it helps to name the specific ways generated tests go wrong. Almost every bad test falls into one of these buckets:

• Tautological tests. The test calls a function and asserts that the result equals whatever the function returned, or asserts only that the result is not None. It passes forever and catches nothing.
• Over-mocked tests. The model mocks the very dependency that contains the logic, so the test exercises the mock instead of your code.
• Bug-preserving tests. The model reads the implementation, works out what it currently returns, and asserts that. If the implementation is wrong, the test now defends the wrong behavior.
• Tests that do not run. Wrong import paths, invented helper functions, fixtures that do not exist, or API calls from a different library version.
• Flaky tests. Dependence on wall-clock time, dictionary or set ordering, random values without a seed, network access, or shared state between tests.
• Duplicates. Five tests that cover the same happy path with slightly different input values and add no new coverage.

The fix is not a cleverer prompt alone. The fix is to treat every generated test as a candidate and make it pass a series of automatic checks before a human ever reviews it.

Photo: Test tubes in rack by Jeskra1, CC BY-SA 4.0, via Wikimedia Commons

Step 1: Give the Model the Right Context

The quality of generated tests depends heavily on what you send. A bare function body pushes the model toward describing the implementation back to you. Send context that describes intended behavior instead:

• The function signature, docstring, and type hints.
• Any written specification, ticket text, or README section describing what the function should do.
• One or two existing tests from the same project, so the model copies your framework, fixture names, and assertion style.
• The import path the test file should use, stated explicitly.
• A short list of what the model may and may not mock. For example: "You may mock the HTTP client. Do not mock pricing.apply_discount; it is the code under test."

If you have a choice, send the spec before the implementation and tell the model to derive expected values from the spec. This one change reduces bug-preserving tests, because the model is reasoning about what should happen rather than what does happen.

Step 2: Ask for Behaviors First, Then Code

Split generation into two calls. In the first, ask for a plain list of behaviors worth testing, each with an input and an expected outcome and a one-line reason. In the second, ask for test code that implements only the behaviors you kept.

Behaviors for parse_duration(text) -> int (seconds):
1. "90s" returns 90 (plain seconds suffix)
2. "2m30s" returns 150 (combined units)
3. "1h" returns 3600 (hours)
4. "" raises ValueError (empty input)
5. "-5s" raises ValueError (negative durations are not allowed per spec)
6. "10x" raises ValueError (unknown unit)
7. " 45s " returns 45 (spec says surrounding whitespace is ignored)

The behavior list is cheap to review, easy to deduplicate, and makes gaps obvious. If nothing in the list covers an error path or a boundary value, you can ask for more before any code is written. You can also store the list next to the test file as documentation of intent.

Step 3: Run Every Candidate in Isolation

Never merge a test you have not executed. Write each candidate test into a temporary file and run it in a sandboxed environment that matches CI: same interpreter version, same dependencies, no network unless the test suite normally has it. In Python that can be as simple as:

pytest tests/generated/test_parse_duration.py -q -p no:cacheprovider --timeout=30

(The --timeout flag comes from the pytest-timeout plugin.) Sort the results into three groups:

1. Errors at import or collection time. Feed the error message back to the model once and ask for a fix. If it still fails, drop the test.
2. Tests that fail. Do not automatically "fix" these by changing the assertion. A failing test that was derived from the spec may have found a real bug. Route these to a human with the behavior description attached.
3. Tests that pass. These move on to the next filters. Passing is the entry ticket, not the finish line.

The second group is the most valuable part of the whole exercise and the one most pipelines throw away. If the spec says negative durations raise ValueError and the code returns -5, you have learned something. Rewriting the assertion to == -5 would erase that.

Step 4: Check for Flakiness

Run each passing test several times, in random order, before trusting it. With pytest, the pytest-randomly plugin shuffles test order automatically once it is installed, and the pytest-repeat plugin adds a --count option to repeat runs:

pytest tests/generated/ --count=5 -q

Any test that fails even once is dropped or sent back with the failure output. Common causes to look for in the code are calls to the current time without freezing it, unseeded random values, reliance on set or dictionary iteration order in assertions, and temporary files written to fixed paths that collide when tests run in parallel.

Step 5: Prove Each Test Can Fail

A test that cannot fail is worthless, and "it passes" says nothing about whether it can fail. Two checks help here.

Coverage delta

Measure line and branch coverage of the target module with and without the new test, using a tool such as coverage.py (or pytest-cov). A new test that adds no covered lines or branches is a candidate for removal unless it checks an important edge case on already-covered lines. This filter alone removes most duplicate happy-path tests.

pytest --cov=myapp.durations --cov-branch --cov-report=json tests/

Mutation testing

Coverage only proves a line ran. Mutation testing proves an assertion noticed when the line changed. A mutation tool makes small changes to your code, such as flipping < to <=, replacing a constant, or deleting a statement, and reruns the tests. If no test fails, that mutant "survived," which means nothing is actually checking that behavior. Established tools include mutmut for Python, Stryker for JavaScript and TypeScript, and PIT for Java.

Mutation testing is slow on a large codebase, so run it only on the module the new tests target. Then keep a generated test only if it kills at least one mutant that the existing suite did not kill, or if it covers new branches. This is the single strongest filter against tautological and over-mocked tests, because those tests keep passing no matter what you change.

This filter-first approach is not new. Meta described a similar pipeline in its 2024 TestGen-LLM paper, where generated tests were kept only if they built, passed reliably, and improved coverage. The mutation check above simply raises the bar from "ran the code" to "noticed when the code broke."

Photo: Crash test dummies by the National Institute of Standards and Technology, public domain, via Wikimedia Commons

Step 6: Lint the Tests Themselves

Add a few cheap static checks on the generated test code. These catch problems that slip past execution:

• Every test function contains at least one assertion (or an explicit pytest.raises block).
• No assertion compares a value to itself or to a variable computed by calling the same function under test.
• No mock or patch targets the module under test, using the allowlist from Step 1.
• No network calls, no sleep calls, and no hard-coded absolute paths.
• Test names describe the behavior, for example test_negative_duration_raises rather than test_parse_duration_3.

Most of these can be written as a small script that walks the abstract syntax tree of the test file. Rejecting on these rules before running mutation testing also saves time.

Step 7: Consider Property-Based Tests for Pure Functions

For pure functions such as parsers, formatters, and math helpers, ask the model for property-based tests instead of, or in addition to, example-based ones. Libraries like Hypothesis for Python and fast-check for JavaScript generate many inputs and check that a stated property always holds. Properties are often easier to state correctly from a spec than individual expected values:

from hypothesis import given, strategies as st
from myapp.durations import parse_duration, format_duration

@given(st.integers(min_value=0, max_value=10**6))
def test_format_then_parse_round_trips(seconds):
    assert parse_duration(format_duration(seconds)) == seconds

A round-trip property like this exercises many inputs at once and is hard to write in a tautological way. Run it through the same flakiness and mutation filters as any other test.

Step 8: Human Review, With the Evidence Attached

Once a test has survived execution, repetition, coverage, mutation, and lint checks, put it in a pull request with its evidence: the behavior it checks, the coverage it adds, and the mutants it kills. Reviewers can then focus on one question: is this the behavior we actually want? That is the question only a person who knows the product can answer, and it goes much faster when the mechanical checks are already done.

Keep the failing-test group from Step 3 as a separate list, not a separate pull request full of red tests. Each item is a possible bug report with a ready-made reproduction.

A Practical Checklist

1. Send the spec, docstring, an example test, the import path, and a mock allowlist.
2. Generate a behavior list first and review it before generating code.
3. Execute every candidate in a sandbox matching CI; allow one repair attempt for import errors.
4. Treat spec-derived failures as possible bugs, never auto-rewrite the assertion.
5. Repeat passing tests in random order to catch flakiness.
6. Keep a test only if it adds coverage or kills a mutant the existing suite missed.
7. Lint for missing assertions, self-comparisons, and forbidden mocks.
8. Prefer property-based tests for pure functions.
9. Merge through normal review with the evidence attached.

The model is good at producing many plausible tests quickly. Your pipeline's job is to be skeptical of all of them. Most candidates will be thrown away, and that is fine: a smaller suite of tests that each detect a real failure is worth far more than a large suite that only turns green.