Writing Code Documentation With LLMs in October 2026: Generate Docstrings From Real Signatures, Run Every Example, and Catch Doc Drift in CI

Most codebases have the same documentation problem: the docs that exist are out of date, and the functions that most need explaining have nothing at all. Language models are good at turning code into readable prose, so it is tempting to point one at a repository and ask it to "document everything." That works for about a week. Then the code changes, the generated comments stay put, and you have a new problem: confident documentation that is wrong.

This guide covers a workflow that holds up over time. The model drafts, but the facts come from the code itself, every example is executed, and a CI check flags when code and docs stop agreeing.

Photo: Clickety Clackety by Thad Zajdowicz, CC BY 2.0, via Wikimedia Commons

Decide what kind of documentation you actually need

"Documentation" covers several different jobs, and a model handles them with very different reliability:

• Reference docs (docstrings, parameter and return descriptions, raised errors). These are tightly bound to the code and are the best fit for generation, because the source of truth is right there.
• Usage examples. Very useful, but a model will happily invent a call that does not run. These need execution, not review by eye.
• Explanations of why (design decisions, tradeoffs, history). The code rarely contains this. A model can only guess, so this part should come from a person, with the model at most tidying the wording.
• Guides and tutorials. Multi-step, opinionated, and audience-specific. A model can draft an outline from your public API, but a human owns the result.

Start with reference docs for public functions and classes. That is where generation saves the most time with the least risk.

Feed the model facts, not just source text

The most common failure in generated docstrings is describing what a function looks like it does rather than what it does. Reduce that by giving the model structured facts you extract with real tooling, alongside the source:

• The exact signature, including type hints and default values, pulled from the parser or language server rather than copied by hand.
• Every exception the function raises directly, found by walking the syntax tree for raise statements.
• The function's callers and callees within the repository, so the model sees how it is used.
• Existing tests that exercise the function. Tests are often the most accurate description of intended behavior you have.
• Your docstring style (Google, NumPy, reStructuredText, JSDoc, and so on) with one good example from your own codebase.

In Python, the standard library's ast and inspect modules are enough to collect most of this. In TypeScript, the compiler API gives you resolved types. The point is that the signature and error list should never be left to the model to infer.

import ast

def raised_exceptions(func_node: ast.FunctionDef) -> list[str]:
    names = []
    for node in ast.walk(func_node):
        if isinstance(node, ast.Raise) and node.exc is not None:
            exc = node.exc
            if isinstance(exc, ast.Call):
                exc = exc.func
            if isinstance(exc, ast.Name):
                names.append(exc.id)
    return sorted(set(names))

Write a prompt that forbids guessing

Tell the model plainly what it may and may not claim. A prompt along these lines works well:

Write a Google-style docstring for the function below.
Rules:
- Describe only behavior visible in the code, the tests, or the facts provided.
- Use the parameter names and types exactly as given in SIGNATURE.
- List only the exceptions in RAISES. Do not add others.
- If the purpose of a parameter is unclear, write "TODO: clarify" instead of guessing.
- Do not describe performance, thread safety, or side effects unless the code shows them.
- Keep the summary line under 80 characters.
Return only the docstring text.

The "TODO: clarify" escape hatch matters. Without it, the model fills gaps with plausible inventions. With it, you get an honest list of places where a human needs to add knowledge, which is useful output on its own.

Check generated docs against the code mechanically

Before a human even looks at the output, run cheap automated checks. Each one catches a specific class of mistake:

1. Parameter match. Parse the docstring and confirm every documented parameter exists in the signature and every signature parameter is documented. Tools such as pydoclint or the docstring rules in ruff can do this for Python; eslint-plugin-jsdoc covers JavaScript and TypeScript.
2. Type consistency. If your docstrings repeat types, compare them to the annotations. Better still, configure your style to omit types from docstrings when annotations exist, so there is nothing to drift.
3. Exception match. Compare the documented exceptions with the list you extracted from the syntax tree.
4. Default values. If the docstring mentions a default, confirm it matches the signature.
5. Leftover TODOs. Count them and route those functions to a person instead of merging them silently.

Reject and regenerate anything that fails these checks. They are deterministic, fast, and they remove most of the errors that would otherwise slip past a tired reviewer.

Run every example

An example that does not run is worse than no example, because readers copy it. Treat examples in documentation as tests:

• In Python, write examples in doctest format and run them with python -m doctest or pytest --doctest-modules.
• In Rust, code blocks in doc comments are compiled and run by cargo test by default, so generated examples are checked for free.
• In Go, write Example functions in test files; go test runs them and compares their output to the // Output: comment.
• For Markdown guides, extract fenced code blocks and run them in a scratch environment as part of CI.

If a generated example fails, feed the error back to the model once with the failing output, then rerun. If it still fails, drop the example and flag the function for a human. Do not loosen the check to make it pass.

def parse_duration(text: str) -> int:
    """Convert a duration such as "1h30m" to seconds.

    Args:
        text: Duration made of hour, minute, and second parts, e.g. "2m15s".

    Returns:
        Total number of seconds.

    Raises:
        ValueError: If the text contains an unknown unit.

    Example:
        >>> parse_duration("1h30m")
        5400
        >>> parse_duration("45s")
        45
    """

Photo: Wooden Card Catalog Furniture by MarkBuckawicki, CC0, via Wikimedia Commons

Catch doc drift in CI

Generating docs once is the easy part. Keeping them true as the code changes is where most efforts fall apart. Add two checks to your pipeline:

1. Static drift checks on every pull request

Run the same parameter, exception, and default-value checks from earlier on changed files. If someone adds a parameter and forgets the docstring, the build tells them. This catches the majority of drift without any model involved.

2. A model-assisted "does this still match" check

For functions whose body changed but whose docstring did not, send the old docstring and the new code to a model with a narrow question: "List any statements in this docstring that are no longer true for this code. Answer NONE if all statements still hold." Post the answer as a review comment rather than failing the build. This is a hint for the author, not a gate, because a model can be wrong in both directions.

Keep the scope tight. Only check functions that changed in the pull request, and only public ones. Checking the whole repository on every push is slow, costly, and produces comment noise people learn to ignore.

Roll it out without a giant pull request

A single pull request that adds docstrings to 2,000 functions will not be reviewed carefully. Break the work up:

1. Start with the public API surface: exported modules, endpoints, and SDK entry points.
2. Generate docs for one module or package per pull request, sized so a reviewer can actually read it.
3. Assign the reviewer who owns that code, not whoever is free. Owners spot wrong claims quickly.
4. Merge only functions that pass the mechanical checks and have no remaining TODOs; leave the rest for the owner to complete.
5. Turn on the CI drift checks for that module as soon as it is documented, so it does not regress while you move on to the next one.

What to keep humans responsible for

Some documentation should not be generated, even as a draft that gets approved with a quick glance:

• Security-relevant behavior: authentication requirements, input validation guarantees, what is and is not escaped. A wrong statement here leads people to build unsafe code.
• Stability promises: deprecation notices, versioning guarantees, and anything a customer might rely on contractually.
• Design rationale: why the code works this way. The model does not know, and a plausible invented reason is harder to correct than a blank.
• Migration guides between versions, which need someone who understands what actually broke.

Common mistakes to avoid

• Restating the code. A docstring that says "Increments counter by one" above counter += 1 adds nothing. Ask the model to explain intent and constraints, and delete comments on trivial private helpers.
• Inline comments everywhere. Generated line-by-line comments go stale fastest. Prefer docstrings on public units and leave inline comments for genuinely surprising code.
• Sending secrets to the model. Source files sometimes contain keys or internal hostnames. Run a secret scanner over what you send, and use a provider and plan whose data handling terms you have actually checked.
• Treating approval as verification. A reviewer skimming 300 plausible docstrings will approve them. The mechanical checks and executed examples are what actually verify correctness.

A short checklist

1. Extract signatures, raised exceptions, and related tests with real tooling.
2. Prompt the model to describe only what the code shows, with a TODO escape hatch.
3. Reject output that fails parameter, exception, or default-value checks.
4. Execute every example as a test.
5. Review in small, module-sized pull requests with code owners.
6. Run drift checks on every pull request, with model hints as comments, not gates.
7. Leave security behavior, stability promises, and design rationale to people.

Done this way, a model becomes a fast first-draft writer that is kept honest by your toolchain. The docs get written, and more importantly, they stay true.