LLM Code Review Bots in October 2026: Feed the Right Diff Context, Cap Comment Noise, and Measure Which Suggestions Developers Actually Accept
An LLM that reviews pull requests sounds like an easy win: point it at a diff, let it leave comments, and catch bugs before a human reviewer has to. In practice, most teams that try this hit the same wall within a couple of weeks. The bot leaves too many comments, half of them are style nitpicks or restate what the code obviously does, and developers learn to scroll past it. Once that happens, the occasional real bug it finds gets ignored too.
This guide covers how to build or configure an LLM review bot that people keep reading: what context to send, how to keep the comment count low, how to post comments that land on the right line, and how to measure whether the bot is helping. The patterns apply whether you use a hosted review product or wire a model into your CI yourself.
Decide What the Bot Is For
Before writing any prompt, pick a narrow job. "Review this code" invites the model to comment on everything. A focused scope produces fewer, better comments. Good scopes for an automated reviewer include:
- Correctness bugs: off-by-one errors, null or undefined access, wrong comparison operators, inverted conditions, unhandled error returns.
- Security issues: SQL built from string concatenation, unescaped output in templates, secrets committed in code, missing authorization checks on new endpoints.
- Resource and concurrency problems: files or connections that are opened but never closed, shared state mutated without a lock, missing timeouts on network calls.
- Mismatch with the PR description: the description says "only logs a warning" but the code throws.
Leave formatting, naming, and import order to your linter and formatter. They are deterministic, free, and never argue. If your bot comments on anything a linter could catch, that is a configuration problem, not a model problem.
Send the Right Context, Not Just the Diff
A raw unified diff shows three lines of context around each change. That is often not enough for the model to know what a variable holds or what a function it calls returns. On the other hand, dumping the whole repository into the prompt is slow, expensive, and dilutes attention. Aim for a middle layer:
- The diff itself, with line numbers from the new file version so comments can be anchored later.
- The full current version of each changed function or class, not just the changed hunk. Use your language's parser (tree-sitter works for most languages) to find the enclosing definition for each hunk.
- Signatures of functions the changed code calls, including docstrings, when they live in other files. This is where many real bugs hide: the caller assumes a function returns a list, and it actually returns a list or
None. - The PR title and description, so the model can flag code that does not match stated intent.
- A short repository conventions file if you have one, for example "all database access goes through the repository layer" or "money is stored as integer cents."
Skip generated files, lockfiles, vendored dependencies, snapshots, and minified assets entirely. Filter them by path pattern before anything reaches the model. They burn tokens and invite meaningless comments.
Large pull requests
When a PR touches dozens of files, review it file by file or in groups of related files, and keep each request under a fixed token budget. Give every request the PR description and a list of all changed file paths so the model knows the bigger picture. Then merge and deduplicate the findings before posting. If a PR is too large to review within your budget, say so in a single summary comment instead of silently reviewing only part of it.
Ask for Structured Findings
Do not let the model write free-form review prose. Ask for a list of findings that match a schema, using your provider's structured output or JSON schema mode:
{
"findings": [
{
"file": "src/billing/invoice.py",
"line": 42,
"category": "correctness",
"severity": "high",
"confidence": "high",
"title": "Discount applied twice when coupon and promo both present",
"explanation": "apply_discount() is called inside the loop and again after it...",
"suggested_fix": "Move the second call inside the else branch."
}
]
}Structured output gives you three things free-form text cannot. You can filter by severity and confidence in code. You can validate that file and line point at lines that actually changed. And you can count and track findings over time.
Tell the model explicitly that an empty findings list is a valid and good answer. Without that instruction, models tend to find something to say on every PR, because the prompt implies there must be issues.
Cap Comment Noise in Code, Not in the Prompt
Prompt instructions like "only comment on important issues" help a little. Hard filters in your own code help a lot more. Apply them after the model returns findings and before anything is posted:
- Drop low-confidence findings, or move them into a collapsed summary instead of inline comments.
- Drop findings on unchanged lines. Check each
lineagainst the set of added or modified lines in the diff. Comments on code the author did not touch feel unfair and are usually out of scope. - Cap inline comments per PR, for example five. Sort by severity, post the top ones, and list the rest in one summary comment.
- Deduplicate across pushes. When the author pushes a new commit, the bot reviews again. Store a fingerprint for each posted finding (file, a normalized snippet of the flagged code, and category) and skip findings you already posted. Repeating the same comment on every push is the fastest way to get a bot muted.
- Respect dismissals. If a developer resolves or replies "won't fix" to a comment, do not raise the same finding again on that PR.
Anchor Comments to the Right Line
Line numbers are a common failure point. Models are not reliable at counting lines in a diff, so do not ask them to. Instead, prefix each line in the context you send with its new-file line number, like 42 |, and ask the model to return that number. Then validate it: the line must exist in the new version and fall inside a changed hunk. If validation fails, try matching the quoted code snippet from the finding against the file to recover the right line. If that fails too, move the finding into the summary comment rather than posting it on a wrong line.
Most code hosts' review APIs let you post comments tied to a specific commit, path, and line, and some support suggested-change blocks that the author can apply with one click. Use suggestion blocks only for small, self-contained fixes where the replacement is certain. A suggestion that does not compile damages trust more than no suggestion.
Keep the Bot Safe
A review bot reads untrusted input: anyone who can open a pull request controls the diff, the description, and code comments. Treat that text as data, never as instructions.
- Give the bot a token that can read code and post review comments, and nothing else. It should not be able to approve, merge, push, or change settings.
- Do not let the bot's review count toward required approvals. It advises; humans decide.
- For PRs from forks, avoid running the bot with secrets in contexts the fork author can influence. Follow your CI provider's guidance for workflows triggered by external contributors.
- Render the bot's comments as plain Markdown without raw HTML, and strip any links the model invents that do not point at your own repository or documentation.
- If your code is sensitive, confirm where the model provider stores and retains prompts before sending source code to it.
Measure Whether It Helps
"It feels useful" is not a metric. Track a few numbers from day one, using the fingerprints you already store:
- Acceptance rate: the share of inline comments where the flagged lines changed in a later commit on the same PR. This is an imperfect proxy, since code can change for other reasons, but it trends in the right direction and is cheap to compute.
- Dismissal rate: comments resolved without a code change, or replied to with disagreement.
- Comments per PR: watch the distribution, not just the average. A long tail of PRs with many comments usually means a filter is missing.
- Escaped bugs: when a bug reaches production, check whether the bot reviewed that change and what it said. This is slow to gather but the most honest signal.
Review a sample of dismissed comments every week or two. Group them by category. If one category is mostly dismissed, tighten the prompt for it, raise its confidence threshold, or drop it from scope. If developers keep accepting a certain kind of finding, consider whether a linter rule or a static analysis check could catch it deterministically instead.
Roll It Out Gradually
- Shadow mode first. Run the bot on real PRs for a week or two, but write findings to a log or a private channel instead of the PR. Have a couple of engineers grade a sample as useful, wrong, or noise.
- Summary-only mode. Post one collapsed summary comment per PR. This is low-friction and easy to ignore if it is wrong.
- Inline comments for high-severity, high-confidence findings only. Expand scope only when the acceptance rate holds up.
- Offer an opt-out label such as
skip-ai-reviewfor PRs like large migrations or generated code updates where review would be pure noise.
A Short Checklist
- Narrow scope: correctness, security, resources, intent mismatch. Leave style to the linter.
- Send the diff plus enclosing functions, called signatures, and the PR description. Skip generated files.
- Require structured findings with severity and confidence, and accept an empty list.
- Filter in code: confidence threshold, changed lines only, per-PR cap, dedupe across pushes, respect dismissals.
- Number the lines yourself and validate every anchor before posting.
- Read-only, comment-only token. The bot never approves or merges.
- Track acceptance and dismissal rates and prune categories that developers keep rejecting.
A review bot earns its place by being quiet most of the time and right when it speaks. Build the filters, measure what gets accepted, and keep cutting whatever developers keep ignoring.
Comments
Post a Comment