Generating Release Notes From Merged Pull Requests With LLMs in October 2026: Group Changes by Audience, Link Every Line to a PR, and Keep a Human Editor

Release notes are one of those jobs everybody agrees matter and nobody wants to write. The raw material already exists: every merged pull request has a title, a description, labels, and a diff. The hard part is turning forty terse PR titles like "fix race in cache warmup" and "bump deps" into something a customer, a support agent, or an operator can actually use.

Language models are a good fit for that translation step. They read messy engineering text quickly and rewrite it in plain language. They are also happy to invent a feature that was only half merged, describe a fix as bigger than it was, or quietly drop the one breaking change that matters most. This guide walks through a pipeline that keeps the speed and keeps the notes honest.

The drum of an old Miehle printing press
Photo: Old Threshers Miehle Press Drum by Douglas W. Jones, CC0, via Wikimedia Commons

Start with the exact list of changes, computed in code

The single most important rule: the model never decides what shipped. Your version control system does. Compute the set of merged pull requests between the previous release tag and the new one with git or your forge's API, and treat that list as the source of truth.

A simple approach with git and the GitHub CLI looks like this:

# commits between two tags, merge commits only
git log v2.13.0..v2.14.0 --merges --pretty=format:'%s'

# or ask the forge for PRs merged into main in that window
gh pr list --state merged --base main \
  --search "merged:2026-10-01..2026-10-21" \
  --json number,title,body,labels,author,url --limit 200

Tag-based ranges are more reliable than date ranges, because a PR merged on the last day may or may not be in the build. If you use squash merges, the commit range still works; parse the PR number from the "(#1234)" suffix most forges add to the squashed commit title.

Save that list as structured data, one record per PR, before any model sees it. Every later step checks its output against this list.

Collect the right context for each PR, and no more

The model needs enough to understand what changed for a user, not the whole diff. For each PR, gather:

  • The title and the description, with templates and checklists stripped out.
  • Labels such as feature, bug, breaking, security, internal, or docs.
  • Linked issue titles, which often describe the user-facing problem better than the PR does.
  • A short list of changed file paths, so the model can tell a UI change from a CI tweak.

Skip full diffs for most PRs. They burn tokens and tempt the model into narrating implementation details nobody asked for. Include a trimmed diff only when the description is empty, and cap it at a few hundred lines.

Also strip anything that should not leave the building: internal ticket links, customer names mentioned in bug reports, hostnames, and security details for fixes that are not public yet. Do this in code before the call, not by asking the model to "leave out sensitive things."

Classify first, then write

Splitting the job into two steps makes both easier to check. In the first step, ask the model to classify each PR into a fixed set of categories and audiences, and return structured output:

{
  "pr": 1234,
  "category": "fix",            // feature | improvement | fix | breaking | security | deprecation | internal
  "audience": ["end_user"],     // end_user | admin | api_developer | operator
  "user_visible": true,
  "summary": "Exports no longer time out for workspaces with more than 10,000 rows."
}

Use your provider's structured output or JSON schema mode so you get valid records every time. Then apply some rules in code:

  • If a PR carries a breaking or security label, that wins over whatever the model picked. Labels set by humans are stronger evidence than a guess from a title.
  • If the model marks a PR as internal and not user visible, drop it from customer notes but keep it in an engineering changelog.
  • Every input PR number must appear exactly once in the output. If any are missing or duplicated, retry or flag the run.

That last check catches the most common failure, which is the model silently skipping items in a long list. If you have more than about fifty PRs, classify them in batches of twenty or so rather than all at once.

Write separate notes for separate readers

One set of notes rarely serves everyone. A customer wants to know what is new and what got fixed. An API developer needs every changed endpoint and deprecation. An operator running a self-hosted install needs migration steps, new configuration keys, and changed defaults. Generate a section, or a separate document, for each audience from the same classified records.

For the writing step, give the model the classified records for one audience and a short style guide. A useful prompt includes:

  • The audience and what they care about.
  • Tone rules: present tense, plain language, say what the user can now do, no marketing adjectives.
  • Structure: headings in a fixed order, such as Breaking changes, New, Improved, Fixed.
  • A hard rule that every bullet ends with the PR reference, like "(#1234)", and that the model may only describe changes present in the records.
  • Two or three examples of good bullets from past releases.

Good bullets read like this: "You can now filter the activity log by date range. (#1218)" or "Fixed a crash when uploading files with non-ASCII names on Windows. (#1240)". They describe the effect, not the code.

A wooden village notice board for public announcements
Photo: Burstall Village Notice Board by Geographer, CC BY-SA 2.0, via Wikimedia Commons

Verify the draft against the source list

Once the model writes the notes, check them mechanically before a person ever reads them:

  1. Every bullet has a PR reference. Parse each bullet with a regular expression. A bullet with no reference is a claim with no source; send it back or remove it.
  2. Every reference is real. Each PR number in the notes must be in the computed list for this release. A number that is not in the list means the model invented or misremembered something.
  3. Nothing important is missing. Every PR classified as breaking, security, or deprecation must appear in the notes. Missing a breaking change is far worse than including a minor one.
  4. Names are consistent. Compare product, feature, and setting names against a short glossary. Models like to "improve" names, and a renamed setting in the notes sends users looking for something that does not exist.
  5. Numbers come from the source. If a bullet contains a number, such as a limit, a version, or a timeout, check that the same number appears in that PR's title, description, or linked issue. Flag any number that does not.

These checks are cheap, run in milliseconds, and catch the failures that would embarrass you in public. When one fails, regenerate only the affected section with the specific error in the prompt rather than starting over.

Handle breaking changes and security fixes with extra care

These two categories deserve rules of their own.

For breaking changes, do not let the model write migration steps from a title. Require the PR author to fill in an "Upgrade notes" section in the PR description, and have the pipeline copy that text, lightly edited, into the notes. If the section is missing, fail the run and ask the author. A model guessing at migration steps is how people lose data.

For security fixes, follow your disclosure process. Often the public note should say only that a security issue was fixed, with a severity and an advisory link published at the right time. Feed the model only the approved public wording, never the private report.

Keep a human editor in the loop

The pipeline produces a draft, not a publication. Open it as a pull request against your docs or changelog, or post it in the release channel, and assign one person to edit it. Their job is short and focused:

  • Read the breaking changes and upgrade notes closely.
  • Reorder the "New" section so the most important change is first.
  • Cut anything that is technically true but not useful to the reader.
  • Confirm the tone matches past releases.

Track what the editor changes. If they keep rewriting the same kind of bullet, add an example of the fix to the prompt. If they keep deleting a category of PR, tighten the classification rules. Over a few releases, edits shrink and the review takes minutes.

Improve the input, not just the prompt

The best release notes come from good PR descriptions. A few small habits make the whole pipeline better:

  • Add a one-line "User-facing change" field to your PR template, with "None" as a valid answer.
  • Require a label from a small fixed set before merge, enforced by a CI check.
  • Link the issue the PR resolves, so the user's own wording is available.

With those in place, the model mostly rewrites and groups text that humans already wrote, which is exactly what it does well. Without them, it has to guess, and guesses are where inaccurate notes come from.

A minimal pipeline you can build this week

  1. On release tag, compute merged PRs between the previous tag and the new one.
  2. Fetch title, cleaned description, labels, linked issue titles, and changed paths for each PR.
  3. Redact internal links and private details in code.
  4. Classify PRs in batches with structured output, then apply label overrides and a coverage check.
  5. Write one section per audience with a style guide, examples, and a required PR reference on every bullet.
  6. Run the reference, coverage, glossary, and number checks; regenerate failing sections.
  7. Open the draft as a pull request for a human editor, and publish after approval.

Most teams can wire this into an existing release workflow in a day or two. The result is not notes the model wrote on its own. It is notes your engineers effectively wrote across forty pull requests, translated, organized, and checked so a person can approve them quickly and readers can trust every line.

Comments

Post a Comment

Popular posts from this blog

Grok Bot - a step closer to AGI

Three years ago, Grok was a late arrival with a punchline. Today it is one of the most visible AI assistants on Earth: baked into a global social network, sitting in cars, writing code in developer tools, generating billions of images a month, and shipping new frontier models on a cadence that would have looked reckless in 2024. “Taking the world by storm” is marketing language. In Grok’s case, the numbers, the distribution, and the cultural footprint make the phrase unusually hard to dismiss. The storm did not start with a polite product launch. It started with a personality. A chatbot that refused to sound like a chatbot When xAI introduced Grok in late 2023, the market already belonged to ChatGPT. Google was racing Gemini into every surface it owned. Anthropic was building a reputation for careful, high-end reasoning. Grok’s first advantage was not raw benchmark supremacy. It was attitude. The model was designed to be useful, maximally truth-seeking, and willing to answer questions ...
Read more

Structured Outputs and Constrained Decoding for Reliable LLM APIs in Production (September 2026)

Structured Outputs and Constrained Decoding for Reliable LLM APIs in Production (September 2026) Large language models are excellent at producing useful text, but production software rarely needs “useful text” alone. It needs data that can be parsed, validated, stored, routed, and acted on without surprising the rest of the system. That is why structured outputs and constrained decoding have become important parts of reliable LLM API design. Instead of asking a model to “return valid JSON,” production teams can define an explicit JSON Schema, constrain generation to an allowed grammar, and validate the result before it reaches application code. Why ordinary JSON prompting is not enough A prompt such as “Return your answer as JSON” is a formatting request, not a guarantee. The model may wrap the object in Markdown fences, add an explanation before or after it, omit a required field, return a number as a string, or produce a value that is syntactically valid but unusable. Even a...
Read more

Tencent Hy4 Preview: Open 770B MoE Built for Real Work

Image
Late August brought another big open-weight release from China, and this one is aimed squarely at people who ship software and run analyses for a living. On August 28, 2026, Tencent’s Hunyuan (Hy) team published Hy4 preview : a mixture-of-experts language model with 770 billion total parameters , about 49 billion active per token , a 1 million token context window, and weights under Apache 2.0 . Official Hy4 preview branding from Tencent’s Hugging Face model card (tencent/Hy4-preview). That combination matters more than the headline parameter count. You get frontier-scale capacity with MoE routing that keeps per-token compute closer to a ~50B dense model, a license that allows commercial and on-prem use, and serving recipes for vLLM and SGLang that already include speculative decoding. Reuters also noted that Tencent plans to wire the model into products such as CodeBuddy and WorkBuddy, so the release is not only a research drop. What Hy4 preview actually is Hy4 preview is ...
Read more