Near-Duplicate Detection for Production RAG Ingestion in October 2026: Exact Hashes, MinHash, and Embedding Checks So Retrieval Stops Returning the Same Passage Five Times
Most retrieval problems people blame on embeddings are really data problems. One of the most common: the same content is indexed many times. A help article that was copied into three product spaces, a policy PDF uploaded as v2, v2-final, and v2-final-signed, a wiki page with a boilerplate footer that appears on every page, or a support macro pasted into thousands of tickets. When a user asks a question, the top five retrieved chunks can all be the same paragraph with slightly different whitespace. The model sees one fact five times, misses the four other facts it needed, and you pay for the extra context tokens.
This post walks through a practical deduplication pipeline for RAG ingestion: what to deduplicate, which technique catches which kind of duplicate, how to pick thresholds without guessing, and how to keep the rest of your system (citations, permissions, deletions) consistent once duplicates are collapsed.
Why duplicates hurt retrieval more than they hurt storage
Storage is cheap, so it is tempting to ignore duplicates. The real costs show up at query time:
- Crowded top-k. If you retrieve 8 chunks and 5 are near-copies, your effective top-k is 4. Recall on multi-part questions drops even though every individual result looks relevant.
- Overweighted content. Text that appears many times wins more similarity matches simply because there are more copies of it. Boilerplate, disclaimers, and navigation text get pulled into answers they have nothing to do with.
- Version confusion. When an old and a new version of a document are both indexed, the model may quote the outdated one, or mix the two into an answer that matches neither.
- Wasted tokens and reranker calls. Every duplicate chunk you pass to a cross-encoder or into the prompt costs the same as a useful one.
- Skewed evaluation. If your eval set was built from the same corpus, duplicated passages make retrieval metrics look better than real-world behavior.
Three kinds of duplicates, three tools
Not all duplicates look alike, and no single technique catches all of them cheaply. It helps to separate them:
- Exact duplicates: byte-identical or identical after normalization (same text, different whitespace, casing, or Unicode forms). Catch these with a hash.
- Near duplicates: mostly the same words with small edits, such as a changed date, an added sentence, or a different header. Catch these with MinHash or SimHash over word shingles.
- Semantic duplicates: different wording, same meaning, such as a translated FAQ or a paraphrased policy summary. Only embeddings catch these, and you should be careful about collapsing them at all.
Run them in that order. Each stage is more expensive and less precise than the one before, so you want the cheap stages to remove as much as possible first.
Stage 1: Normalize, then hash
Exact hashing is only useful if you normalize first. Two chunks that differ only in trailing spaces or smart quotes should hash the same. A reasonable normalizer:
import hashlib, re, unicodedata
def normalize(text: str) -> str:
text = unicodedata.normalize("NFKC", text)
text = text.lower()
text = re.sub(r"\s+", " ", text)
return text.strip()
def content_hash(text: str) -> str:
return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()
Store the hash on every chunk record. At ingestion, look it up before embedding: if the hash already exists, skip the embedding call and link the new source to the existing chunk instead. This alone often removes a large share of duplicate work in corpora built from file shares and ticket systems, and it saves embedding cost directly.
Be deliberate about what normalization removes. Lowercasing is usually safe for retrieval, but stripping numbers or punctuation can merge chunks that genuinely differ, such as two price tables or two config snippets. Keep normalization conservative and let the near-duplicate stage handle fuzzier cases.
Stage 2: Near duplicates with MinHash and LSH
MinHash estimates the Jaccard similarity between two sets, which for text means the overlap between their sets of shingles (overlapping word n-grams). Locality-sensitive hashing (LSH) then buckets signatures so you only compare chunks likely to be similar, instead of comparing every pair.
The open-source datasketch library in Python implements both. A minimal version:
from datasketch import MinHash, MinHashLSH
NUM_PERM = 128
def shingles(text: str, n: int = 5):
words = normalize(text).split()
if len(words) < n:
return {" ".join(words)}
return {" ".join(words[i:i + n]) for i in range(len(words) - n + 1)}
def minhash(text: str) -> MinHash:
m = MinHash(num_perm=NUM_PERM)
for s in shingles(text):
m.update(s.encode("utf-8"))
return m
lsh = MinHashLSH(threshold=0.8, num_perm=NUM_PERM)
def check_and_insert(chunk_id: str, text: str):
m = minhash(text)
candidates = lsh.query(m)
if candidates:
return candidates # near-duplicate of existing chunks
lsh.insert(chunk_id, m)
return []
A few practical notes:
- Shingle size matters. Word 5-grams are a common starting point for prose. Shorter shingles make unrelated text look more similar; longer ones make small edits look like big differences.
- The LSH threshold is approximate. LSH returns candidates that are probably above the threshold. If precision matters, compute the estimated Jaccard with
m.jaccard(other)for each candidate before deciding. - Short chunks are noisy. A two-sentence chunk has few shingles, so one changed word swings the score a lot. Either skip near-dup checks below a minimum length or use a stricter threshold for short text.
- Persist the signatures. Store MinHash signatures alongside chunks so you can rebuild the LSH index without recomputing everything when the service restarts.
SimHash is a common alternative: it produces a single 64-bit fingerprint per document, and near duplicates differ in only a few bits (a small Hamming distance). It is compact and fast for whole documents. MinHash is usually easier to reason about for chunk-level dedup because its score maps directly to word overlap.
Stage 3: Semantic duplicates with embeddings, carefully
You already compute embeddings for retrieval, so checking cosine similarity against the nearest existing chunk is cheap. The problem is precision. Two chunks can be very close in embedding space and still differ in the detail that matters: "refunds are allowed within 30 days" and "refunds are allowed within 14 days" will often score as extremely similar.
Because of that, treat embedding similarity at ingestion as a flag, not an automatic delete:
- Use a very high threshold, and pick it from your own data, not from a blog post. Sample pairs at several similarity levels, label them as "same content" or "different content," and find the point where false merges become rare.
- Combine signals. Collapse automatically only when embedding similarity is high and MinHash similarity is also reasonably high. Pairs that are close in meaning but low in word overlap are often translations or rewrites that may deserve to stay separate.
- Route ambiguous pairs to a review queue or keep both and handle them at query time.
Query-time diversification as a safety net
Ingestion-time dedup will never be perfect, and some duplicates arrive through sources you do not control. Add a cheap pass after retrieval and before the prompt:
- Collapse by content hash and MinHash cluster so only one member of each duplicate group survives.
- Apply Maximal Marginal Relevance (MMR). MMR picks results that are relevant to the query but dissimilar to results already chosen. Many vector stores and frameworks offer it as a search option; it trades a little raw relevance for coverage.
- Cap results per source document, for example no more than two chunks from the same file, so one long document cannot fill the whole context window.
Choosing which copy to keep
When you collapse a group, you need a canonical member. Write the rule down so it is consistent:
- Prefer the most recently modified authoritative source (for example, the official docs site over a pasted copy in a ticket).
- Prefer the version with the most complete text if lengths differ.
- Break ties deterministically, such as by lowest source ID, so reruns produce the same result.
Do not throw away the other copies' metadata. Keep a mapping from each canonical chunk to all of its source locations. That mapping is what keeps the rest of the system correct.
Keep citations, permissions, and deletions consistent
Dedup interacts with three things that are easy to break:
- Citations. If the canonical chunk came from a source the user cannot open, citing it is useless. Store all source locations and cite one the user has access to.
- Permissions. This is the dangerous one. A merged chunk must never become visible to someone who could not see at least one of its sources. For text that is truly identical, it is safe to grant access to anyone who can open any of the sources, because they can already read those exact words. Never merge across permission boundaries based on semantic similarity alone, since "similar" text from a restricted document can carry details the public version does not.
- Deletions. When a source is deleted, remove it from the chunk's source list. Delete the chunk itself only when no sources remain. Otherwise, deleting one copy silently removes content that other live documents still contain.
Handling boilerplate separately
Headers, footers, legal disclaimers, and navigation menus are a special case of duplication: the same block appears inside many otherwise unique documents. Chunk-level dedup does not remove them because each chunk contains unique text too.
A simple fix is to count how often each normalized line or paragraph appears across the corpus during ingestion. Lines that appear in a large fraction of documents from the same source are almost always boilerplate; strip them before chunking. Review the list once per source, because some repeated lines (like a product name in a heading) carry meaning.
Measuring whether it worked
Track a few numbers before and after rolling out dedup:
- Duplicate rate in top-k: for a sample of real queries, the share of retrieved chunks that belong to the same duplicate cluster as another result.
- Unique sources per answer: how many distinct documents contribute to the context.
- Retrieval recall on your golden set, rebuilt so that duplicates of the same passage count as one hit.
- Embedding calls and index size, which should drop with exact-hash dedup.
- False merges, found by spot-checking collapsed groups. One wrong merge of two different policies is worse than a dozen duplicates left in the index.
A rollout checklist
- Add normalized content hashes to every chunk and skip re-embedding exact matches.
- Strip corpus-wide boilerplate lines per source before chunking.
- Add MinHash signatures and an LSH index; start with a high threshold and verify candidates with the estimated Jaccard score.
- Store a canonical-chunk-to-sources mapping with each source's ACL and modification time.
- Flag, but do not auto-merge, embedding-only near matches until you have labeled examples.
- Add query-time collapse, MMR, and a per-document cap.
- Update deletion handling to remove sources from chunks before deleting chunks.
- Measure duplicate rate in top-k and spot-check merges every time you change thresholds.
Deduplication is unglamorous, but it is one of the cheapest ways to make a RAG system feel smarter. The model can only reason over what you put in front of it, and giving it five different facts instead of one fact five times is a direct improvement in answer quality.
Comments
Post a Comment