If you’ve built a RAG (Retrieval-Augmented Generation) system in the past year, you’ve probably hit this wall: your LLM returns confidently wrong answers, cites information that doesn’t exist, or completely misses relevant context sitting right there in your vector database.
The problem isn’t your embedding model or vector store. Most RAG implementations treat context like a keyword search problem when it’s actually a meaning problem. Traditional RAG chunks documents, embeds them, retrieves the “closest” chunks, and feeds them to the LLM. In practice, this breaks down when chunks lose their surrounding context. A sentence like “It increased by 40%” is useless without knowing what “it” refers to or when this happened.
Contextual retrieval explicitly preserves and leverages the relationships between chunks, their document structure, and their semantic meaning rather than treating each chunk as an isolated island of text.
In this guide, we’ll break down what context means in RAG systems, why naive chunking fails, and how modern contextual retrieval techniques solve these problems without over-engineering your infrastructure.
What is Context in RAG Systems?
Before we talk about retrieval, let’s be precise about what “context” actually means in a RAG pipeline. Context isn’t one thing; it’s several layers that interact.
1. Chunk Context (Local)
This is the immediate surrounding text for any given chunk. Without this, references like “as mentioned above” or “this approach” become meaningless.
Failure mode: Your chunk says “This reduced latency by 60%” but doesn’t mention that “this” refers to switching from EBS to local NVMe, which was explained two paragraphs earlier in a different chunk.
2. Document Context (Structural)
This is metadata about where the chunk lives: which document, section, content type (API docs vs. blog), purpose, and audience.
Failure mode: Your LLM retrieves a chunk from a 2023 deprecation notice when the user asked about current 2026 best practices. The content was relevant once, but temporal context makes it dangerously wrong now.
3. Semantic Context (Global)
This is the web of relationships between concepts across your entire knowledge base. How does this chunk relate to others semantically, even across different documents?
Failure mode: A user asks “How do I optimize cold starts?” and your system retrieves chunks about Lambda functions but misses critical chunks about VPC configuration, provisioned concurrency, and SnapStart because they live in different documents without shared keywords.
Most RAG implementations only handle the first type, if that. Contextual retrieval systems explicitly address all three.
The Problem with Naive Chunking
Traditional RAG follows a simple recipe:
- Split documents into chunks (fixed size, e.g., 512 tokens with 50-token overlap)
- Generate embeddings for each chunk
- Store embeddings in a vector database
- On query: embed the query, find nearest neighbors, return top-k chunks
- Stuff those chunks into the LLM prompt
This worked well enough for early demos but in production, it falls apart quickly.
Why Fixed-Size Chunking Breaks
Imagine you are chunking technical documentation about database configuration. A naive fixed-size chunker might produce:
Chunk 1:
our benchmark results on the z3-highmem-14 instance.
MongoDB was configured with WiredTiger and 100GB cache.
Testing Methodology
We used YCSB 0.18.0 with 1 billion records and uniform
distribution. Each test ran 2 million operations across
varying thread counts.
Chunk 2:
varying thread counts. Read throughput peaked at 8,000 QPS
for MongoDB and 10,000 QPS for FerretDB. However, EloqDoc
reached 129,000 QPS at 512 threads due to its use of local
NVMe storage rather than network-attached disks.
See the problem?
- Chunk 1 contains critical setup information but gets cut off mid-context
- Chunk 2 starts with “varying thread counts” (meaningless without Chunk 1) and references “its use of local NVMe” without explaining what “it” is
- The most important finding (EloqDoc’s 16x performance advantage) is explained using a pronoun that references content in a completely different chunk.
When someone searches for “database performance comparison,” they might retrieve Chunk 2, which confidently states “129,000 QPS” without any context about what system that refers to, what workload was tested, or how it compares to alternatives.
Why Partial Overlap Alone Fails to Solve the Problem
Many developers add 10-20% overlap between chunks assuming it fixes the problem. It doesn’t. Overlap helps with boundary splits (not cutting sentences in half), but does nothing for semantic coherence. If relevant context is 200 or 500 tokens away, overlap won’t help.
Common Failure Patterns
Common failure modes from production RAG systems that can occur in your system too, are:
- Pronoun hell: “It supports both modes” - what is “it”?
- Orphaned comparisons: “This is 3x faster” - faster than what?
- Broken procedures: Step 3 of a tutorial in a different chunk than steps 1-2
- Lost temporal markers: “As of last quarter” - which quarter?
- Missing prerequisites: Code assumes imported libraries mentioned in a different chunk
The core issue is that fixed-size chunking treats documents as strings to split, not as structured information with semantic boundaries.
How Contextual Retrieval Works
Contextual retrieval solves these problems by explicitly preserving and leveraging context at chunk creation time, not retrieval time. The key insight is that you can’t recover lost context later; you must embed the context into the chunk itself before embedding and indexing.
Think of it like this: naive chunking is like ripping pages out of a book at random. Contextual retrieval is like carefully extracting sections while writing a summary of the book on each page so that each page makes sense in isolation.
Anthropic’s Contextual Embeddings Approach
Anthropic published a technique called Contextual Retrieval in late 2024 that aims at improving RAG accuracy. The approach is that before embedding a chunk, prepend it with a brief context summary that explains what this chunk is about and where it sits in the document.
Here’s how it works in practice:
Original Chunk (Naive RAG):
varying thread counts. Read throughput peaked at 8,000 QPS
for MongoDB and 10,000 QPS for FerretDB. However, EloqDoc
reached 129,000 QPS at 512 threads due to its use of local
NVMe storage rather than network-attached disks.
Contextualized Chunk (Contextual RAG):
This chunk is from a database performance benchmark comparing
MongoDB, FerretDB, and EloqDoc on a 1TB dataset with 1 billion
records, conducted in January 2026. The section discusses read
throughput results under high concurrency.
varying thread counts. Read throughput peaked at 8,000 QPS
for MongoDB and 10,000 QPS for FerretDB. However, EloqDoc
reached 129,000 QPS at 512 threads due to its use of local
NVMe storage rather than network-attached disks.
Now when this chunk is embedded, the vector representation includes the context. When a user searches for “database read performance 2026” this chunk will match more accurately because the embedding captures both the content AND its context.
Generating Contextual Summaries with LLMs
The trick is generating these contextual summaries efficiently. Anthropic’s approach uses an LLM (like Claude) with a prompt like this:
Here is the document:
<document>
{{FULL_DOCUMENT}}
</document>
Here is the chunk we want to situate within the document:
<chunk>
{{CHUNK_CONTENT}}
</chunk>
Please provide a concise context (2-3 sentences) that explains
what this chunk is about and where it fits in the document.
This context will be prepended to the chunk to improve retrieval.
The LLM reads the full document and the specific chunk, then generates a summary that situates the chunk in its broader context. This summary is prepended to the chunk before embedding.
Hybrid Retrieval: BM25 + Contextual Embeddings
Anthropic’s research also found that combining contextual embeddings with traditional BM25 (keyword search) dramatically outperforms either method (as above) alone. The reason is that embeddings capture semantic meaning, while BM25 captures exact keyword matches.
Here’s a realistic scenario where hybrid search would work efficiently:
User Query: “What is the pricing for Claude Sonnet API in 2026?”
- BM25 Result: Finds chunks with exact matches for “pricing”, “Claude Sonnet”, “API”, “2026”
- Semantic Result: Finds chunks about billing, costs, API plans, even if they don’t use those exact words
- Hybrid Result: Combines both, heavily weighting chunks that match both semantically AND contain the key terms
Implementation Pattern
The practical workflow is straightforward i.e. to split documents along meaningful and semantic boundaries. For each chunk, use an LLM to generate a brief context summary and prepend it to the chunk before embedding. Store both the contextualized embedding and the original chunk in your vector store. When retrieving, use a hybrid approach that combines BM25 with vector similarity, then rerank the results with a dedicated model for relevance. Finally, pass only the original chunks (without the generated context) to the LLM, minimizing prompt size. The context summary boosts retrieval accuracy but isn’t needed by the LLM itself during answer generation.
Smarter Chunking Strategies
Contextual retrieval is most effective when chunking is based on document structure instead of fixed token counts.
Three Approaches to Better Chunking
1. Semantic Chunking splits based on meaning by embedding sentences and creating boundaries when similarity drops. Libraries like LangChain provide this out of the box (source):
from bs4 import Tag
from langchain_text_splitters import HTMLSemanticPreservingSplitter
headers_to_split_on = [
("h1", "Header 1"),
("h2", "Header 2"),
]
def code_handler(element: Tag) -> str:
data_lang = element.get("data-lang")
code_format = f"<code:{data_lang}>{element.get_text()}</code>"
return code_format
splitter = HTMLSemanticPreservingSplitter(
headers_to_split_on=headers_to_split_on,
separators=["\n\n", "\n", ". ", "! ", "? "],
max_chunk_size=50,
preserve_images=True,
preserve_videos=True,
elements_to_preserve=["table", "ul", "ol", "code"],
denylist_tags=["script", "style", "head"],
custom_handlers={"code": code_handler},
)
documents = splitter.split_text(html_string)
2. Structural Chunking uses document structure (headers, sections, code functions) as natural boundaries (source):
from langchain_text_splitters import MarkdownHeaderTextSplitter
markdown_document = "# Foo\n\n ## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo\n\n Hi this is Lance\n\n ## Baz\n\n Hi this is Molly"
headers_to_split_on = [
("#", "Header 1"),
("##", "Header 2"),
("###", "Header 3"),
]
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on)
md_header_splits = markdown_splitter.split_text(markdown_document)
3. Agentic Chunking uses an LLM to identify logical breakpoints. This is expensive but produces the highest quality chunks for high-stakes applications (medical, legal, financial) (source).
Reranking, a two-stage retrieval
Even with contextual embeddings and smart chunking, vector similarity alone isn’t enough. This is where reranking comes in.
Reranking is a two-stage retrieval process: first retrieve a large candidate set (e.g., top 100 chunks), then use another model to rerank those candidates and return the true top-k.
The reason this works is that the first-stage retriever (vector search) is fast but imprecise. It casts a wide net. The reranker is slow but accurate. It carefully evaluates each candidate against the query and scores them properly.
Why Reranking Matters
Vector embeddings capture semantic similarity, but they don’t capture relevance. A chunk can be semantically similar to a query without actually answering it.
Example Query: “How do I reduce cold starts in Lambda?”
- Vector Match: Returns chunks about Lambda, cold starts, initialization time, etc. Some are relevant, some are tangentially related.
- Reranked Results: Prioritizes chunks that contain actionable advice (provisioned concurrency, SnapStart, VPC configuration) over chunks that merely mention cold starts in passing.
Rerankers are trained specifically to predict relevance given a (query, document) pair. They’re much better at this task than general-purpose embedding models.
Graph-Based Contextual Retrieval
An emerging alternative to chunk-based RAG is graph-based retrieval, where you model your knowledge base as a graph of entities and relationships.
Why Graphs Work
Chunks are isolated units, even with contextual embeddings. Graphs explicitly model relationships between information.
Example: For a company’s internal docs with chunks about “Project Phoenix”, “Sarah Chen” (project lead), and “Q4 2025 roadmap”, a vector database has no connection between them unless they mention each other explicitly.
With a graph, you create nodes (entities) and edges (relationships): Sarah Chen → leads → Project Phoenix → part_of → Q4 2025 Roadmap. When someone asks “What projects is Sarah working on?”, you traverse the graph to gather all related context in one query.
Common Pitfalls and How to Avoid Them
From building production RAG systems, here are the mistakes that may happen:
- Over-Optimizing Embeddings, Under-Optimizing Chunking: Obsessing over embedding models while using terrible fixed-size chunking. Chunking quality matters more than embedding quality. The fix is to invest efforts in semantic/structural chunking first.
- Ignoring Metadata: Not using metadata filters even though your vector database can. Simple info like
{document_type: "api_docs", last_updated: "2026-03"}can make search much better. The fix is to collect detailed metadata when you add documents and use it to filter results first. - Single-shot retrieval: More effective systems use iterative retrieval, where they retrieve some information, generate a partial answer, then perform another retrieval if needed before producing the final response. To enable this approach, you can use agentic frameworks like AutoGPT.
- No Fallback Strategy: When retrieval finds zero relevant chunks, most systems pass empty context to the LLM, which then hallucinates. The fix is to implement a threshold logic, i.e. if score < threshold, respond “I don’t have enough information.”
Context is Everything
If there’s one takeaway from this guide, it’s that context is not a nice-to-have in RAG systems, it’s the foundation for ensuring high quality output.
Naive chunking and pure vector similarity search worked well enough when RAG was new and expectations were low. In 2026, users expect answers that are accurate, complete, and grounded in your actual data. You can’t deliver that with fixed-size chunks and a simple nearest-neighbor search.
Contextual retrieval whether through contextual embeddings, graph-based approaches, or hybrid methods explicitly preserves and leverages the relationships between chunks, their document structure, and their semantic meaning.
You can start simply by adding contextual embeddings to your existing chunks, layer in a reranker, and switch from fixed-size to semantic chunking. These three changes alone will help optimize your retrieval failures.