ACE Reference: Retrieval Augmentation

Retrieval augmentation transforms how agents access and use knowledge. Instead of relying solely on what fits in a context window or what a model learned during training, agents can dynamically pull relevant information from external sources at inference time. This capability fundamentally changes what agents can know and do.

The patterns in this reference specification address a core challenge in agentic systems: how to give an agent access to vast knowledge bases while operating within fixed context window constraints. The answer isn't to cram more tokens into the window. It's to retrieve the right tokens at the right time.

Without retrieval augmentation, agents face a fundamental limitation: they can only work with information explicitly provided in the prompt or learned during training. Training data becomes stale. Context windows fill up. Agents confidently hallucinate answers they should have looked up. Retrieval changes this dynamic by making external knowledge accessible on demand.

The investment in retrieval infrastructure pays off across use cases. Customer support agents can cite current documentation. Research assistants can ground analysis in recent publications. Code assistants can reference the actual codebase. Personal assistants can recall past conversations and decisions. Each use case benefits from the same underlying retrieval patterns.

This document covers retrieval augmentation from foundational concepts through production implementation. Readers looking for a quick overview can stop after the Core Concepts section. Those building systems should continue through Implementation Patterns. Advanced practitioners will find edge cases and optimization strategies in the final sections.

Core Concepts

What Retrieval Augmentation Enables

Retrieval augmentation decouples knowledge from training. A model trained months ago can answer questions about documents created yesterday, because retrieval happens at inference time rather than during training. This temporal decoupling is the first capability gain.

The second capability gain is scale. An agent's knowledge is no longer bounded by what fits in memory or context. A retrieval system can index millions of documents while the agent operates with a fixed context window. The agent doesn't need to hold everything - it needs to find the right things.

The third capability gain is grounding. Retrieved content provides concrete evidence that the agent can cite, quote, and reference. This reduces hallucination by giving the model actual text to draw from rather than relying entirely on parametric knowledge.

The Retrieval Pipeline

Every retrieval augmentation system follows a common flow:

Query Processing - The user query or agent state gets transformed into a form suitable for retrieval. This might involve query expansion, intent extraction, or embedding generation. A well-processed query captures user intent even when the original phrasing is ambiguous or incomplete.

Candidate Retrieval - The processed query searches an index to find potentially relevant documents. This stage prioritizes recall - finding everything that might be relevant. Better to retrieve some irrelevant documents than to miss critical ones; later stages can filter.

Reranking - Retrieved candidates get scored and ordered by relevance to the original query. This stage prioritizes precision - sorting the wheat from the chaff. Expensive reranking models become affordable when applied only to the candidate set, not the entire corpus.

Context Assembly - Top-ranked documents get formatted and packed into the context window. This stage handles token budgets, ordering effects, and metadata inclusion. The format of assembled context affects how the model uses it.

Generation - The model produces output conditioned on both the query and the retrieved context. The quality of this output depends entirely on what the previous stages delivered. A perfect generation model cannot compensate for bad retrieval.

Each stage has distinct failure modes. Poor query processing retrieves irrelevant documents. Weak candidate retrieval misses critical information. Bad reranking buries important evidence. Careless context assembly wastes tokens or introduces noise. Understanding the pipeline means understanding where things can go wrong.

The pipeline is not always linear. Iterative retrieval loops back from generation to query processing when initial results prove insufficient. Multi-stage retrieval chains candidate retrieval through multiple indexes. The core stages remain, but their arrangement varies.

Vector Search Fundamentals

Vector search converts the retrieval problem from keyword matching to similarity matching in a learned representation space. Documents and queries get mapped to high-dimensional vectors where proximity indicates semantic relatedness.

An embedding model performs this mapping. The model processes text and outputs a fixed-length vector - typically 384 to 3072 dimensions depending on the model. These vectors capture semantic content: documents about similar topics cluster together even when they use different vocabulary. Two documents describing authentication flows will be closer together than either is to a document about database indexing, even if all three use technical jargon.

Similarity computation finds nearby vectors. Cosine similarity compares vector angles and remains robust to magnitude differences. Dot product offers efficiency when vectors are normalized. Euclidean distance measures raw geometric distance but is sensitive to magnitude. The choice matters less than consistency - use the same metric at index time and query time.

Approximate nearest neighbor (ANN) algorithms make vector search fast at scale. HNSW (Hierarchical Navigable Small World) graphs build navigable structures that allow quickly narrowing to relevant regions. IVF (Inverted File Index) structures partition the space and search only relevant partitions. Both trade tiny accuracy loss for orders-of-magnitude speedup. A billion-vector index that would take minutes to search exactly can return results in milliseconds with ANN.

The quality of vector search depends entirely on embedding quality. An embedding model that doesn't understand your domain produces vectors where similar documents might not be close. Evaluation on domain-specific queries reveals whether the embedding model captures relevant similarity.

Hybrid Retrieval Strategies

Vector search excels at semantic similarity but can miss exact matches. Someone searching for "RFC 7231" expects documents containing that exact string, not documents semantically similar to HTTP specification topics.

Hybrid retrieval combines dense vector search with sparse keyword search (typically BM25). The two retrieval paths run in parallel, and their results get merged through score fusion.

The combination captures both semantic matches (finding documents about the same concept using different words) and lexical matches (finding documents with specific terms, codes, or identifiers). Production systems almost always benefit from hybrid approaches.

Score fusion requires calibration. Vector similarity scores and BM25 scores exist on different scales with different distributions. Reciprocal rank fusion (RRF) sidesteps this by combining based on rank position rather than raw scores. More sophisticated approaches learn fusion weights from relevance judgments.

Context Window Optimization

The context window is a fixed budget. Every token spent on retrieved content is a token not available for system instructions, conversation history, or generation. Efficient context assembly means maximizing signal per token.

Token budgeting allocates the window across competing needs. A typical breakdown might reserve 20% for system prompt, 30% for conversation history, 40% for retrieved context, and 10% for generation headroom. These proportions shift based on use case. A simple Q&A system might allocate more to retrieval; a multi-turn reasoning system needs more history.

Chunk ordering matters. Models exhibit position bias - information at the beginning and end of context gets more attention than information in the middle. Critical evidence should appear early. The "lost in the middle" phenomenon is real and affects retrieval accuracy measurably. Experiments show that identical evidence placed at position 3 versus position 10 can change answer quality.

Deduplication removes redundant information. When multiple retrieved chunks contain overlapping content, including all of them wastes tokens and can confuse the model with repetition. Near-duplicate detection at retrieval time or assembly time improves efficiency. Semantic similarity between chunks, not just exact matching, identifies this redundancy.

Metadata inclusion adds context without adding prose. A chunk tagged with source, date, and confidence score can be processed by the model without needing those facts embedded in the text itself. Structured metadata formats (JSON, XML) parse reliably; prose metadata can blur into content.

Compression techniques reduce token count while preserving information. Summarization condenses long documents. Extraction pulls key sentences. Both trade fidelity for efficiency. The right compression depends on what downstream generation needs - some tasks require exact quotes, others need only gist.

Dynamic allocation adjusts budgets based on query complexity. A simple factual question might need only two chunks; a complex analysis might need twenty. Static allocation either wastes tokens on simple queries or starves complex ones. Adaptive systems estimate complexity and allocate accordingly.

Overflow handling addresses cases where relevant content exceeds the window. When twenty chunks are relevant but only five fit, the system must either truncate, summarize, or split the request into multiple retrievals. Each approach has tradeoffs - truncation loses information, summarization risks distortion, splitting increases latency.

Implementation Patterns

Chunking Strategies

Documents must be segmented into chunks before indexing. Chunking decisions propagate through the entire system - they affect embedding quality, retrieval precision, context efficiency, and generation coherence.

Fixed-size chunking splits documents into segments of consistent token count with overlap. Simple to implement, predictable for token budgeting, but ignores document structure. A 512-token chunk might split mid-sentence or mid-paragraph.

Semantic chunking respects document structure - paragraphs, sections, code blocks. Chunks correspond to meaningful units of content. More complex to implement, requires document structure detection, but produces more coherent retrieval units.

Hierarchical chunking maintains both coarse and fine representations. A document might exist as a single summary chunk and also as multiple detail chunks. Query routing can select the appropriate granularity.

Sliding window with overlap addresses boundary effects. If a 500-token chunk has 100 tokens of overlap with adjacent chunks, information near boundaries appears in multiple chunks. This redundancy improves recall at the cost of index size.

Chunk size involves tradeoffs. Larger chunks preserve more context but dilute the embedding signal - a 2000-token chunk about three topics produces a fuzzy embedding. Smaller chunks produce sharper embeddings but lose surrounding context. Most systems land between 200 and 1000 tokens.

Embedding Selection

The embedding model determines what "similarity" means. Different models capture different notions of relatedness.

General-purpose embeddings (OpenAI text-embedding-3, Cohere embed, BGE) work across domains but may miss domain-specific nuance. A medical question and a legal question that share structure might embed similarly even though they need different expertise.

Domain-tuned embeddings specialize in specific vocabularies and relationships. A code embedding model understands that forEach and for...of are related in ways a general model might miss.

Asymmetric embeddings use different models for queries and documents. The query encoder learns to produce embeddings that match relevant documents, while the document encoder learns to produce embeddings that capture content. This can outperform symmetric approaches where one model does both.

Multi-vector embeddings (ColBERT-style) produce multiple vectors per document rather than collapsing to a single vector. Late interaction between query and document vectors can capture fine-grained matching that single-vector approaches miss.

Embedding dimension affects storage and computation. Higher dimensions can capture more nuance but increase memory requirements and search latency. Matryoshka embeddings allow truncating vectors to lower dimensions with graceful degradation.

Retrieval Pipelines

Production retrieval rarely uses a single stage. Pipelines chain multiple retrieval and filtering operations.

Two-stage retrieval combines fast candidate generation with accurate reranking. The first stage (typically ANN vector search) retrieves many candidates quickly. The second stage (typically a cross-encoder) scores each candidate more accurately but expensively.

Query expansion generates multiple variants of the original query. If the user asks "How do I fix connection timeout errors?", expansion might add queries like "troubleshooting network connectivity" and "handling socket exceptions." Multiple queries increase recall.

Filters and facets narrow the search space before semantic matching. Date ranges, document types, access permissions, and metadata tags can all constrain retrieval. Filtering happens in the retrieval stage, not after - this improves performance and relevance.

Iterative retrieval uses initial results to refine subsequent queries. The agent retrieves initial context, reasons about what's missing, and issues follow-up retrievals. This multi-turn pattern captures information that a single query would miss.

Re-ranking Approaches

Reranking transforms rough retrieval into precise ranking. The quality of final results depends heavily on reranking effectiveness.

Cross-encoder rerankers process query and document together, producing a relevance score. Unlike bi-encoders that embed query and document separately, cross-encoders capture fine-grained interactions. They're more accurate but more expensive - typically applied to 10-100 candidates rather than millions.

LLM-based reranking uses a language model to score or sort candidates. The model receives the query and each candidate, outputting a relevance judgment. This can capture nuanced relevance that dedicated rerankers miss, but inference costs limit scale.

Learned sparse reranking uses models like SPLADE that output sparse term weights. These combine the efficiency of sparse retrieval with learned representations.

Ensemble reranking combines multiple rerankers. A cross-encoder, an LLM reranker, and a learned sparse model each produce scores that get fused into a final ranking. Ensembles improve robustness when individual rerankers have different failure modes.

Agentic Retrieval Patterns

Agents don't just receive queries - they formulate them. This changes how retrieval gets used.

Self-directed retrieval lets the agent decide when to retrieve. Instead of retrieving on every turn, the agent assesses whether its current context is sufficient. If not, it issues a retrieval call. This reduces unnecessary retrieval and keeps context focused. The agent might reason: "I need to answer a question about API rate limits. I don't have that information in my current context. I should search the API documentation."

Tool-mediated retrieval exposes retrieval as a callable tool. The agent can search specific indexes, specify filters, control result count, and iterate based on results. Retrieval becomes a capability the agent orchestrates rather than a fixed preprocessing step. Tool schemas define what parameters the agent can control.

Context accumulation builds retrieval results across conversation turns. Early turns might retrieve background information, later turns might retrieve specific details. The agent manages what stays in context and what gets dropped. This requires explicit context management - deciding what to keep, what to summarize, and what to discard.

Retrieval planning has the agent reason about what information it needs before issuing queries. Rather than searching immediately, the agent might identify three distinct information needs and plan retrievals for each. This front-loaded reasoning produces better queries than reactive retrieval.

Verification retrieval searches for evidence that confirms or contradicts a claim. After the agent generates an answer, it can retrieve documents to check whether that answer is supported. This self-verification pattern reduces hallucination by adding an evidence-based review step.

Comparative retrieval gathers multiple perspectives on a topic. Instead of retrieving the single best match, the agent retrieves documents with different viewpoints, methodologies, or conclusions. This supports more nuanced analysis than single-document retrieval.

Chain-of-retrieval decomposes complex questions into sequential retrievals where each step informs the next. The answer to "What's the performance impact of the authentication change introduced in v2.3?" requires first retrieving what changed in v2.3, then retrieving performance data related to that specific change. The first retrieval shapes the second query.

Advanced Topics

Query Understanding and Transformation

The quality of retrieval depends on understanding what the user actually needs, which often differs from what they literally asked.

Intent classification categorizes queries to route them appropriately. A factual lookup ("What is the API rate limit?") needs different handling than an exploratory query ("How should I architect this system?"). Classification can trigger different retrieval strategies, index selections, or result presentation.

Query decomposition breaks complex questions into retrievable parts. The question "How does our authentication system compare to the new OAuth 2.1 spec?" contains two implicit retrievals: understanding the current system, understanding OAuth 2.1. An agent that recognizes this structure can retrieve both independently and synthesize.

Temporal reasoning interprets time references in queries. "Recent changes to the API" means something different depending on when the query is issued. Systems must ground temporal language to actual date ranges for filtering.

Entity resolution connects mentions to canonical references. When a user asks about "the Smith proposal," the system must resolve which Smith, which proposal, and whether aliases or abbreviations point to the same document.

Negative constraints identify what should be excluded. "API documentation but not the deprecated v1 endpoints" requires filtering out relevant-seeming results that the user explicitly doesn't want. Standard similarity search doesn't naturally handle negation.

Retrieval Memory and State

Retrieval in agentic systems isn't stateless. The agent's history affects what should be retrieved next.

Conversation context shapes query interpretation. If the last three turns discussed authentication, a query about "the configuration file" probably means the auth configuration, not any configuration file. Retrieval should condition on conversation state.

Previously retrieved context affects what to retrieve next. If the agent already has document A in context, retrieving document A again wastes tokens. More subtly, if A and B overlap significantly, retrieving B when A is present might add little new information.

User model integration personalizes retrieval based on known preferences, expertise level, or past interactions. A senior engineer and a new hire asking the same question might benefit from different retrieved content.

Session continuity maintains retrieval state across interactions. If a user returns to a conversation after a break, the agent should remember what was previously retrieved and what information needs might remain unfulfilled.

Multi-modal Retrieval

Text isn't the only modality worth retrieving. Images, code, tables, and structured data all contain information agents might need.

Image retrieval uses vision encoders (CLIP, SigLIP) to embed images into the same space as text. A text query can retrieve relevant images; an image query can retrieve similar images or related text. The challenge is that image-text alignment is imperfect - an image of a cat and the word "cat" should be close, but the model must learn this mapping.

Code retrieval benefits from specialized embeddings that understand programming language structure. Code search indexes can match based on functionality rather than just variable names. A query about "sorting algorithms" should find implementations even if they use variable names like arr and pivot rather than sortingArray.

Table retrieval must handle structured data that doesn't embed well as prose. Approaches include converting tables to natural language descriptions, embedding row-by-row, or maintaining separate table indexes with specialized matching. Each approach trades fidelity against searchability.

Diagram and chart retrieval requires understanding visual relationships. A flowchart showing system architecture contains information not captured in any accompanying text. Vision-language models can extract this information, but the representations remain coarser than text embeddings.

Hybrid content - documents with text, images, code, and tables together - requires multi-modal pipelines that can retrieve and rank across modalities. The ranking challenge intensifies when a text chunk and an image are both relevant but in different ways.

Production Considerations

Moving from prototype to production introduces constraints that change system design.

Latency budgets limit pipeline complexity. If total response time must stay under 2 seconds and generation takes 1.5 seconds, retrieval gets 500 milliseconds. This constrains candidate count, reranking depth, and index complexity.

Index freshness determines how quickly new documents become searchable. Real-time indexing requires streaming architectures. Batch indexing is simpler but introduces delay. The right choice depends on how fast the underlying knowledge changes.

Embedding drift happens when embedding models get updated. An index built with one embedding model version won't work correctly with a different version. Model updates require re-indexing or careful version management.

Failure modes must be handled gracefully. Empty retrieval results shouldn't crash the system - the agent should recognize when retrieval found nothing useful. Slow retrieval should timeout rather than blocking indefinitely.

Monitoring and observability track retrieval quality over time. Metrics include retrieval latency, empty result rate, relevance scores, and downstream task performance. Without monitoring, retrieval quality degrades invisibly.

Evaluation and Iteration

Retrieval quality is measurable. Standard IR metrics - precision, recall, MRR, NDCG - quantify how well the system finds relevant documents. These metrics require relevance judgments, either human-labeled or derived from user behavior.

Offline evaluation measures retrieval against a static test set. Build a corpus, label relevant documents for queries, measure metrics. This catches regressions and enables A/B testing of retrieval changes. The challenge is building representative test sets - queries that cover the distribution of real user needs.

Online evaluation measures end-to-end task performance. If better retrieval leads to better agent outputs, the connection should be measurable. Track task completion rates, user satisfaction, or factual accuracy. This closes the loop between retrieval quality and actual user value.

Retrieval debugging identifies specific failure cases. When an agent produces a bad output, trace back: what did retrieval return? Was the relevant document in the index? Was it retrieved but ranked low? Was it retrieved and ranked high but not used? Each failure point suggests different fixes.

Iterative improvement uses debugging insights to refine the system. Add missing documents to the index. Tune reranking to surface buried evidence. Adjust chunking to preserve needed context. Improve query processing to expand narrow searches. Retrieval systems get better through cycles of measurement and refinement.

Scaling Considerations

As document collections grow, retrieval systems face scaling challenges across multiple dimensions.

Index size grows with document count. A million documents with 1536-dimension embeddings requires roughly 6GB of vector storage. At ten million documents, hardware requirements shift. At a billion, architecture decisions that worked at smaller scale may need revisiting.

Query throughput determines how many concurrent users the system can serve. ANN indexes trade query latency against throughput - serving more queries per second might mean accepting higher per-query latency. Caching frequent queries can reduce load.

Index updates become more expensive at scale. Adding a new document to a billion-document HNSW index takes longer than adding to a million-document index. Real-time indexing requires careful capacity planning.

Sharding distributes indexes across machines. Queries fan out to multiple shards, results get merged. Sharding introduces coordination overhead and complicates consistency guarantees.

Cost optimization becomes critical at scale. Embedding generation, vector storage, and similarity computation all have costs that scale with usage. Tiered storage (hot/warm/cold), aggressive caching, and query rate limiting help manage costs without sacrificing quality.

Failure Modes and Mitigations

Retrieval systems fail in predictable ways. Understanding failure modes enables building robust systems.

Empty results happen when no documents match the query. The agent should recognize this and either reformulate the query, inform the user, or proceed without retrieved context. Silently returning nothing breaks downstream processing.

All-irrelevant results happen when retrieved documents don't actually answer the query. High similarity scores don't guarantee relevance - documents can be semantically close but not useful. Confidence thresholds and relevance classifiers help detect this case.

Missing ground truth happens when the answer exists in the corpus but retrieval doesn't find it. This is the most dangerous failure - the agent confidently produces wrong output because it never saw the right evidence. Only end-to-end evaluation catches these cases.

Stale results happen when index updates lag behind source changes. The document was updated, but retrieval returns the old version. Timestamp tracking and freshness signals help users recognize potentially outdated information.

Adversarial content can poison retrieval. If malicious documents enter the index, they might be retrieved and influence agent outputs. Content filtering, provenance tracking, and output guardrails provide defense in depth.

Common Pitfalls

Experience reveals recurring mistakes in retrieval system design. Awareness prevents repetition.

Ignoring the Retrieval-Generation Gap

High retrieval metrics don't guarantee good generation. A system might achieve 95% recall - the relevant document is almost always in the retrieved set - but if it's ranked tenth and the model ignores it, the output fails anyway. End-to-end evaluation matters more than component metrics.

Over-engineering Query Processing

Sophisticated query processing can hurt more than help. Complex intent classification, aggressive expansion, and heavy transformation add latency and can distort the original query. A user who searched for exactly what they meant shouldn't have their query rewritten into something else. Simple processing with well-tuned retrieval often outperforms elaborate pipelines.

Neglecting the Index

The index is infrastructure that enables retrieval. Bad chunking, stale documents, missing content, and poor embeddings all degrade results. Many teams optimize reranking and generation while ignoring the foundation. Time spent on index quality pays dividends throughout the pipeline.

Treating Retrieval as One-Shot

Initial retrieval results are rarely sufficient for complex queries. Building systems that can only retrieve once, at the start of processing, leaves capability on the table. Agents that can retrieve iteratively - searching, reasoning, and searching again - handle more sophisticated information needs.

Underestimating Token Economics

Retrieval costs tokens, and tokens cost money and latency. Retrieving twenty chunks when three would suffice burns budget that could go to generation. Conversely, retrieving too few chunks misses critical information. The right balance requires measurement and tuning, not assumptions.

Forgetting About Freshness

Information changes. Documentation updates, policies evolve, code gets refactored. A retrieval system that worked perfectly at deployment degrades as the world changes and the index doesn't. Freshness monitoring and update pipelines are operational requirements, not optional enhancements.

Implementation Checklist

When building a retrieval augmentation system, validate each component.

Index quality:

Documents chunked at appropriate granularity for the use case
Embedding model selected for domain fit
Index tested with representative queries
Hybrid retrieval configured if exact matching matters

Retrieval pipeline:

Query processing handles common user patterns
Candidate retrieval achieves target recall
Reranking improves precision over baseline
Context assembly respects token budgets

Integration:

Agent can trigger retrieval at appropriate moments
Retrieved context flows correctly into generation
Failure cases handled gracefully
Latency within acceptable bounds

Operations:

Index updates automated
Monitoring tracks retrieval quality
Evaluation dataset exists for regression testing
Debugging tools enable failure analysis

This checklist provides a starting point. Production systems will have additional requirements based on scale, domain, and reliability needs.

Connections

This reference intersects several threads in the agentic context engineering domain.

Memory architecture patterns provide the broader context for retrieval - how agents maintain state across sessions, what gets persisted versus what gets retrieved fresh, how episodic and semantic memory interact. Retrieval is one mechanism among several for getting relevant information into agent context.

Context compilation determines how retrieved content gets assembled into effective prompts. Retrieval finds the raw material; compilation shapes it into usable context. The two capabilities work together - neither is sufficient alone.

Tool orchestration positions retrieval as one capability among many. An agent might combine retrieval with calculation, API calls, and code execution in a single reasoning chain. The retrieval tool sits alongside other tools in the agent's capability set.

Observability and debugging become critical as retrieval systems grow complex. When something goes wrong, you need to trace the failure through the pipeline. Was the document indexed? Was it retrieved? Was it ranked highly enough? Was it used? Each question requires different observability data.

Guardrails and safety apply to retrieved content as much as generated content. Malicious or incorrect information in the index can poison agent outputs. Content filtering, source verification, and output validation all play roles.

The artifacts in the agentic-context-engineering thread explore these connections in depth. Start with the thread overview for navigation guidance.

Summary

Retrieval augmentation gives agents access to knowledge beyond their context window and training data. The capability rests on a pipeline: query processing transforms requests into searchable form, candidate retrieval finds potentially relevant documents, reranking orders them by relevance, and context assembly packs the best evidence into the agent's context.

Vector search enables semantic matching - finding documents about the same concepts even when vocabulary differs. Hybrid retrieval adds lexical matching for exact terms. Together they cover the range of information needs.

Implementation requires attention to chunking, embedding selection, retrieval pipelines, and reranking. Production systems add concerns around latency, freshness, and failure handling. Agentic patterns like self-directed retrieval and retrieval planning extend the capability beyond simple search-then-generate workflows.

The common pitfalls - ignoring the retrieval-generation gap, neglecting the index, treating retrieval as one-shot - reveal where systems typically fail. Awareness enables avoidance.

Build retrieval systems incrementally. Start simple. Measure end-to-end performance. Iterate based on observed failure modes. The patterns in this reference provide the vocabulary and mental models for that iteration.