RAG (Retrieval-Augmented Generation)

RAG connects LLMs to external knowledge at query time. Instead of relying on what the model memorised during training, you retrieve relevant documents and pass them as context for generation. The model answers grounded in your data, with citations back to the source.

Over 70% of RAG failures trace back to poor retrieval or chunking decisions, not the LLM itself. This article covers the full pipeline - from document ingestion to production evaluation - with emphasis on the decisions that actually determine quality.

The RAG Pipeline, Step by Step

Before the architecture diagrams and production patterns below, it helps to walk through what actually happens between a document landing in a source system and an LLM producing a grounded answer. RAG runs in two phases: an offline indexing phase that prepares the knowledge base, and an online query phase that runs on every user request.

Phase 1: Document indexing (offline)

This phase runs once per document, and again whenever source content changes.

1. Collect documents - pull source material from PDFs, Word documents, wikis, databases, or wherever the knowledge lives.
2. Extract text - convert each source format into plain text, stripping layout artefacts while preserving structure worth keeping (headings, tables, page boundaries).
3. Chunk the text - split documents into smaller units sized for retrieval. See Chunking Strategies below for how this decision is actually made in production.
4. Generate embeddings - run each chunk through an embedding model to produce a numerical vector.
5. Store in a vector database - persist chunk text, embedding vector, and metadata (source document, page number, section heading) together. Metadata is what turns a vector match into a citable source.

Nothing in this phase happens in response to a user query. It runs as a batch or scheduled job, and letting that schedule go stale is one of the most common causes of the "stale answers" failure mode covered later in this article.

Phase 2: Query flow (online, per request)

Everything here happens synchronously, inside the request path, on every question a user asks.

6. User asks a question - the query arrives as free text.
7. Query understanding and framing - the system identifies intent, extracts key concepts, and expands ambiguous terms before anything gets embedded.
8. Query chunking - complex or multi-part questions may be broken into logical sub-questions ahead of retrieval. This is query decomposition, covered under Query Transformation below.
9. Convert the query to an embedding - using the same embedding model used during indexing. A mismatch here (different model, different version, different dimensionality) silently degrades every retrieval that follows.
10. Run similarity search - the vector database compares the query vector against stored chunk vectors (approximate nearest-neighbour search at scale, not exhaustive comparison) and surfaces the closest matches.
11. Retrieve the top-K chunks - keep the highest-scoring candidates. In production this list is rarely final; see Re-ranking below for why an initial top-50 typically gets cut down to a re-ranked top-5.
12. Build the prompt - assemble the retrieved chunks and the original question into a single prompt for the LLM, along with any system instructions and citation formatting.
13. Generate the final answer - send the assembled prompt to the LLM, which generates an answer grounded in the retrieved context rather than in training-time memory alone.

Architecture

Naive RAG (baseline)

Advanced RAG (production)

Agentic RAG (2026 dominant pattern)

The agent decides what to retrieve, evaluates whether the context is sufficient, and re-retrieves if needed. This handles multi-hop questions and ambiguous queries that naive RAG fails on.

Chunking Strategies

Chunking is where most RAG pipelines fail. The wrong chunk size means either too little context (model hallucinates) or too much noise (model ignores relevant information).

Chunking rules that hold across strategies

• Always use overlap (10-20% of chunk size) to avoid splitting mid-thought
• Preserve metadata (source file, page number, section heading) on every chunk
• Smaller chunks = more precise retrieval but less context per chunk
• Larger chunks = more context but noisier retrieval
• Test with your actual queries, not synthetic benchmarks

# Recursive chunking with LangChain
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " "]
)

chunks = splitter.split_documents(documents)

Retrieval Patterns

Hybrid search (recommended default)

Combines vector similarity with keyword matching. Catches both semantic meaning and exact terms that vector search misses.

# Azure AI Search hybrid query
results = search_client.search(
    search_text="rate limiting Azure OpenAI",  # keyword
    vector_queries=[VectorizableTextQuery(
        text="rate limiting Azure OpenAI",     # vector
        k_nearest_neighbors=10,
        fields="content_vector"
    )],
    query_type="semantic",                      # re-rank
    semantic_configuration_name="default",
    top=5
)

Re-ranking

Initial retrieval (vector or hybrid) returns candidates. A cross-encoder re-ranker scores each candidate against the query with full attention, producing much better ordering than cosine similarity alone.

Re-ranking adds 50-200ms latency but typically improves answer quality by 15-30%.

Query transformation

The user's raw query is often not the best search query. Transform it before retrieval:

Evaluation

Without automated evaluation, RAG quality degrades silently as document collections grow. Implement evaluation before production deployment.

RAGAS metrics (industry standard)

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

results = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision],
    llm=eval_llm
)

print(results)
# {'faithfulness': 0.92, 'answer_relevancy': 0.87, 'context_precision': 0.83}

Evaluation pipeline

Production Failure Modes

Technology Stack (2026)

Key Takeaways

• Hybrid search (vector + keyword) with semantic re-ranking is the production default. Pure vector search is not enough.
• 70% of RAG failures are retrieval or chunking problems, not LLM problems. Debug retrieval first.
• Implement RAGAS evaluation before production. Target: faithfulness >= 0.9, relevancy >= 0.85.
• Agentic RAG (agent decides what/when to retrieve) is the 2026 dominant pattern for complex queries.
• Chunk size is not one-size-fits-all. Test with your actual queries and documents, measure retrieval precision.