RAG (Retrieval-Augmented Generation)

What Is RAG?

RAG is a pattern where you retrieve relevant documents first, then feed them to the LLM as context so it can answer questions grounded in your data — not just its training data.

Without RAG:  User question → LLM (uses training data only) → May hallucinate
With RAG:     User question → Search your docs → LLM (uses your docs + training) → Grounded answer

This is how "chat with your docs" products work. It is the most practical way to give an LLM access to private, up-to-date, or domain-specific knowledge.

---

The Full RAG Pipeline

Step-by-Step

┌─────────────────── INDEXING (done once, updated as docs change) ───────────────────┐
│                                                                                     │
│  1. LOAD         2. CHUNK         3. EMBED           4. STORE                       │
│  PDF, MD, HTML → Split into      → Convert chunks    → Save vectors                │
│  code, docs      small pieces      to vectors          in vector DB                 │
│                                                                                     │
└─────────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────── QUERYING (every user question) ─────────────────────────────────┐
│                                                                                     │
│  5. EMBED QUERY   6. SEARCH        7. AUGMENT          8. GENERATE                 │
│  User question → Find similar    → Build prompt with  → LLM answers                │
│  → vector         chunks in DB     retrieved chunks     using context               │
│                                                                                     │
└─────────────────────────────────────────────────────────────────────────────────────┘

---

Step 1: Document Loading

Load your source documents into raw text:

---

Step 2: Chunking

LLMs have context limits, and embeddings work better on focused text. Split documents into chunks.

Chunking Strategies

Chunking Best Practices

Chunk size:    200-500 tokens per chunk (sweet spot)
Overlap:       50-100 tokens between consecutive chunks
               (prevents losing context at boundaries)
Metadata:      Store source file, page number, section title with each chunk

Example of overlap:

Chunk 1: "...transformers use self-attention to process tokens in parallel.
          This allows the model to capture long-range dependencies."

Chunk 2: "This allows the model to capture long-range dependencies.
          Unlike RNNs, transformers don't process tokens sequentially..."
          ↑ overlap — same sentence appears in both chunks

---

Step 3: Embedding

Convert each chunk into a vector (list of numbers) that captures its meaning. Similar chunks have similar vectors. (See 05 - Embeddings & Vector Search for the theory.)

python
from openai import OpenAI

client = OpenAI()

def embed_texts(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    return [item.embedding for item in response.data]

chunks = ["Transformers use self-attention...", "RAG retrieves documents..."]
vectors = embed_texts(chunks)
# Each vector: [0.023, -0.041, 0.078, ...] (1536 dimensions)

Embedding Models

---

Step 4: Vector Storage

Store embeddings in a vector database for fast similarity search:

Example: Storing in Chroma

python
import chromadb

client = chromadb.Client()
collection = client.create_collection("my_docs")

collection.add(
    ids=["chunk_1", "chunk_2", "chunk_3"],
    documents=["Transformers use...", "RAG retrieves...", "Embeddings capture..."],
    metadatas=[
        {"source": "transformers.md", "page": 1},
        {"source": "rag.md", "page": 3},
        {"source": "embeddings.md", "page": 1},
    ]
)

---

Steps 5-6: Query and Search

When a user asks a question:

python
# Embed the question using the SAME embedding model
query = "How does self-attention work?"

results = collection.query(
    query_texts=[query],
    n_results=5  # top K chunks
)

# Returns the 5 most semantically similar chunks

Retrieval Strategies

Reranking

Initial retrieval casts a wide net. A reranker (a separate model) rescores results for relevance:

python
# 1. Retrieve top 20 chunks (fast, broad)
candidates = vector_db.search(query, top_k=20)

# 2. Rerank to find the best 5 (slower, more precise)
reranked = reranker.rank(query, candidates, top_k=5)
# Reranker models: Cohere Rerank, cross-encoders, bge-reranker

---

Steps 7-8: Augment and Generate

Build a prompt that combines instructions, retrieved context, and the question:

python
def build_rag_prompt(question: str, chunks: list[str]) -> list[dict]:
    context = "\n\n---\n\n".join(chunks)
    return [
        {
            "role": "user",
            "content": f"""Answer the question based on the provided context.
If the context doesn't contain the answer, say "I don't have enough information."

Context:
{context}

Question: {question}"""
        }
    ]

messages = build_rag_prompt("How does self-attention work?", retrieved_chunks)
response = client.messages.create(
    model="claude-sonnet-4-6-20250514",
    max_tokens=1024,
    messages=messages
)

---

Common Pitfalls

---

Production Considerations

• Document update pipeline — watch for file changes, re-chunk and re-embed incrementally
• Metadata filtering — filter by date, source, category before vector search
• Caching — cache frequent queries and their results
• Evaluation — measure retrieval precision, answer accuracy, and faithfulness (see 18 - Evaluations & Testing AI)
• Hybrid search — combine keyword and semantic for best results
• Chunk citations — return source references so users can verify answers

---

Tools & Frameworks

---

Resources

• Anthropic RAG Guide
• Pinecone Learning Center
• LlamaIndex Documentation
• Chunking Strategies Comparison

---

Previous: 16 - AI Agents | Next: 18 - Evaluations & Testing AI