RAG Architecture, Vector DBs & HNSW Indexing

Eliminate LLM hallucinations by grounding generation in external knowledge bases using vector similarity search.

1. The Retrieval-Augmented Generation (RAG) Triad

RAG decomposes knowledge-intensive QA into three sequential phases:

Ingestion & Chunking: Document parsing, recursive character chunking with overlap \(\mathcal{C} = \{c_1, c_2, \dots, c_m\}\), and dense vector embedding generation.
Approximate Nearest Neighbor (ANN) Retrieval: Query vector \(\mathbf{q} = \text{Embed}(\text{prompt})\) is queried against an **HNSW (Hierarchical Navigable Small World)** vector index.
Grounded Generation: Retrieved top-k contexts are injected into the LLM system prompt context window.

2. Two-Stage Retrieval with Cross-Encoder Re-Rankers

Bi-encoders (embedding search) are fast but lose fine-grained token-level cross-attention. High-performance RAG pipelines use bi-encoders to retrieve Top-100 candidates, then pass them through a Cross-Encoder (e.g. Cohere Re-ranker or BGE-Reranker) to select the Top-5 most relevant chunks.

🎯 Module Mastery Certification Quiz

+100 XP

Why is a Cross-Encoder used as a second-stage re-ranker in advanced RAG systems?

Because cross-encoders compute all-to-all cross-attention between query and document tokens, providing superior relevance scoring compared to independent bi-encoder cosine similarity.

Because cross-encoders compress PDF documents to 10% of their size.

Because bi-encoders cannot run on NVIDIA GPUs.

Because cross-encoders eliminate the need for an LLM.