If you want to understand why your Retrieval-Augmented Generation (RAG) system is hallucinating, stop looking at your prompt. A perfect prompt with the wrong data yields a confident hallucination. An average prompt with the correct data yields a useful answer.

The distance between a database returning a conceptually similar chunk and returning a factually useful chunk is the hardest engineering problem in modern AI. We call it the Retrieval Gap.

High semantic similarity does not guarantee factual usefulness. To bridge the gap, you need a hybrid pipeline that combines vector search, BM25 keyword retrieval, reranking, and metadata filtering.

The RAG Maturity Curve

Understanding where your architecture currently sits is the only way to anticipate the next bottleneck.

• Stage 1: The Toy. Jupyter notebook + ChromaDB + 1 PDF. (~10% Retrieval Recall)

• Stage 2: Persistent Infrastructure. Python scripts + PostgreSQL (pgvector) + Docker.

• Stage 3: Operational Reality. FastAPI + PgBouncer + HNSW Indexing.

• Stage 4: Hybrid Retrieval. Vector search + Keyword search (BM25) + Reranking APIs. (~90%+ Retrieval Recall)

• Stage 5: The Semantic Microservice. Autonomous ingestion, Reciprocal Rank Fusion, Semantic Caching, and continuous Recall evaluation.

Production RAG systems evolve through multiple infrastructure stages. Retrieval quality improves not through prompt engineering alone, but through indexing, hybrid retrieval, and AI observability.

The Minimal Deployment Topology

Before diving into the failures, you need to understand where the pieces actually live. A Stage 4/5 production architecture completely separates the orchestration wrapper from the data persistence layer.

The Stack:

• Orchestration: FastAPI + LangGraph (Stateless async workers).

• Connection Pooler: PgBouncer (Protects the DB from connection exhaustion).

• System of Record: PostgreSQL + pgvector (Stores metadata, BM25 text, and HNSW embeddings).

• Semantic Cache: Redis (Bypasses the DB for repeated queries).

• External Compute: OpenAI (Embeddings) + Cohere (Cross-Encoder Reranking).

The PDF Lie and Ingestion Contamination

Most RAG failures happen before the embedding model ever sees the text. Your PDF loader is lying to you. It does not “see” a cleanly formatted manual; it sees a fragmented mess of text, hidden whitespace, and floating footers.

If a document has “Confidential – Internal Use Only” on every page, a naive character splitter will attach that string to every single chunk. Embedding models amplify repeated noise. If your ingestion layer is dirty, retrieval quality collapses.

import re
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Remove repeating headers BEFORE chunking to prevent semantic poisoning
cleaned_text = re.sub(r"Confidential - Internal Use Only", "", raw_text)

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200, 
    separators=["\n\n", "\n", " ", ""]
)

Postmortem: The Night the Index Rebuild Killed Production

Everyone talks about the magic of vector search. Nobody talks about the database locks.

In pgvector, the distance between vectors is often calculated using Cosine Distance. The formula for the distance \(d\) between vectors \(\mathbf{A}\) and \(\mathbf{B}\) is:

$$d(\mathbf{A}, \mathbf{B}) = 1 - \frac{\mathbf{A} \cdot \mathbf{B}}{|\mathbf{A}| |\mathbf{B}|}$$

Six months into a deployment, we hit 5 million vectors. A nightly cron job triggered a mass re-ingestion, which forced the HNSW (Hierarchical Navigable Small World) graph to recalculate. HNSW index builds are incredibly RAM and CPU intensive. The rebuild starved the database of resources, locking out our read traffic.

The Unresolved Tension: HNSW vs. IVFFlat

We eventually traded a 3% precision drop (switching to IVFFlat) for database stability. Real engineering is choosing which tradeoff you can survive.

The Latency Budget Breakdown

Once you move to a Stage 4 architecture (Hybrid Search), you are stacking API calls. Here is what a realistic P95 latency budget looks like:

The Tenant Leakage Problem

In enterprise RAG, searching the “whole database” is a security incident. A vector search returning another customer’s document is a massive security risk.

You must implement strict metadata filtering before the vector distance is calculated, or rely on Postgres Row-Level Security (RLS).

# The Metadata Filter is your security boundary
retriever = db.as_retriever(
    search_kwargs={
        "k": 5,
        "filter": {"tenant_id": current_user.tenant_id} # Mandatory security boundary
    }
)

How to Measure Retrieval Quality (Evaluation Science)

You cannot fix what you do not measure. If you judge your RAG system by reading the LLM’s final output, you are flying blind. You must isolate retrieval failure from generation failure.

We measure Recall@10 (Did the correct, ground-truth chunk appear in the top 10 results?).

The Benchmark Results:

• Pure Vector Search: Recall@10 = 74%.

• Hybrid (Vector + BM25) + Reranker: Recall@10 = 96%.

If your Recall@10 is 74%, that means 1 out of 4 times, the LLM is physically incapable of answering the user’s question because the vector database didn’t hand it the right text.

Scaling with Semantic Caching

As you scale, you will notice users ask the same questions 80% of the time. Running the embedding model and Postgres query for “How do I reset my password?” is a waste of compute.

By putting a caching layer (like Redis) in front of your pipeline, you can store previous responses keyed to the vector embedding of the question. If a new query has a 0.99 cosine similarity to a cached query, you bypass the entire RAG pipeline and return the result in 15ms.

Models will change. Abstractions like LangChain will evolve. But the fundamental physics of database connections, latency budgets, and index rebuilds remain the same. The plumbing is what matters.

For the full implementation guide, architecture breakdown, and production deployment walkthrough, read the original article here:

How to Build a RAG System with PGVector and LangChain