Retrieval-Augmented Generation at Scale: Vector Databases & Semantic Search

Three months after our RAG pipeline went from "cool demo" to "thing 400 people use every day," we got a Slack message that made our stomachs drop: "the bot keeps citing the wrong policy version." We pulled up the trace, and the retrieval step had returned three chunks from a document that had been superseded two quarters earlier. The embeddings looked fine. The similarity scores looked fine. The answer, confidently generated from stale context, was wrong.

That was the moment retrieval quality stopped being a "we'll get to it" problem and became the problem. Generation is the easy 20%. Once your corpus crosses a few thousand documents and your query volume crosses a few thousand a day, everything that felt like an implementation detail in the first pipeline — chunk size, index type, whether you bothered to keyword-match anything — turns into the thing that decides whether the system is trustworthy.

This post assumes you've already built a RAG pipeline and know what embeddings and retrieval are for. We're not re-deriving RAG from first principles here. We're talking about what breaks when you take it from a weekend project to something serving real traffic against a real corpus, and what we changed to fix it.

Where Retrieval Quality Actually Degrades

In a demo, you embed fifty documents, ask five questions you already know the answers to, and everything works because the corpus is small enough that cosine similarity can't get confused. At scale, three things erode retrieval quality independently, and they compound:

1. Corpus growth dilutes the signal. With 50 documents, the top-k nearest neighbors for a query are almost always relevant just by process of elimination. With 50,000 documents, there are dozens of near-miss chunks that are topically adjacent but not actually responsive to the question, and pure vector similarity has no mechanism to tell them apart from the right answer.
2. Chunking decisions you made once now apply to content you never anticipated. A fixed 512-token chunk size that worked fine for prose docs slices tables and code blocks into meaningless fragments the moment someone uploads a technical spec.
3. The embedding model itself is not static. Providers version their embedding models, and vectors from model v1 are not comparable to vectors from model v2. If you don't track which model produced which vector, you can end up doing similarity search across two incompatible vector spaces without any error being thrown — just quietly wrong scores.

Each of these is fixable, but only if you're measuring retrieval quality as a first-class metric instead of eyeballing outputs. We'll get to evaluation, but first, the infrastructure decisions that determine what's even possible.

Vector Embeddings at Scale

The embedding model choice is the single decision with the most downstream leverage, and it's usually made once, early, and never revisited — which is itself a risk.

Dimensionality is a latency and storage tax, not a quality knob past a certain point. A 1536-dimension embedding (OpenAI's text-embedding-3-small default) costs more to store and search than a 384-dimension one (many sentence-transformers models), but higher dimensionality doesn't linearly buy you better retrieval. Past roughly 768–1024 dimensions, we've seen diminishing returns on retrieval accuracy for typical enterprise document corpora, while index size and query latency keep climbing linearly. If you're running at a few million vectors, the difference between 384 and 1536 dimensions is the difference between an index that fits comfortably in memory and one that doesn't. Most embedding providers now let you truncate (Matryoshka-style embeddings) to a lower dimensionality with minimal quality loss — worth testing before you commit to the largest available model by default.

Model choice is a trade-off between retrieval quality, latency, and vendor lock-in, not just an accuracy leaderboard number. A few things we weigh:

• Domain fit matters more than benchmark rank. A general-purpose embedding model trained mostly on web text will underperform a domain-tuned one on dense technical or legal corpora, even if the general model wins on MTEB leaderboards. If your corpus is heavy on code, API references, or regulatory language, test on your actual documents, not a public benchmark.
• Batch throughput for ingestion is a real operational constraint. If you're re-embedding a large corpus after a model upgrade, the difference between an API that batches 100 texts per call and one that's strictly per-request changes a multi-day re-embedding job into a multi-hour one.
• Self-hosted embedding models remove per-token cost but add GPU operational overhead. For very high query volume, we've moved query-time embedding (not ingestion-time) to a self-hosted model behind a lightweight inference server, because at scale the marginal cost of an API call per query search adds up faster than the cost of keeping a small GPU warm.

Embedding drift is the failure mode nobody plans for. When a provider ships a new embedding model version, two things can happen silently: either your provider auto-migrates and vectors you embedded last month are no longer in the same space as vectors embedded today, or they keep the old model available and you have to explicitly decide to migrate. Either way, the fix is the same — version-stamp every vector with the embedding model and version that produced it, and never mix vector spaces in a single similarity search. When we upgraded embedding models, we ran a full re-embed of the corpus as a batch job and cut over the index atomically, rather than trying to embed new documents with the new model while old documents sat in the old vector space. Partial migrations are how you get retrieval that "sometimes works" for no discoverable reason.

Choosing a Vector Database

By the time you're past a demo, "which vector database" stops being a feature-matrix exercise and becomes a question about who operates it and what your query patterns actually look like.

Pinecone is the path of least resistance if you want managed infrastructure and don't want to think about index tuning, sharding, or replication. It's genuinely good at "give me approximate nearest neighbors fast, and don't wake me up at 3am." The trade-off is cost at volume — Pinecone bills per pod/replica and per read/write unit depending on plan, and if your query volume is bursty, you're either over-provisioned most of the time or throttled during peaks. It's also a dedicated system: you're maintaining a vector store that lives entirely separately from your operational database, with its own auth, its own backup story, and its own place in your incident runbooks.

Weaviate sits in a similar managed-or-self-hosted middle ground, with the advantage of built-in hybrid search (vector + BM25) and a GraphQL-ish query interface. We've found it a reasonable choice when you want hybrid search out of the box without stitching together two systems yourself, but self-hosting it means owning cluster operations — sharding, replication factor, and the JVM-adjacent tuning that comes with any distributed system holding your source of truth.

Milvus is the option we reach for when the corpus is large enough (tens of millions of vectors) that index build time and horizontal scaling become the dominant concern. It has the most mature story for sharding a single collection across nodes and swapping index types (HNSW, IVF, DiskANN) per use case. The cost is genuine operational complexity — running Milvus well means running etcd, object storage, and a handful of coordinator/worker processes, which is a real team commitment, not a docker-compose afternoon.

Postgres + pgvector is the one we default to unless we have a specific reason not to, and it surprises people how far it goes. If you already run Postgres — and on the JVM stack, you almost certainly do — pgvector means your vector index lives next to your relational data, in the same transaction boundary, with the same backup and access-control story as everything else. You get JOINs between vector similarity and relational filters (tenant ID, document status, permissions) for free, which is something you'd otherwise bolt on as metadata filtering in a dedicated vector DB's query API. The honest limitation: pgvector's HNSW index build and query performance falls off past roughly 10–20 million vectors on a single instance unless you're deliberate about index parameters (m, ef_construction) and give it real memory. For most internal knowledge-base or product-documentation RAG systems — low millions of chunks — that ceiling never gets hit, and the operational simplicity of "one fewer distributed system to run" wins outright.

Our rule of thumb: start with pgvector unless you already know your corpus will exceed tens of millions of vectors or your query latency budget is sub-20ms at high QPS, in which case a dedicated vector database earns its operational cost.

Retrieval Strategy: Semantic, Hybrid, and Reranking

Pure vector search is excellent at matching intent and paraphrase — "how do I cancel my plan" retrieves a chunk titled "Subscription Termination" even with zero shared vocabulary. It's also bad at exact-match precision. Ask about a specific error code, a product SKU, a config flag name, or an acronym, and pure semantic search will often surface generically related content instead of the chunk that contains the literal string, because the embedding space compresses exact tokens into a general topic vector.

This is exactly the gap hybrid search closes. Hybrid search runs a sparse keyword method (BM25 or Postgres full-text search) alongside dense vector search, then merges the two ranked lists — typically with reciprocal rank fusion or a weighted score combination. The keyword side catches exact terms, part numbers, error codes, and rare vocabulary that get diluted in embedding space; the vector side catches paraphrase and conceptual matches that keyword search would miss entirely.

Hybrid beats pure vector search specifically when the query contains a term that's rare, technical, or an identifier — and it adds no benefit, sometimes even slight noise, on purely conversational natural-language queries where nothing in the query is an exact-match anchor. The failure mode we watch for with hybrid search is over-weighting the keyword score on queries where the wording genuinely differs from the source document — hybrid is a blend, not a strict improvement, and the fusion weights need to be tuned against your actual query distribution, not assumed.

Reranking is the step most teams skip too long, and it's usually the highest-leverage one they eventually add. The idea: cheap retrieval (vector + keyword) pulls a broad candidate set — say top 50 — cast wide enough to have high recall, and a cross-encoder reranker, which sees the actual query and candidate text together rather than comparing precomputed embeddings, re-scores that candidate set for genuine relevance. Cross-encoders are more expensive per comparison than a vector lookup, which is exactly why you only run them over 50 candidates instead of the whole corpus. In our pipeline, adding a reranking stage after hybrid retrieval was the single change that most reduced "technically related but not actually the answer" chunks reaching the LLM.

The pure vector path finds the conceptually related troubleshooting page because "timeout" and "fix" carry semantic weight, but the exact error code ERR_4092 gets averaged away in the embedding. The hybrid path's keyword arm anchors on the literal code and surfaces the runbook section that actually resolves the specific error — a category of miss that shows up constantly once your corpus has enough near-duplicate "troubleshooting" content that semantic similarity alone can't discriminate between them.

Chunking Strategy: The Decision That Compounds

Chunk size and overlap feel like a tuning knob you set once and forget. In practice, chunking is where most retrieval failures actually originate, because a bad chunk boundary poisons everything downstream — no reranker or hybrid fusion can recover information that was never captured coherently in the chunk in the first place.

Size and overlap trade-offs: smaller chunks (150–300 tokens) give you more precise retrieval — the embedding represents a narrower, more specific idea — but risk losing context a reader would need to actually understand the fragment. Larger chunks (500–800 tokens) preserve more context but dilute the embedding, since a chunk covering three different subtopics produces a vector that's a blurry average of all three, matching none of them well. Overlap (typically 10–20% of chunk size) exists to avoid severing a sentence or idea exactly at a chunk boundary, at the cost of redundant storage and, at query time, occasionally retrieving two overlapping chunks that say almost the same thing.

There's no universal right answer here — it depends on document structure — which is why metadata-aware, structure-preserving chunking matters more than the raw size number. Splitting purely on token count, blind to document structure, routinely cuts a chunk in half mid-table, or separates a code block from the paragraph explaining it, or drops a chunk from deep in a document with no idea which section or heading it came from. The fix that mattered most for us wasn't finding the "correct" chunk size — it was chunking along natural document boundaries (headings, list items, table boundaries, code fences) and attaching the surrounding heading path as metadata on every chunk. A chunk that's just "...and in that case, set the timeout to 30 seconds" is nearly useless on its own; the same chunk tagged with section: "Configuring Retry Behavior > Timeout Settings" gives both the retriever and the LLM something to anchor on, and lets you inject that heading context into the prompt without re-embedding the whole document.

Practically, this means our chunking step is closer to a small parser than a token-count loop: split on markdown/HTML structure first, keep tables and code blocks intact as atomic chunks even if they exceed the target size, and only fall back to token-count splitting within a long unstructured paragraph. It's more ingestion code to write and maintain, and it is worth every line of it.

Measuring Retrieval Quality, Not Guessing at It

"It looks fine when I try it" is not an evaluation strategy, and it's the reason retrieval quality problems get discovered by users instead of by monitoring. Once you have real query logs, you can build an actual evaluation set: take a sample of real user queries, hand-label (or LLM-assist label, then spot-check) which chunks in the corpus are genuinely relevant to each, and measure retrieval against that ground truth on every change to chunking, embedding model, or ranking logic.

The three metrics that matter, applied specifically to RAG retrieval rather than generic search:

• Precision@k — of the top-k chunks you retrieved, what fraction are actually relevant. This is the metric that tells you whether you're feeding the LLM noise. Low precision means the model has to work through irrelevant context to find the answer, which increases hallucination risk even when the right chunk is technically present.
• Recall@k — of all the relevant chunks that exist in the corpus, what fraction did you retrieve in your top-k. This is the metric that tells you whether the answer was ever reachable at all. If recall is low, no amount of prompt engineering or reranking fixes it — the answer simply wasn't given to the model.
• NDCG (Normalized Discounted Cumulative Gain) — unlike precision/recall, which treat every relevant result in the top-k as equally good, NDCG rewards putting the most relevant chunk first rather than fourth. This matters specifically for RAG because LLMs weight earlier context more heavily in practice, and because you're paying token cost for every chunk you include — ranking quality, not just retrieval membership, decides both answer quality and cost.

Here's a simplified version of what our evaluation runs actually look like, on a 40-query sample set built from real support queries, comparing pure vector search against hybrid + rerank:

The pattern that mattered most to us: hybrid search moved recall — it found relevant chunks pure vector search never surfaced, largely on queries with exact-match technical terms. Reranking barely moved recall at all (it can't find chunks that weren't in the candidate set), but it moved precision and NDCG substantially, because its job is purely about ordering an already-retrieved set correctly. That distinction should directly inform where you invest: if your problem is "the answer isn't in the corpus results at all," hybrid search and better chunking are the fix; if your problem is "the right chunk is buried at position 8," reranking is the fix. Measuring both separately, rather than one blended "does it look right" judgment, is what tells you which lever to pull.

Run this evaluation on every meaningful pipeline change — new embedding model, new chunk size, new reranker — the same way you'd run a test suite before a deploy. We treat a drop in any of these three metrics as a blocking regression, not a "we'll monitor it."

Scaling Operationally

Multi-index / sharding strategy becomes relevant once a single index can't be rebuilt or queried fast enough as one unit. The pattern we use is domain-based sharding: separate vector indexes per document category or tenant, rather than one enormous index with a metadata filter bolted on. This has two benefits — index rebuilds after a chunking or embedding change can happen incrementally per shard instead of one long lock on the whole corpus, and query-time filtering by category becomes an index selection decision instead of a post-filter on a larger result set, which is both faster and avoids the classic ANN pitfall where filtering after retrieval can return fewer than k results because the filter excluded most of the top matches.

Caching is underused in RAG systems because people assume every query is unique. In practice, query traffic is heavily skewed — a small set of common questions accounts for a large share of volume in most internal or product-support RAG deployments. We cache at two layers: the embedding of a normalized query string (so re-asking a semantically identical question doesn't recompute the embedding), and the full retrieval result for exact or near-duplicate queries within a short TTL. This alone cut our average retrieval latency meaningfully during peak hours, and it's a pure win — retrieval results for a cacheable query don't change between requests seconds apart, and the LLM generation itself is intentionally left uncached (or cached separately) since you may want to vary temperature or reasoning depth independent of retrieval.

Corpus growth needs a re-indexing plan from day one, not as an afterthought. Adding new documents incrementally to an HNSW-style index works fine for a while, but periodic full re-indexing — rebuilding rather than incrementally appending — meaningfully improves index quality and query latency, because incremental inserts alone tend to produce a less optimal graph structure over time. We schedule full re-index as a background batch job on a fixed cadence rather than waiting for query latency complaints to force it.

Cost Optimization

RAG cost is not just "LLM tokens." It hides in three places, and only one of them shows up on the obvious dashboard:

• Retrieval cost: embedding API calls at query time, vector DB compute/storage, and reranker inference all cost money per query, independent of the LLM call. At meaningful query volume, query-time embedding calls to a hosted API are often a larger recurring line item than people expect, which is a strong argument for self-hosting the embedding model for query-time inference specifically, even if ingestion-time embedding stays on a managed API.
• Ingestion cost: re-embedding a large corpus after any pipeline change (new chunking strategy, new embedding model) is a bulk cost event. Batch these calls — most embedding APIs charge close to the same per-token rate for batched versus single requests, but batching cuts wall-clock time and request overhead dramatically, which matters when a corpus refresh is blocking a deploy.
• Generation cost, driven by context size: this is the one people actually track, but they often miss that it's directly a function of retrieval decisions — more retrieved chunks, larger chunk size, and looser reranking-driven filtering all directly inflate the prompt token count, and therefore the per-query LLM cost, before the model generates a single output token.

The lever with the best cost-to-quality ratio in our experience: spend more on reranking (cheap relative to LLM generation) to justify retrieving fewer, better-ordered chunks, rather than compensating for weak retrieval by just stuffing more chunks into the prompt and hoping the model sorts it out.

Fitting Retrieval Into the Context Window

Once reranking gives you an ordered, high-precision set of chunks, the naive move — concatenate the top 10 and paste them into the prompt — still wastes context and money. A few things we do instead:

• Truncate by relevance score, not by a fixed chunk count. A fixed "always include top 8" throws in low-relevance chunks on queries where only 2 chunks actually clear a reasonable relevance bar, and starves genuinely complex queries that need 6 good ones. We set a relevance-score cutoff post-rerank and take however many chunks clear it, up to a cap.
• Deduplicate near-identical chunks from overlap or multiple document versions before they hit the prompt — this is a direct consequence of chunk overlap and multi-version corpora, and it's pure wasted context if left unhandled.
• Attach the metadata heading path, not the whole document, as context, so the model knows a chunk is from "Billing > Refund Policy > Enterprise Plans" without needing three paragraphs of surrounding text to establish that same context.
• Summarize or compress lower-ranked-but-included chunks more aggressively than the top result — an asymmetric compression strategy where your best chunk gets full text and marginal ones get a shorter extractive summary keeps total token count bounded without a hard cliff between "included in full" and "excluded entirely."

Here's a simplified version of the hybrid scoring service we run in front of pgvector, combining vector similarity with a full-text search score before handing candidates to the reranker:

@Service
public class HybridRetrievalService {

    private final JdbcTemplate jdbcTemplate;
    private final EmbeddingClient embeddingClient;
    private final CrossEncoderClient rerankerClient;

    private static final double VECTOR_WEIGHT = 0.6;
    private static final double KEYWORD_WEIGHT = 0.4;

    public HybridRetrievalService(JdbcTemplate jdbcTemplate,
                                   EmbeddingClient embeddingClient,
                                   CrossEncoderClient rerankerClient) {
        this.jdbcTemplate = jdbcTemplate;
        this.embeddingClient = embeddingClient;
        this.rerankerClient = rerankerClient;
    }

    public List<RankedChunk> retrieve(String query, int candidatePoolSize, int finalTopK) {
        float[] queryEmbedding = embeddingClient.embed(query);

        // Hybrid scoring: cosine similarity (pgvector) combined with
        // Postgres full-text rank, fused with fixed weights tuned
        // against our evaluation set.
        String sql = """
            SELECT id, content, section_path,
                   (1 - (embedding <=> ?::vector)) AS vector_score,
                   ts_rank_cd(content_tsv, plainto_tsquery('english', ?)) AS keyword_score
            FROM document_chunks
            ORDER BY (? * (1 - (embedding <=> ?::vector))
                    + ? * ts_rank_cd(content_tsv, plainto_tsquery('english', ?))) DESC
            LIMIT ?
            """;

        List<CandidateChunk> candidates = jdbcTemplate.query(sql,
            (rs, rowNum) -> new CandidateChunk(
                rs.getString("id"),
                rs.getString("content"),
                rs.getString("section_path"),
                rs.getDouble("vector_score"),
                rs.getDouble("keyword_score")
            ),
            toVectorLiteral(queryEmbedding), query,
            VECTOR_WEIGHT, toVectorLiteral(queryEmbedding),
            KEYWORD_WEIGHT, query,
            candidatePoolSize
        );

        // Cross-encoder reranking over the candidate pool, not the full corpus.
        return rerankerClient.rerank(query, candidates, finalTopK);
    }

    private String toVectorLiteral(float[] embedding) {
        StringBuilder sb = new StringBuilder("[");
        for (int i = 0; i < embedding.length; i++) {
            if (i > 0) sb.append(",");
            sb.append(embedding[i]);
        }
        return sb.append("]").toString();
    }
}

The candidate pool from the hybrid query is deliberately wider than what we actually send to the LLM — retrieval optimizes for recall, the reranker optimizes for precision, and conflating those two jobs into a single scoring pass is a common reason teams underinvest in one or the other.

Common Mistakes

Chunking without preserving structural context. Splitting purely by token count and discarding heading/section information produces chunks that are individually incoherent, and no amount of clever retrieval logic fixes information that was never captured properly at ingestion time.

Never measuring retrieval quality until users complain. Precision, recall, and NDCG against a real evaluation set should be a standing metric, checked on every pipeline change, not a one-time audit triggered by a support ticket about a wrong answer.

Treating embedding model upgrades as a drop-in swap. Vectors from different model versions are not comparable. An upgrade without a full re-embed and atomic index cutover produces silently degraded retrieval with no error message anywhere in the stack.

Compensating for weak retrieval by stuffing more chunks into the prompt. More context is not more signal — past a point it's more noise the model has to sort through, plus more token cost, when the actual fix is better ranking or better chunking, not a bigger k.

Ignoring query cost concentration. Assuming every query is unique and skipping caching wastes both latency and money on a workload where a relatively small set of common questions typically accounts for a disproportionate share of total query volume.

Final Takeaway

The gap between a RAG demo and a RAG system that survives contact with real users and a growing corpus isn't the LLM — it's everything upstream of it. Chunking decisions made in week one determine what's retrievable in month six. Pure vector search is necessary but not sufficient the moment your corpus has enough near-duplicate or technically dense content that semantic similarity alone can't discriminate the right answer from an adjacent one. And none of this is verifiable by reading outputs and nodding — you need precision, recall, and NDCG on a real evaluation set, tracked the same way you'd track any other production metric, or you're finding out about regressions from your users instead of your dashboards.

Next in this AI engineering series: Prompt Engineering at Scale, where we'll cover how to actually structure what you send the model once retrieval has done its job.

Subscribe to get system design and backend engineering posts in your inbox every two weeks.