Design an enterprise RAG service with 10 million chunks
A hypothetical multi-tenant knowledge assistant. Design permission-aware retrieval, versioned ingestion, grounded answers and safe embedding migrations. Every capacity number below is an interview assumption or a calculation, not a production benchmark.
Interview assumptions
- •500 organizations, 10 million indexed chunks and 2 million queries/day.
- •Peak traffic is 8 times average; embedding dimension is 1,536 and storage uses float32.
- •A source edit should become searchable within a proposed 5-minute freshness target. Revocation needs a separately defined, tighter serving policy.
Sources & further reading
Lost in the Middle ↗Self-RAG ↗Corrective Retrieval Augmented Generation ↗Original RAG paper: retrieval and generation components ↗vLLM prefix caching: distinguish KV reuse from answer caching ↗Functional Requirements
- •Ingest versioned documents, updates and deletions idempotently.
- •Retrieve within tenant and permission scope using lexical and dense indexes.
- •Rerank candidates and answer with source versions and supporting citations.
- •Clarify ambiguous requests and abstain when evidence is insufficient.
- •Rebuild or migrate embeddings with shadow comparison and rollback.
NFR / SLO
- •Proposed read availability target: 99.9%; define which failures count.
- •Proposed p95 first-token target: 2 seconds for a benchmarked request class; measure the full path.
- •Unauthorized evidence must never enter a generator prompt or cross-tenant answer cache.
- •Bound queue age, context size, retry count and per-tenant query work.
End-to-End Architecture
Scale & QPS Calculations
| Metric | Formula | Result |
|---|---|---|
| Average query rate | 2,000,000 / 86,400 | 23.15 requests/s |
| Peak query rate | 23.1481 × 8 | 185.19 requests/s |
| Raw vectors | 10,000,000 × 1,536 × 4 bytes | 61.44 GB ≈ 57.22 GiB |
| Three vector copies | 61.44 GB × 3 | 184.32 GB; excludes index, text, metadata and backups |
Global Scale
- •Keep regulated source content and retrieval in its allowed region.
- •Route by tenant home region and use an explicit residency-aware disaster recovery policy.
- •Replicate authorized metadata and indexes under stated consistency/freshness limits.
Latency Budget
- •Illustrative typical-path allocation: edge/auth 80 ms, query processing 70 ms, parallel retrieval 150 ms, rerank 200 ms, evidence checks 100 ms, first model token 900 ms: total 1.5 seconds before extra queueing.
- •These stage allocations are planning assumptions, not p95 values that can simply be added.
- •Measure the joint end-to-end distribution and reserve slack for queueing and cold paths.
SQL / Data Model
CREATE TABLE document_versions ( tenant_id TEXT NOT NULL, document_id TEXT NOT NULL, version BIGINT NOT NULL, content_hash TEXT NOT NULL, acl_version BIGINT NOT NULL, deleted BOOLEAN NOT NULL, PRIMARY KEY (tenant_id, document_id, version) ); CREATE TABLE index_generations ( tenant_id TEXT NOT NULL, generation TEXT NOT NULL, embedding_model TEXT NOT NULL, parser_version TEXT NOT NULL, state TEXT NOT NULL, PRIMARY KEY (tenant_id, generation) );
- •A chunk records its document version and index generation; never infer freshness from vector similarity.
- •Keep source text in versioned object storage and enforce current access at query time.
- •Publish a complete generation atomically; use tombstones during physical deletion.
Caching Strategy
- •Separate document, retrieval, prefix and answer caches.
- •Include tenant and permission policy/version in private cache scope; revalidate sensitive evidence.
- •Invalidate affected generations after mutations and propagate deletions to old and new indexes.
Database Trade-offs
- •Lexical search handles exact identifiers; ANN supports semantic candidates. Benchmark their combination under restrictive filters.
- •A shared index improves packing but needs strong isolation; per-tenant indexes simplify boundaries at a utilization cost.
- •The relational metadata store governs versions and visibility; the vector store is a derived retrieval index.
Principal Trade-offs
| Decision | Chosen | Alternative | Why | Downside | When the alternative wins |
|---|---|---|---|---|---|
| Retrieval | Hybrid candidates with bounded reranking | Dense-only retrieval | Mixed exact identifiers and paraphrases | More indexing and serving work | Dense-only already meets measured quality and latency requirements |
| Migration | Versioned dual-write and alias switch | In-place re-embedding | Rollback and separation of vector spaces | Temporary storage and pipeline overhead | A small corpus can tolerate a complete maintenance window |
Reliability
- •Replay a durable mutation log with document/version idempotency.
- •Honor revocations in the authoritative read policy before index compaction.
- •Degrade from reranked to simpler authorized retrieval only if evaluated quality meets policy; otherwise explain unavailability.
Security
- •Authorize before context construction; prevent cross-tenant cache reuse.
- •Treat retrieved instructions as untrusted content and keep tool permission decisions outside the model.
- •Minimize retained prompts and private document text in logs.
Observability
- •Evidence hit rate@k on answerable questions, claim support and citation completeness.
- •Deletion propagation delay, document freshness, filtered latency and tenant isolation checks.
- •Version every prompt, embedding, index generation, reranker and generator used in a response.
Cost
- •Estimate tokens, rerank candidates and embedding updates per useful answer, including retries.
- •Measure the quality impact of context pruning and candidate budgets before buying more GPU capacity.
- •Estimate the one-off embedding backfill cost separately from the ongoing query-serving budget.
Interview Q&A
1. Retrieval metrics improved but answers got worse. What would you inspect?
Inspect the final context rather than the candidate list alone. More candidates may increase duplication, introduce conflicting versions or push supporting evidence out of the token budget. Compare reranker order, context placement and claim-level support on the same questions. Run an oracle-context experiment and keep the model and prompt fixed while changing retrieval. Report answer quality and latency together; a higher retrieval score does not establish end-to-end improvement.
2. When would you choose hybrid retrieval over a vector-only index?
Use the query distribution to decide. Exact product codes, names and rare terms often benefit from lexical matching, while paraphrases can benefit from embeddings. Fuse ranked lists and then evaluate reranking within a bounded candidate budget. Hybrid retrieval adds indexing and serving work; if dense retrieval already meets quality and latency targets, the extra path may not earn its cost. Inspect failures by query type rather than only the average.
3. How do you choose between RAG and a long-context prompt?
Compare the same tasks under a fixed cost and latency budget. A small, stable document set may fit naturally in context. A large changing corpus needs selection, freshness controls and authorization regardless of context length. Long-context capacity does not prove that all evidence is used reliably. Evaluate passage position, contradictory versions, citation completeness and abstention. Consider retrieval plus a larger context as a third option rather than treating the choices as mutually exclusive.
4. What should a RAG assistant do when its evidence is insufficient?
Use a policy calibrated on labeled answerable and unanswerable questions. It can ask for the missing product version, retrieve once more with a bounded query rewrite, or decline to make an unsupported claim. Measure unsupported-answer rate alongside coverage so that refusing everything cannot win. Treat retrieval scores and model confidence as signals requiring validation, not proof. A citation is useful only when the cited passage supports the actual claim.
5. Why is filtering unauthorized results after generation unsafe?
The generator has already received private text. Its answer, logs, tool calls or model-provider request may expose that content even if you remove the final citation. Filter before context construction, enforce tenant boundaries at the serving layer and verify authorization again at sensitive read points. Include cross-tenant retrieval and stale-permission scenarios in tests. Treat retrieved instructions as untrusted data rather than allowing them to override system policy.
6. How would you delete a document from a live RAG system?
Write a versioned deletion or revocation to the authoritative metadata first. Query serving should exclude that document immediately under the stated consistency policy. Propagate the tombstone through text, lexical, vector, cache and replica stores, and track acknowledgments. Existing conversations must revalidate evidence when required by the product policy. Physical index compaction can be asynchronous; semantic deletion cannot depend solely on a future rebuild. Audit the propagation delay.
7. How do you migrate from one embedding model to another safely?
Keep old and new vector spaces separate. Backfill a new generation with its own preprocessing and matching query encoder. Dual-write mutations and tombstones while the backfill runs, then replay a stable query set and shadow production traffic. Compare quality, filtered retrieval, latency and storage. Switch an alias atomically and keep a rollback window. Do not concatenate scores from different spaces as if they share a scale.
8. What belongs in the cache key for an enterprise RAG answer?
Include tenant identity, a suitable authorization scope or policy version, normalized query, document/index generation, prompt version and model version where they affect validity. Avoid caching private answers under a global query-only key. A cache hit must still meet the current access and freshness policy. Separate retrieval, prefix and final-answer caches because they have different invalidation rules. Bound retention and remove sensitive payloads from general operational logs.
Principal Engineer Checklist
- •Separate missing knowledge from failed retrieval and failed generation.
- •Explain deletion, authorization and cache behavior during an in-flight request.
- •State the index migration cutover and rollback policy.
- •Design a quality/latency experiment with answerable, unanswerable and adversarial examples.