Principal40 Minutes

System Design: Enterprise Multimodal RAG Platform

An enterprise RAG system must handle not just semantic retrieval, but enterprise realities: role-based access control (RBAC), multi-tenancy, asynchronous document ingestion pipelines, evaluation telemetry, and strict latency SLAs.

Prerequisites

Vector databasesMicroservicesKafka

Learning Objectives

•Design an enterprise RAG system serving 10,000 concurrent queries with <1s p99.
•Handle real-time document ingestion with change data capture (CDC) and vector updates.
•Incorporate document access control lists (ACLs) into vector search without data leakage.

In-Depth Architecture & Mechanics

Key Architecture Components

1. **Ingestion Plane (Asynchronous)**: - Connectors (Google Drive, Confluence, S3) emit events to Kafka. - Workers parse PDFs, extract OCR for diagrams, and chunk documents hierarchically. - Text chunks are embedded and indexed into Milvus/Qdrant alongside tenant IDs and User/Group ACL IDs.

2. **Serving Plane (Synchronous)**: - API Gateway verifies OAuth JWT tokens and tenant scope. - Semantic Cache checks Redis for identical or high-similarity recent queries. - Retrieval Engine executes **Hybrid Search** with pre-filtered ACL tags (`tenant_id = X AND acl_groups IN (user_groups)`). - Cross-Encoder reranks the top 30 candidates to top 5. - Guardrails check query and context for PII/prompt injection. - Streaming LLM inference over Server-Sent Events (SSE).

System Architecture Flow:

User ──> [ API Gateway & OAuth ] ──> [ Semantic Cache (Redis) ]
                     │ (Cache Miss)
                     ▼
         [ Hybrid Retriever (Milvus + OpenSearch) ]
                     │ With ACL Filters
                     ▼
             [ Cross-Encoder Reranker ]
                     │
                     ▼
           [ Guardrails & LLM Inference ] ──> SSE Stream to User

Worked code example:

Milvus ACL Pre-Filtering Query Example

python

search_params = {"metric_type": "COSINE", "params": {"nprobe": 16}}
# Enforce zero data leakage across tenants and unauthorized groups
expr = 'tenant_id == "tenant_123" and acl_group in ["engineering", "all_staff"]'

results = collection.search(
    data=[query_embedding],
    anns_field="vector",
    param=search_params,
    limit=30,
    expr=expr,
    output_fields=["chunk_id", "parent_id", "text"]
)

Architectural Tradeoffs

Pre-filtering vs Post-filtering in Vector Search: Post-filtering vector results by ACL can yield fewer than k results if top matches are unauthorized. Pre-filtering guarantees k authorized items but requires metadata indexing.

Common Anti-Patterns

Failing to enforce ACLs at the vector query stage, allowing unauthorized document fragments to leak into LLM prompts.

Staff & Principal Interview Questions

Confidence: