Design a large-scale GPT-style chat service
Turn model inference into a reliable product: conversations, streams, admission control, regional failure handling and a release process that protects users.
- •Convert daily usage into peak input and output token demand.
- •Plan idempotent turns, stream replay and cancellation.
- •State realistic regional failover behavior for active generations.
Sources & further reading
PagedAttention and serving workloads ↗vLLM disaggregated serving considerations ↗OpenAI evaluation best practices ↗In-Depth Architecture & Mechanics
Interview brief
Design a reference chat service with one million daily users and ten turns per user per day. These are hypothetical workload assumptions, not traffic figures for ChatGPT or any other product. Separate the product architecture from the architecture of the model it calls.
Quantify work in tokens and time
Ten million turns per day average 115.74 requests/second. At an assumed 8× peak, plan for about 925.93 requests/second. With 2,000 prompt and 500 output tokens per turn, peak demand is roughly 1.852 million prompt tokens/second and 462,963 output tokens/second before caching. A hypothetical measured decode capacity of 1,000 output tokens/second per replica implies at least 463 replicas at full saturation, or 662 at a 70% planning utilization. This is not a GPU benchmark or full capacity model.
Use Little’s law carefully
At 925.93 requests/second and a mean active generation duration of 15 seconds, expect about 13,889 active generations in a stable system. Queueing, long tails and retries can raise outstanding work. Use means consistently; substituting p95 latency into the formula does not produce a p95 concurrency guarantee.
Separate durable state from streaming state
An authenticated API creates a turn with an idempotency key and a pinned model/prompt version. Store conversation metadata durably and stream token events with sequence numbers. Propagate cancellation to queues and serving workers. A disconnected browser may reconnect to retained events or request the committed response according to the retention policy. Do not silently regenerate a different answer and present it as the same completed turn.
Protect the failure boundary
Use bounded token-aware queues, per-tenant quotas and retry budgets. Route new requests around a failed region, but describe whether in-flight generations can resume, restart or fail explicitly. Model fallback must respect quality, residency and capability requirements. Tools run behind a separate authorization boundary; generated arguments are not automatically trusted.
Follow-up exercise
Draw separate paths for a cached prefix, a retrieval-backed answer, a tool action and a long-running task. State consistency requirements for conversations, what is persisted during streaming, and which operations can be retried. Define a release canary by model version and task class, with both quality and serving metrics.
Client → authentication + quotas → conversation API → token-aware queue
→ model router → prefill / decode → sequenced stream events → client
Conversation store ← turn status and committed response
Tool proposal → policy + approval → idempotent executor → tool resultfrom math import ceil
turns_per_day = 1_000_000 * 10
peak_rps = turns_per_day / 86_400 * 8
output_tokens_per_second = peak_rps * 500
assumed_measured_rate = 1_000 # tokens/s per replica, not a GPU claim
replicas = ceil(output_tokens_per_second / (assumed_measured_rate * 0.70))
print(f"Peak request rate: {peak_rps:.2f}/s")
print(f"Decode replicas: {replicas}")
assert replicas == 662
# Add prefill, KV-memory limits, tail behavior and failure reserve separately.