L
LLM Large
Sign In
Advanced AI Interview Lab
Advanced40 Minutes

LLM serving: prefill, decode and speculative sampling

Optimize the stage that limits the user experience. An improvement in aggregate tokens per second can still make an interactive service feel slower.

Prerequisites
Transformer fundamentalsDistributed systems basics
Learning Objectives
  • •Distinguish first-token latency, per-token latency and throughput.
  • •Estimate the transfer cost of disaggregated serving.
  • •Explain when speculative sampling preserves target behavior.

Sources & further reading

Disaggregated prefilling — experimental ↗Speculative sampling ↗vLLM speculative decoding ↗Automatic prefix caching ↗

In-Depth Architecture & Mechanics

Interview brief

A chat service shares GPUs between short chats and long document prompts. Aggregate throughput looks healthy, but existing streams pause when long prompts arrive. Propose a measurement plan before changing the deployment topology.

Separate the clocks

Time to first token includes queueing, routing, prompt processing and first-token generation. Time per output token describes ongoing generation. Track their distributions by prompt/output length and priority class. For an illustrative request with 0.8 seconds to first token and 499 remaining tokens averaging 0.025 seconds each, completion takes about 13.275 seconds. This simple estimate excludes stalls and is not a percentile calculation.

Schedule within a token budget

Continuous batching admits and retires requests as work progresses. Chunked prefill can reduce interruption of decode work, with a trade-off in prompt completion timing. Use deadlines, bounded queues, per-tenant fairness and cancellation propagation. Autoscale on queued token work and SLO violations, not only request count or GPU utilization.

Decide whether to disaggregate

Separate prefill and decode pools when workload evidence justifies independent scaling or reduced interference. The KV state must move between pools. Moving a hypothetical 10-GiB cache over a link delivering 25 GB/s has a payload-only lower bound of about 0.43 seconds. Serialization, contention and synchronization add overhead. vLLM documents disaggregated prefilling as experimental; connector support and behavior are version-dependent.

Use speculation carefully

A draft model or other proposal mechanism suggests tokens that the target verifies. Exact speculative sampling needs a correct acceptance and correction procedure; accepting tokens whenever both models have the same argmax does not reproduce arbitrary target sampling. Speed depends on proposal cost, acceptance, verification efficiency and load. A faster small model can still make the combined service slower.

Follow-up exercise

Compare a colocated baseline, chunked prefill and disaggregated serving on the same arrival trace. Then add speculation and report time to first token, inter-token stalls, accepted draft tokens, throughput, cost and quality. Preserve output-budget and decoding settings when comparing configurations.

System Architecture Flow:
Ingress → bounded token queue → prefill pool → KV transfer → decode pool → stream
                  ↘ cancellation and deadlines apply through every stage
Architectural Tradeoffs
Disaggregation permits independent pools but adds KV transfer and failure handling.
Speculation can reduce serial target work while spending extra compute on proposals that may be rejected.
Common Anti-Patterns
Calling prefix caching an answer cache.
Adding stage p95 values and presenting the sum as measured end-to-end p95.
Benchmarking only homogeneous prompts at zero queue depth.

Staff & Principal Interview Questions

Confidence:
LLM serving: KV cache, GQA and memory budgetsDesign a GPT-style decoder and training pipeline