Design a GPT-style decoder and training pipeline
Start from tokenization and a causal objective, then work through training memory, distributed execution and evaluation. This is a reference decoder design, not a claim about private GPT internals.
- •Trace shifted targets and causal attention through a decoder.
- •Estimate optimizer state and explain sharding trade-offs.
- •Design a restart that restores the training state and data position.
Sources & further reading
Language Models are Few-Shot Learners ↗RoFormer and rotary position embeddings ↗ZeRO: memory optimization for large models ↗In-Depth Architecture & Mechanics
Interview brief
Design a hypothetical 7-billion-parameter text decoder. Explain the data path, attention mask, training objective and checkpoint recovery. The model size is an interview assumption; architecture choices must be justified rather than attributed to an unreleased model.
Explain one forward pass
Token IDs map to embeddings. Repeated decoder blocks apply causal self-attention and feed-forward transformations with residual connections and normalization. The output projection produces vocabulary logits. Modern reference designs may use rotary positions, gated feed-forward layers and grouped-query attention, but these are design alternatives rather than universal GPT requirements.
Get the objective and mask right
Train next-token prediction with shifted targets: input tokens [t0, t1, t2] predict [t1, t2, t3]. A token cannot attend to future tokens. Padding and packed-example boundaries need explicit handling consistent with the training objective. Never report a loss that accidentally includes target leakage. Track held-out loss as well as downstream task behavior.
Budget a training step
An illustrative mixed-precision Adam setup with BF16 weights and gradients, FP32 master weights and two FP32 optimizer moments needs roughly 16 bytes per parameter before activations and buffers: 7 billion × 16 = 112 GB. Implementations vary. Sharding can reduce replicated optimizer and parameter state; activation checkpointing trades recomputation for memory. Tensor and pipeline parallelism add communication and scheduling constraints.
Make the data and run reproducible
Version licenses, provenance, filters, deduplication and train/validation splits. Partition by source or time where random splitting would leak near-duplicates. Persist optimizer, scheduler, random state, data-loader position and shard metadata with checkpoints. Test resumption on a small run and compare losses before committing a large budget.
Follow-up exercise
Draw the tensor shapes for a batch B, sequence S, hidden width H and vocabulary V. Explain why materializing B × S × V logits can be expensive. Then design a tiny causal-mask test, a data-leakage test and an interrupted-run recovery test. Report actual tokens processed per second and communication overhead rather than only advertised device FLOPs.
Versioned data → tokenize and pack → causal decoder → next-token loss
↘ held-out evaluation ↘ optimizer + checkpoints
Base model → supervised/post-training stages → release gates → serving