Frontier systems: MoE, reasoning budgets and tool execution
Distinguish sparse computation, longer inference and tool access. Each changes a different resource or capability and introduces a different failure mode.
- •Distinguish stored and activated parameters in an MoE model.
- •Evaluate escalation policies at equal expected cost.
- •Handle tool timeouts without duplicating side effects.
Sources & further reading
DeepSeek-V3 technical report ↗Scaling test-time compute ↗Switch Transformers ↗In-Depth Architecture & Mechanics
Interview brief
You are asked to improve difficult reasoning while keeping the median request fast. Compare a larger dense model, a mixture-of-experts model, a larger inference budget and a tool-assisted workflow. Do not assume any one choice is universally best.
MoE changes computation and communication
A router selects a subset of feed-forward experts for a token. Activated parameters differ from total stored parameters; both matter. Sparse activation can reduce arithmetic relative to an equally large dense model, but expert weights still consume memory and expert-parallel execution requires communication. Balance load and inspect overflow, routing skew and all-to-all time. DeepSeek-V3 is a public case study of an MoE architecture with latent attention, not a template for every frontier model.
Allocate inference effort where it helps
Longer generation, multiple candidates or verifier-guided search spend extra inference work. A budget policy can use task class, uncertainty signals, remaining deadline and historical outcomes. Compare success under equal expected cost and latency. The test-time compute literature studies gains under specific models and tasks; additional reasoning is not a proof of correctness.
Bound tools and verification
Use external calculators, search or code execution when they supply a checkable capability. Validate arguments, authorize the action and enforce time, spend and retry limits. A model-generated plan is a proposal. For side effects, persist operation IDs and use idempotent APIs where available. A verifier can reject a bad result without being allowed to execute an unapproved action.
Work the expected-cost calculation
Assume a fast path costs 1 unit and 20% of requests escalate to an additional 5-unit path. Expected cost is 1 + 0.2 × 5 = 2 units, before tool and infrastructure overhead. This doubles the baseline average, even though most requests stay cheap. Evaluate the escalation policy using outcome labels, false escalations, missed hard cases and tail latency.
Follow-up exercise
Choose an escalation policy for code fixes, factual questions and ambiguous enterprise actions. Explain how you would detect a verifier that rewards plausible explanations instead of executable correctness. Include a stop rule when additional attempts stop producing measurable progress.