Post-training, preference learning and release evaluation
Choose the intervention that matches the failure. More supervised data, a preference objective and a better evaluation set address different problems.
- •Choose demonstrations or preference data to address a specific failure.
- •Calibrate open-ended judges against independent human labels.
- •Interpret small evaluation gains with uncertainty and critical-error checks.
Sources & further reading
InstructGPT ↗Direct Preference Optimization ↗OpenAI evaluation best practices ↗In-Depth Architecture & Mechanics
Interview brief
A base model knows the domain but ignores response-format instructions and sometimes gives confident unsupported answers. Design the data, training stages and release gate. Keep evaluation independent of the examples used to optimize the model.
Separate capabilities from preferences
Supervised fine-tuning learns from demonstrations. Preference learning uses comparisons or rewards to favor some outputs over others. InstructGPT is a documented example of supervised training followed by reward modeling and reinforcement learning. DPO learns from preference pairs without a separate explicit reward-model training stage in its original formulation. Neither is a guarantee of truthfulness.
Build the dataset around actual mistakes
Collect representative prompts, correct demonstrations, difficult negative examples and explicit labels for missing evidence. Remove personal data as appropriate and keep provenance. For preference pairs, specify whether the comparison concerns correctness, instruction following, brevity or tone. Separate these dimensions when a single preference label would hide disagreement.
Design a release gate
Use deterministic checks for parsable schemas and executable tests when the task supports them. Use a documented rubric for open-ended responses, calibrated against human judgments. Hide answer identity in pairwise judging and swap order to detect position bias. Keep a private holdout and record prompt, model, retrieval and grader versions.
Work an uncertainty example
If a model passes 180 of 200 approximately independent binary tasks, the estimated rate is 90%. The normal-approximation 95% interval is about 90% ± 4.2 percentage points. For small samples or rates near 0 or 1, use an appropriate interval such as Wilson; correlated questions reduce effective sample size. A two-point apparent gain on this set is weak evidence by itself. Use paired comparisons and inspect critical failure categories.
Follow-up exercise
Propose a rollout with a fixed latency and cost budget, explicit non-regression checks, a canary and rollback criteria. Track helpfulness, unsupported claims, refusal/coverage, tool success and user-visible latency separately. Do not reward a system for refusing every hard question. The linked OpenAI guidance supports evaluation principles; this lesson does not depend on any particular hosted eval API.