Fine-tuning and post-training

English · Русский · AI Interview Atlas › Themes

Supervised fine-tuning, preference optimisation, reinforcement learning, parameter-efficient methods and distillation.

On this page: Both tracks (2) · AI Engineering (13)

Both tracks

• Choose between prompt changes, retrieval and finetuning for a failing AI feature; include data freshness, latency and total cost.
  • Applied scenario · Asked at: Databricks †, Microsoft †, OpenAI † · ✍ Answer · Suggest a fix
  • Tests: Whether the intervention matches the observed failure mode.
  • A strong answer covers:
    • Classify missing knowledge, poor instructions and unwanted behaviour separately.
    • Establish a prompt baseline; test retrieval for fresh attributable knowledge.
    • Use finetuning when representative examples justify it, then compare lifecycle cost and held-out quality.
  • Read: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., arXiv) · LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., arXiv)
• Explain how Constitutional AI uses principles, self-revision and AI-generated preferences, and identify what human judgement it still depends on.
  • Knowledge · Asked at: Anthropic † · ✍ Answer · Suggest a fix
  • Tests: Whether reduced labelling effort is distinguished from removing human value choices.
  • A strong answer covers:
    • Separate supervised self-critique and revision from reinforcement learning on AI-generated preferences.
    • Explain how the selected principles shape critiques, comparisons and the learned preference model.
    • Identify human choices in the constitution and evaluation; reduced harmfulness labelling does not remove value judgements.
  • Read: Constitutional AI: Harmlessness from AI Feedback (Bai et al., arXiv)

AI Engineering

• Walk through a preference-based RLHF pipeline and explain the roles of the reward model and reference-policy penalty.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether data collection, reward fitting and policy optimisation are distinguished.
  • A strong answer covers:
    • Separate demonstrations, preference pairs and policy rollouts.
    • Explain reward-model training and policy optimisation with a reference constraint.
    • Evaluate held-out human preferences and reward overoptimisation.
  • Read: Training language models to follow instructions with human feedback (Ouyang et al., arXiv)
• Compare DPO with PPO-based RLHF, including the assumptions behind offline preference learning and reasons to collect fresh rollouts.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether a simpler objective is distinguished from universally better alignment.
  • A strong answer covers:
    • Describe the preference objective using policy and reference log-probabilities.
    • Explain how it avoids a separately trained reward model in the basic formulation.
    • Discuss preference coverage and distribution shift rather than declaring online RL obsolete.
  • Read: Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., arXiv)
• Explain group-relative policy optimisation and the trade-offs of estimating advantages without a learned value model.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether relative rewards and variance are connected to sample groups.
  • A strong answer covers:
    • Sample several completions per prompt and estimate advantages relative to their group rewards.
    • Explain the avoided value-model memory and the remaining rollout and reward-computation costs.
    • Inspect groups with little reward variation; compare held-out task success and training stability.
  • Read: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Shao et al., arXiv)
• Derive the low-rank weight update used by LoRA and propose an experiment for choosing its rank.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether parameter savings are linked to adaptation capacity and validation.
  • A strong answer covers:
    • Represent the update as ΔW = BA and state both matrix shapes.
    • Compare trainable parameter counts with a full weight matrix.
    • Tune rank and target modules on held-out tasks; check regressions and merging constraints.
  • Read: LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., arXiv)
• How does QLoRA reduce finetuning memory, and which tensors remain trainable or require higher precision?
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether quantised base weights are separated from adapter optimisation.
  • A strong answer covers:
    • Keep the base model quantised and frozen while training low-rank adapters.
    • Explain NF4, double quantisation and paged optimisers as separate memory mechanisms.
    • Account for dequantisation compute and remaining activation memory.
  • Read: QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., arXiv)
• Compare low-rank adapters, prefix tuning, soft prompt tuning and full finetuning for one task.
  • Knowledge · Suggest a fix
  • Tests: Whether data, capacity and deployment constraints guide adaptation.
  • A strong answer covers:
    • Place each method: LoRA adds low-rank weight updates, prefix tuning trains per-layer key-value prefixes, prompt tuning trains input embeddings.
    • Weigh capacity against cost: full finetuning stores optimiser states and a model per task; adapters merge or swap per request.
    • Compare all at matched data on held-out task metrics and general regressions, including serving latency and memory.
  • Read: LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., arXiv) · Prefix-Tuning: Optimizing Continuous Prompts for Generation (Li et al., arXiv) · The Power of Scale for Parameter-Efficient Prompt Tuning (Lester et al., arXiv)
• After domain finetuning, general capabilities regress. How would you measure and reduce catastrophic forgetting?
  • Applied scenario · Suggest a fix
  • Tests: Whether task gains are checked against preserved capabilities.
  • A strong answer covers:
    • Fix a retention suite before training and score base and finetuned checkpoints on it alongside the target task.
    • Reduce drift with replayed general data, a lower learning rate or fewer steps, adapters, a KL penalty or weight interpolation.
    • Plot task gain against retention across checkpoints and mixing ratios, and pick a point on that frontier.
  • Read: Training language models to follow instructions with human feedback (Ouyang et al., arXiv) · LoRA Learns Less and Forgets Less (Biderman et al., arXiv)
• Budget memory for full bf16 finetuning of a 7B model with Adam, then recalculate for LoRA and explain your optimiser assumptions.
  • Applied scenario · ✍ Answer · Suggest a fix
  • Tests: Whether gradients, optimiser state, master weights and activations are all considered.
  • A strong answer covers:
    • State the precision and bytes for weights, gradients, Adam moments and any master-weight copy.
    • For LoRA, separate frozen base weights from trainable adapters and their optimiser state.
    • Budget activations, temporary buffers and headroom separately; validate peak memory on a representative batch.
  • Read: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al., arXiv) · LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., arXiv)
• When is a programmatically verifiable reward preferable to a learned reward model, and where can the verifier still be exploited?
  • Knowledge · Suggest a fix
  • Tests: Whether verification is treated as a specification with blind spots.
  • A strong answer covers:
    • Prefer verifiers where correctness is cheap and objective, like exact answers or tests, avoiding reward-model overoptimisation.
    • Name the blind spots: weak tests, special-cased outputs, parser loopholes, edited graders and right answers from wrong reasoning.
    • Audit high-reward samples and track the gap between training reward and hidden tests or independent review.
  • Read: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (v1) (DeepSeek-AI, arXiv) · Scaling Laws for Reward Model Overoptimization (Gao, Schulman and Hilton, arXiv)
• Policy reward keeps rising while humans prefer its outputs less. How would you diagnose and limit reward hacking?
  • Applied scenario · ✍ Answer · Suggest a fix
  • Tests: Whether optimisation of the proxy is distinguished from improvement in the real goal.
  • A strong answer covers:
    • Compare proxy reward with blinded held-out human preferences and inspect newly rewarded behaviours.
    • Test earlier checkpoints and reduced optimisation strength; improve preference coverage before retraining.
    • Treat KL control and early stopping as mitigations, not guarantees; distinguish synthetic gold-model evidence from human evaluation.
  • Read: Scaling Laws for Reward Model Overoptimization (Gao, Schulman and Hilton, arXiv)
• Design a teacher-to-student distillation process and explain how you would detect capabilities the smaller model fails to retain.
  • System design · Suggest a fix
  • Tests: Whether data coverage and independent evaluation constrain teacher imitation.
  • A strong answer covers:
    • Build prompts covering the capability distribution, including rare languages, long context and refusals, not only easy traffic.
    • Choose teacher outputs or token-level logit KL, and add on-policy student samples so it learns from its own mistakes.
    • Evaluate by capability slice against independent labels, not teacher agreement, and flag slices where the student's gap widens.
  • Read: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (v1) (DeepSeek-AI, arXiv) · Distilling the Knowledge in a Neural Network (Hinton et al., arXiv) · On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (Agarwal et al., arXiv)
• A long pretraining run suddenly diverges. How do you investigate data, numerics and optimiser state before resuming from a checkpoint?
  • Applied scenario · Asked at: Google and Google DeepMind † · Suggest a fix
  • Tests: Whether recovery is based on a reproducible diagnosis.
  • A strong answer covers:
    • Locate the first anomalous step in gradient norm, attention-logit and activation magnitudes, and update-to-weight ratios.
    • Replay from the prior checkpoint with identical data order; a reproduced spike implicates data or numerics, otherwise suspect hardware.
    • Apply one targeted fix, such as skipping batches, a lower learning rate, tighter clipping or z-loss, and compare the replayed trajectory.
  • Read: PaLM: Scaling Language Modeling with Pathways (Chowdhery et al., arXiv) · Small-scale proxies for large-scale Transformer training instabilities (Wortsman et al., arXiv)
• Design training for a model that cannot fit on one accelerator; explain how you partition state and keep communication affordable.
  • System design · Asked at: Google and Google DeepMind † · ✍ Answer · Suggest a fix
  • Tests: Whether model state, activations and network topology determine the parallel plan.
  • A strong answer covers:
    • Separate parameters, gradients, optimiser state and activations before choosing a partitioning plan.
    • Explain ZeRO state sharding and its communication trade-offs; distinguish it from splitting layer computation.
    • Measure peak memory, communication and throughput on the target topology before scaling the run.
  • Read: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al., arXiv)

← Agents, tools and protocols · Evaluation and observability →