LLM fundamentals

English · Русский · AI Interview Atlas › Themes

How transformers and LLMs work: attention, tokens, context, sampling, scaling. Engineers derive and implement; leaders need the intuition behind cost, quality and limits.

On this page: Both tracks (2) · AI Engineering (16)

Both tracks

• Under a fixed training compute budget, how do Chinchilla-style scaling results change the balance between model size and training tokens?
  • Knowledge · Asked at: Anthropic † · ✍ Answer · Suggest a fix
  • Tests: Whether empirical training trade-offs are kept separate from deployment economics.
  • A strong answer covers:
    • State the fixed-compute setting and the empirical nature of the fit.
    • Balance parameter count with sufficient training tokens.
    • Separate training-optimal allocation from inference cost and data quality constraints.
  • Read: Training Compute-Optimal Large Language Models (Hoffmann et al., arXiv)
• What objectives and data distinguish pretraining, instruction tuning and preference optimisation?
  • Knowledge · Asked at: Meta † · Suggest a fix
  • Tests: Whether the candidate can identify what each stage can and cannot change.
  • A strong answer covers:
    • Name the objectives: next-token loss on raw corpora, the same loss on demonstrations masked to responses, then pairwise preferences.
    • State what each changes: pretraining sets knowledge and skills, instruction tuning sets format, preferences re-rank behaviours already available.
    • Probe the limits: facts added only in tuning generalise poorly, and preference training can reward length or flattery.
  • Read: Training language models to follow instructions with human feedback (Ouyang et al., arXiv) · Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., arXiv) · Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (Gekhman et al., arXiv)

AI Engineering

• Derive scaled dot-product attention and explain how its scale affects softmax gradients.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether tensor shapes and numerical behaviour support the explanation.
  • A strong answer covers:
    • Track Q, K and V shapes through score computation and weighted summation.
    • Scale by the square root of key width; discuss saturation of softmax.
    • Apply the causal mask before softmax and check numerical stability.
  • Read: Attention Is All You Need (Vaswani et al., arXiv)
• Derive the memory required by a decoder's KV cache as batch size and context length grow.
  • Knowledge · Asked at: Amazon †, OpenAI † · ✍ Answer · Suggest a fix
  • Tests: Whether cache size is calculated from the actual architecture.
  • A strong answer covers:
    • For equal sequence lengths, use 2 × batch × layers × tokens × KV heads × head width × bytes.
    • Explain why queries are not retained for future decode steps.
    • Account separately for weights, activations, padding and allocation overhead.
  • Read: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., arXiv) · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., arXiv)
• Compare multi-head, multi-query and grouped-query attention in quality and serving memory.
  • Knowledge · Asked at: Meta † · ✍ Answer · Suggest a fix
  • Tests: Whether shared keys and values are distinguished from query heads.
  • A strong answer covers:
    • MHA uses separate KV heads; MQA shares one; GQA shares within groups.
    • Relate KV-head count to cache size and memory reads.
    • Discuss quality evaluation and the need to train or adapt the checkpoint.
  • Read: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., arXiv)
• Explain how latent attention compresses the KV representation and what must be reconstructed at inference.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether architectural compression is distinguished from lower numeric precision.
  • A strong answer covers:
    • Cache one low-rank latent vector per token instead of full keys and values for every head.
    • Reconstruct per-head keys and values from that latent, or fold the up-projections into the query and output projections, paying extra compute.
    • Separate this architectural compression from quantising the same cache.
  • Read: DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (DeepSeek-AI, arXiv)
• Why can an exact attention kernel run faster without changing the quadratic attention computation?
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether memory traffic is separated from arithmetic complexity.
  • A strong answer covers:
    • Explain tiled attention and the reduction in reads and writes between accelerator memory levels.
    • Distinguish exact attention with floating-point differences from sparse or approximate attention.
    • Separate quadratic arithmetic from memory traffic; benchmark the relevant shapes and hardware.
  • Read: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., arXiv)
• Explain BPE training and why numbers, source code and different writing systems can tokenize unevenly.
  • Knowledge · Suggest a fix
  • Tests: Whether tokenisation choices are connected to context use and representation.
  • A strong answer covers:
    • Describe training: start from characters or bytes and repeatedly merge the most frequent adjacent pair until the vocabulary is full.
    • Link unevenness to corpus frequency and pre-tokenisation rules: rare scripts fragment, digit groups vary, code indentation splits oddly.
    • Measure tokens per character or word by language and domain, and the effect on context, cost and arithmetic.
  • Read: Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., arXiv) · Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al., arXiv)
• Compare additive positional embeddings, rotary embeddings and attention biases for representing token order.
  • Knowledge · Suggest a fix
  • Tests: Whether different mechanisms for encoding position are understood.
  • A strong answer covers:
    • Show where position enters: added to input embeddings, rotated into queries and keys, or added to attention logits.
    • Contrast absolute learned or sinusoidal tables with RoPE's relative-offset dot products and ALiBi's linear distance penalty.
    • Test behaviour beyond the training length on in-context retrieval, since perplexity hides weak use of distant tokens.
  • Read: Attention Is All You Need (Vaswani et al., arXiv) · RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., arXiv) · Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Press et al., arXiv)
• Explain rotary position encoding and the trade-offs of interpolation when extending context beyond training lengths.
  • Knowledge · Asked at: Meta † · Suggest a fix
  • Tests: Whether a longer accepted input is distinguished from reliable long-context reasoning.
  • A strong answer covers:
    • Explain RoPE: rotate query and key dimension pairs by position times a per-pair frequency, so scores depend on offset.
    • Show that position interpolation divides every frequency by the scale, blurring nearby tokens; NTK and YaRN mostly rescale low frequencies.
    • Separate accepted length from use: test retrieval and multi-hop tasks at varied depths, plus short-context regressions after finetuning.
  • Read: RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., arXiv) · Lost in the Middle: How Language Models Use Long Contexts (Liu et al., arXiv) · Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., arXiv) · YaRN: Efficient Context Window Extension of Large Language Models (Peng et al., arXiv) · RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al., arXiv)
• How does sparse expert routing increase parameter capacity, and what costs remain despite activating only some experts?
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether active computation, total memory and communication are separated.
  • A strong answer covers:
    • Separate the total parameter count from the parameters activated for one token.
    • Account for memory that must hold every expert and for routing traffic between devices.
    • Discuss load imbalance, expert capacity and dropped tokens.
  • Read: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Fedus et al., arXiv)
• Compare greedy decoding, beam search and temperature, top-k and nucleus sampling; give a failure mode for each.
  • Knowledge · Asked at: Google and Google DeepMind †, Perplexity † · Suggest a fix
  • Tests: Whether probability manipulation is connected to output diversity and quality.
  • A strong answer covers:
    • Describe each as a transform: argmax, approximate sequence-likelihood search, logit division by T, top-k truncation, cumulative-mass truncation.
    • Name failures: greedy and beam repeat or go bland, high temperature derails, top-k ignores distribution shape, top-p admits flat tails.
    • Choose by task, deterministic for extraction and sampled for generation, and measure accuracy together with diversity across seeds.
  • Read: The Curious Case of Neural Text Degeneration (Holtzman et al., arXiv)
• How would you test and reduce a model's tendency to miss relevant evidence in the middle of a long context?
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether context utilisation is measured rather than inferred from context capacity.
  • A strong answer covers:
    • Move the same relevant evidence through controlled context positions.
    • Measure accuracy across lengths, distractors and model variants.
    • Compare retrieval, ordering and context reduction without assuming a universal cure.
  • Read: Lost in the Middle: How Language Models Use Long Contexts (Liu et al., arXiv)
• Compare pre-normalisation and post-normalisation transformer blocks, then explain what RMSNorm changes.
  • Knowledge · Suggest a fix
  • Tests: Whether normalisation is linked to optimisation stability and its actual formula.
  • A strong answer covers:
    • Write both blocks: post-norm applies LayerNorm after the residual add; pre-norm normalises only the sublayer input.
    • Link placement to stability: pre-norm keeps an identity gradient path and needs less warmup, but residual norms grow with depth.
    • Give RMSNorm as x divided by its root mean square, times a gain: no mean-centring or bias, fewer operations.
  • Read: Attention Is All You Need (Vaswani et al., arXiv) · On Layer Normalization in the Transformer Architecture (Xiong et al., arXiv) · Root Mean Square Layer Normalization (Zhang et al., arXiv)
• Explain the two branches of SwiGLU and the parameter and compute trade-offs against a conventional feed-forward block.
  • Knowledge · Asked at: Meta † · Suggest a fix
  • Tests: Whether gated activations are explained beyond their name.
  • A strong answer covers:
    • Write SwiGLU: Swish of one projection gates a second linear projection elementwise, then a down projection returns to d_model.
    • Count parameters: three matrices instead of two, so hidden width shrinks to about 8d/3 to keep 8d² weights.
    • Compare at matched parameters and FLOPs on held-out loss, noting extra activation memory and fused gate-up kernels.
  • Read: GLU Variants Improve Transformer (Shazeer, arXiv)
• Trace the tensor shapes through one decoder-only transformer layer and through the final vocabulary projection.
  • Knowledge · Asked at: Anthropic † · Suggest a fix
  • Tests: Whether attention, residual paths and vocabulary logits form a coherent computation.
  • A strong answer covers:
    • Carry [B, T, d_model] through norm, Q, K, V projections split into heads, [B, H, T, T] scores and the output projection.
    • Keep the residual stream at d_model throughout; the MLP expands to d_ff and back before each residual add.
    • Project the final normed stream through a d_model by vocabulary matrix to [B, T, V] logits; recheck shapes for one cached decode step.
  • Read: Attention Is All You Need (Vaswani et al., arXiv)
• Relate cross-entropy, KL divergence and perplexity for next-token prediction; state the assumptions behind each comparison.
  • Knowledge · Asked at: OpenAI † · Suggest a fix
  • Tests: Whether probabilistic definitions are used consistently.
  • A strong answer covers:
    • Show cross-entropy equals data entropy plus KL from data to model, so for fixed data minimising either is equivalent.
    • Define perplexity as exp of mean per-token cross-entropy in nats; it shifts with tokeniser, eval text and context length.
    • Compare across tokenisers in bits per byte, fixing the text, context handling and log base before drawing conclusions.
  • Read: Deep Learning: Probability and Information Theory (Ian Goodfellow, Yoshua Bengio and Aaron Courville)
• Explain how self-attention cost grows with context length and compare ways to reduce that cost.
  • Knowledge · Asked at: OpenAI † · Suggest a fix
  • Tests: Whether efficiency changes are distinguished from changes to model behaviour.
  • A strong answer covers:
    • Show the cost: prefill score and value products scale with T squared; each decode step reads a KV cache linear in T.
    • Separate exact kernels like FlashAttention, which cut memory traffic, from windows, sparsity, KV eviction or GQA, which change outputs.
    • Re-run long-context quality evals for behaviour-changing methods; for exact ones, check numerical equivalence and speed only.
  • Read: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., arXiv) · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., arXiv)

Inference, serving and cost →