Inference, serving and cost

English · Русский · AI Interview Atlas › Themes

Serving models at scale: prefill and decode, batching, caching, quantisation, latency metrics and the levers that move cost.

On this page: Both tracks (4) · AI Engineering (14)

Both tracks

• A service can spend more on one reasoning run or generate several candidates and verify them. How would you allocate compute under a latency deadline?
  • Applied scenario · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
  • Tests: Whether compute allocation follows measured task value and includes verification overhead.
  • A strong answer covers:
    • Compare strategies on the same held-out tasks, including verifier cost and wall-clock latency.
    • Measure success by task difficulty and deploy a routing rule that does not require knowing the correct answer.
    • Set stopping and fallback rules; report total spend per successful task and deadline misses.
  • Read: Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Snell et al., arXiv) · Evaluation best practices (OpenAI API documentation)
• Distinguish time to first token, time per output token, inter-token latency and throughput when comparing serving systems.
  • Knowledge · Asked at: Microsoft †, Perplexity † · ✍ Answer · Suggest a fix
  • Tests: Whether user-visible delay is separated from aggregate capacity.
  • A strong answer covers:
    • Define TTFT as queueing plus prefill, TPOT and inter-token latency as decode pace, throughput as aggregate capacity.
    • Explain how larger batches trade each user's decode speed for throughput, and how admitted prefills stall decodes.
    • Compare at matched load and length distributions: p50 and p99 per latency metric, throughput within the targets.
  • Read: vLLM documentation (vLLM project)
• How would you benchmark vLLM, SGLang, TensorRT-LLM and a custom runtime for a specific workload?
  • Applied scenario · Suggest a fix
  • Tests: Whether a reproducible workload drives the choice instead of a generic ranking.
  • A strong answer covers:
    • Replay a captured workload: real prompt and output length distributions, arrival pattern, shared prefixes and concurrency.
    • Hold model, precision, parallelism and hardware fixed, tune each engine fairly, and count the custom runtime's maintenance cost.
    • Report goodput, requests per second within TTFT and inter-token latency SLOs, and check outputs match a baseline implementation.
  • Read: vLLM documentation (vLLM project) · DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (Zhong et al., arXiv)
• You are asked to reduce serving cost by an order of magnitude. Rank the levers and explain how you would test whether that target is achievable.
  • Applied scenario · Asked at: Amazon †, Microsoft † · ✍ Answer · Suggest a fix
  • Tests: Whether savings are measured under explicit quality and latency constraints.
  • A strong answer covers:
    • Measure cost per successful task at the required quality and SLO.
    • Compare smaller models, shorter contexts, reuse, batching and precision changes.
    • Benchmark each change and check whether the combined savings actually reach the target.
  • Read: Optimizing inference (Hugging Face Transformers docs) · Site Reliability Engineering (Google)

AI Engineering

• A reasoning API request consumes tokens but returns no usable answer. How would you diagnose it and design a bounded recovery policy?
  • Applied scenario · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
  • Tests: Whether token accounting, completion status and retry budgets are treated separately.
  • A strong answer covers:
    • Inspect completion status, token usage, context size and the provider's output-limit semantics.
    • Distinguish exhaustion from tool errors, refusal and timeout before choosing recovery.
    • Bound total attempts, spend and elapsed time; measure recovery success and avoid duplicate tool actions.
  • Read: Reasoning models (OpenAI API documentation)
• Contrast prefill with autoregressive decode and explain when compute or memory bandwidth becomes the bottleneck.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether bottlenecks are conditioned on workload and batch size.
  • A strong answer covers:
    • Prefill processes prompt positions in parallel; decode advances autoregressively.
    • Small-batch decode often reads weights with limited arithmetic reuse.
    • Profile sequence lengths and batching before declaring either phase compute- or bandwidth-bound.
  • Read: Optimizing inference (Hugging Face Transformers docs)
• Explain iteration-level batching and how requests enter and leave a running inference batch.
  • Knowledge · Asked at: Anthropic † · ✍ Answer · Suggest a fix
  • Tests: Whether scheduling is connected to utilisation, fairness and waiting time.
  • A strong answer covers:
    • Admit new sequences at iteration boundaries and remove completed ones.
    • Enforce token and KV-memory budgets rather than request count alone.
    • Measure queueing and starvation alongside throughput.
  • Read: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., arXiv)
• How does paging the KV cache reduce waste, and what remains for the scheduler to solve?
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether memory management is distinguished from request scheduling.
  • A strong answer covers:
    • Explain fragmentation from contiguous allocation sized for the longest possible answer.
    • Describe fixed-size blocks with an indirection table and blocks shared between requests.
    • Name what paging leaves open: admission, preemption and fairness remain scheduling policy.
  • Read: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., arXiv)
• Explain exact speculative decoding, its acceptance step and workloads where the draft model adds overhead instead of speed.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether speedup is explained without promising a universal quality or latency benefit.
  • A strong answer covers:
    • A smaller model proposes several tokens for parallel target verification.
    • Describe acceptance and correction sampling that preserve the target distribution.
    • Evaluate acceptance rate, draft cost and available parallel compute.
  • Read: Fast Inference from Transformers via Speculative Decoding (Leviathan et al., arXiv)
• What makes a prompt prefix reusable, and which changes invalidate or reduce the value of a prefix cache?
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether exact reuse, cache lifetime and tenant isolation are considered.
  • A strong answer covers:
    • State the condition for reuse: an identical token prefix under the same model and serving configuration.
    • Order the prompt so the stable part comes first and volatile content comes last.
    • Bound cache lifetime and scope entries so one tenant cannot read another's prefix.
  • Read: Prompt caching (Claude Platform Docs)
• Compare floating-point and integer serving formats down to four bits; how would you validate an acceptable reduction in precision?
  • Knowledge · Suggest a fix
  • Tests: Whether memory savings are checked against quality, kernels and hardware support.
  • A strong answer covers:
    • Contrast non-uniform FP8 and FP4 grids with uniform INT8 and INT4 grids, and the per-group scales low bit widths need.
    • Separate weight-only 4-bit, which speeds memory-bound decode, from activation quantisation needing native FP8 or INT8 kernels on the target GPU.
    • Validate against bf16 on task evals and per-token KL, with long-context, code and maths slices, then measure real throughput.
  • Read: QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., arXiv) · Optimizing inference (Hugging Face Transformers docs) · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., arXiv) · SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (Xiao et al., arXiv) · FP8 Formats for Deep Learning (Micikevicius et al., arXiv)
• Compare tensor, pipeline, data, sequence and expert parallelism for a large model deployment.
  • Knowledge · Asked at: Amazon †, Google and Google DeepMind †, Meta † · Suggest a fix
  • Tests: Whether communication patterns and memory limits determine the choice.
  • A strong answer covers:
    • Name each split and its traffic: TP per-layer all-reduce, PP stage activations, SP key-value exchange, EP token all-to-all.
    • Fit weights and KV first: TP inside the NVLink domain, PP across nodes, then traffic-free data-parallel replicas for throughput.
    • Measure per-GPU memory headroom, communication share of step time, pipeline bubbles and expert load imbalance at target latency.
  • Read: vLLM documentation (vLLM project) · Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Fedus et al., arXiv) · Efficiently Scaling Transformer Inference (Pope et al., arXiv) · Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al., arXiv)
• Estimate a serving memory budget for a 70-billion-parameter model, explicitly choosing precision, context and concurrency assumptions.
  • Applied scenario · ✍ Answer · Suggest a fix
  • Tests: Whether weights, cache and runtime headroom are all budgeted.
  • A strong answer covers:
    • Weight storage is parameter count times bytes per parameter, before quantisation metadata.
    • Add architecture-specific KV storage for the chosen concurrent sequences.
    • Reserve activations, workspace and allocator headroom; state any sharding assumptions.
  • Read: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., arXiv)
• Build a roofline estimate for single-request decoding of a 70B model on an H100; check first whether the chosen representation fits.
  • Applied scenario · Suggest a fix
  • Tests: Whether bandwidth, capacity and achievable utilisation constrain the estimate.
  • A strong answer covers:
    • Check fit first: 70B in bf16 is about 140 GB, beyond one 80 GB H100; FP8 leaves little KV room.
    • Compare bytes read per token, all weights plus KV, with about 2 FLOPs per parameter: batch-one decode is bandwidth-bound.
    • Bound tokens per second by achievable rather than peak HBM bandwidth, then check the estimate against a measured profile.
  • Read: Efficiently Scaling Transformer Inference (Pope et al., arXiv)
• After a deployment, p99 latency doubles although model weights are unchanged. How do you isolate the cause?
  • Applied scenario · Asked at: Amazon †, Databricks †, OpenAI †, Perplexity † · ✍ Answer · Suggest a fix
  • Tests: Whether diagnosis separates queueing, scheduling, dependencies and traffic changes.
  • A strong answer covers:
    • Compare traffic mix, queue time and per-stage latency before and after release.
    • Inspect cache hit rates, batching, dependency tails and resource saturation.
    • Reproduce one hypothesis at a time and use a rollback or canary to verify causality.
  • Read: The Tail at Scale (Dean and Barroso, Google Research)
• Why can splitting prefill into chunks help mixed prompt and decode traffic, and what trade-off does chunk size introduce?
  • Knowledge · Suggest a fix
  • Tests: Whether throughput and decode stalls are considered together.
  • A strong answer covers:
    • Explain the stall: a long prefill monopolises an iteration, so every running decode waits and inter-token latency spikes.
    • Mix prefill chunks with decodes under a per-iteration token budget; smaller chunks smooth decode but raise TTFT and overhead.
    • Sweep the token budget on mixed traffic, plotting p99 inter-token latency against TTFT and throughput.
  • Read: vLLM documentation (vLLM project) · Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (Agrawal et al., arXiv)
• When would you separate prefill and decode onto different workers, and how would you price the KV transfer overhead?
  • System design · Suggest a fix
  • Tests: Whether independent scaling is weighed against transport and operational complexity.
  • A strong answer covers:
    • Separate phases when prefill bursts break decode latency SLOs or the two phases want different parallelism and hardware.
    • Price transfer as KV bytes per token times prompt length over link bandwidth, overlapped layer by layer with prefill.
    • Compare goodput per GPU against colocated chunked prefill, including transfer tail latency and rebalancing the worker ratio.
  • Read: vLLM documentation (vLLM project) · DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (Zhong et al., arXiv)
• In Anthropic's original performance-engineering take-home, introduced in 2024 and subsequently retired, optimise a serial program for a simulated accelerator using multicore, vector and instruction-level parallelism.
  • Coding · Asked at: Anthropic ✅ · Suggest a fix
  • Tests: Whether you find parallelism methodically and build tooling to measure and debug each step.
  • A strong answer covers:
    • Lock correctness first: check every output against the serial version, and fix any existing bug before optimising.
    • Find independent iterations from data dependencies, then spread them across cores, SIMD lanes and per-cycle instruction slots.
    • Measure simulated cycles after every change, and use a per-instruction trace to find stalls and idle units.
  • Read: Designing AI resistant technical evaluations (Anthropic)

← LLM fundamentals · RAG and retrieval →