Skip to content

Answers: AI Engineering

English · Русский · AI Interview Atlas › Tracks and roles

Written answers to the priority questions of this track, in the same order and numbering as Start here. Answer each question yourself first: what follows is one good answer, not the only correct one, and an interviewer is listening to your reasoning rather than checking your wording.

Answers written: 87 of 87.

Contents

1. Derive scaled dot-product attention and explain how its scale affects softmax gradients.

Knowledge · LLM fundamentals · Checklist

Attention is softmax(QKᵀ/√d_k)V, and the shapes carry the derivation. Q and K are batch × heads × n × d_k, so QKᵀ is an n × n score matrix per head; softmax over the key axis makes each row a distribution, and multiplying by V gives n × d_v, a weighted sum of values. The scale is one line: if q and k have independent zero-mean, unit-variance components, q·k sums d_k unit-variance terms, so its variance is d_k. Unscaled, logits spread as √d_k, softmax puts almost all mass on one key, and its Jacobian, diag(p) − ppᵀ, goes to zero, so gradients to Q and K vanish; dividing by √d_k restores unit variance. For causal attention I set future scores to minus infinity before softmax and subtract each row's maximum. I'd test against a naive implementation on random tensors and check that changing future tokens leaves earlier outputs intact. The scale only fixes variance at initialisation: if query and key norms grow in training, softmax saturates anyway, which is why some models normalise queries and keys.

Read: Attention Is All You Need

↑ Contents

2. Derive the memory required by a decoder's KV cache as batch size and context length grow.

Knowledge · LLM fundamentals · Checklist

With equal lengths the size is 2 × batch × layers × tokens × KV heads × head width × bytes per element. The two is keys and values; queries aren't kept, because the current step's Q is needed for exactly one iteration, while the keys and values of every past position are read again at each following step. The cache grows linearly in both batch and context length, so on long conversations it can limit concurrency beyond the weight budget. To that formula you always add weights, activations and working buffers, allocator headroom, and padding waste when lengths differ and memory is handed out as a rectangle; paged allocation, as in PagedAttention, reduces that fragmentation. I'd compute the budget over the real length distribution and reconcile it with measured memory under load. The formula ignores cache compression, offload and architectures that store the cache differently.

Read: Efficient Memory Management for Large Language Model Serving with PagedAttention · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

↑ Contents

3. Compare multi-head, multi-query and grouped-query attention in quality and serving memory.

Knowledge · LLM fundamentals · Checklist

All three keep every query head with its own attention pattern and differ only in keys and values: MHA has a K and V pair per query head, MQA one pair for all, GQA one per group. The cache shrinks by the ratio of query heads to KV heads: with 64 query heads and 8 KV heads it's eight times smaller than MHA, with MQA sixty-four. That pays off where KV reads dominate decode, at large batch and long context; at small batch and short prompts each step is bound by reading the weights, and fewer KV heads barely move latency. The price is quality: MQA loses more, GQA with a few groups stays close to MHA, and converting an MHA checkpoint takes mean-pooling KV heads per group plus uptraining on about five percent of the original pretraining. I'd measure concurrency at fixed memory, long-context decode throughput and quality against MHA. Under tensor parallelism MQA's single head is copied to every device, so GQA with one group per shard costs the same memory there and keeps more quality.

Read: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

↑ Contents

4. Under a fixed training compute budget, how do Chinchilla-style scaling results change the balance between model size and training tokens?

Knowledge · LLM fundamentals · Checklist

Chinchilla shows that under a fixed compute budget parameter count and training tokens have to grow together, whereas earlier practice took too large a model and too little data. The relationship is empirical, fitted over many runs of different sizes, so carrying its coefficients over to another architecture, tokeniser or corpus deserves caution. The main trade-off is that the training optimum is not the serving optimum: if the model will serve an enormous stream of requests, you're better off with fewer parameters trained longer on more tokens, paying in training time and winning on every request for years. I'd work out total cost of ownership, training plus projected inference over the model's lifetime, and separately check whether there's enough quality data without harmful duplication. The law says nothing about post-training, about limited corpora, or about whether the quality you need is reachable at all.

Read: Training Compute-Optimal Large Language Models

↑ Contents

5. How would you test and reduce a model's tendency to miss relevant evidence in the middle of a long context?

Knowledge · LLM fundamentals · Checklist

You test it by moving the evidence: take a question whose answer sits in exactly one passage and slide that passage through input positions, changing nothing else. A dip in the middle with identical content is the positional effect, not a shortage of context. Measure at several input lengths, with varying numbers of similar distractor passages, and separately for each model: the behaviour depends on training and on length, and window size on its own promises nothing. Three things soften the problem: retrieve more precisely so there's less text to read; rerank and place strong candidates nearer the edges; shrink the context by summarising or dropping what isn't needed. I'd put a positional slice into the regular eval set and recheck it on every model change. There's no universal cure: long context stays more expensive in money and latency, and it doesn't replace good retrieval.

Read: Lost in the Middle: How Language Models Use Long Contexts

↑ Contents

6. Contrast prefill with autoregressive decode and explain when compute or memory bandwidth becomes the bottleneck.

Knowledge · Inference, serving and cost · Checklist

Prefill computes all prompt positions in one pass and fills large matrix blocks, so unless the prompt is very short it's compute bound. Decode runs autoregressively, one token per iteration: every step has to read all the weights of a layer to do arithmetic over a single token per sequence, so at small batch the arithmetic intensity is low and memory bandwidth becomes the limit. Batching moves that boundary: the more sequences run together, the better the weights you've read are reused and the closer decode gets to the compute limit, but the KV cache grows and so does reading the cache itself. So before any optimisation I'd take a profile on the real distribution of prompt and output lengths and look per phase at achieved memory bandwidth and compute utilisation. The limit: at long contexts decode stays bandwidth bound however large the batch, because each sequence reads its own cache and that traffic isn't shared, so batching stops helping and only a smaller cache does.

Read: Optimizing inference

↑ Contents

7. Explain iteration-level batching and how requests enter and leave a running inference batch.

Knowledge · Inference, serving and cost · Checklist

The scheduler rebuilds the batch at every generation step instead of waiting for everyone to finish: a new sequence joins at the next iteration boundary, and one that emitted a stop token or hit its length limit leaves immediately and frees the slot. That way utilisation doesn't collapse because of one long answer. We pay with scheduler complexity and the risk that some requests get preempted or wait longer than the rest. What you cap is not the number of requests but the total tokens per step and the available KV memory, otherwise the cache pool runs dry and you start evicting and recomputing. I'd watch not only throughput but time to first token, the distribution of queue wait and the share of requests that waited past a threshold. A scheduler doesn't cure overload: with sustained excess demand you need rejection or degradation, not a cleverer queue.

Read: Efficient Memory Management for Large Language Model Serving with PagedAttention

↑ Contents

8. Explain exact speculative decoding, its acceptance step and workloads where the draft model adds overhead instead of speed.

Knowledge · Inference, serving and cost · Checklist

The scheme is this: a cheap draft model generates several tokens ahead, and the target model verifies them in one parallel pass, confirming several positions within its single step. Accuracy is preserved honestly: each proposed token is accepted by comparing the target and draft probabilities, and on rejection the token is resampled from a corrected distribution, so the final distribution matches the target model's. The gain is set by three quantities: the average share of accepted tokens, the cost of running the draft, and spare compute available for verification. I'd measure the average number of accepted tokens per pass and end to end latency with and without the draft model on my own workload, not on a demo example. Large compute-bound batches leave less spare compute; draft-target mismatch lowers acceptance. Either can erase the speedup or cause a slowdown.

Read: Fast Inference from Transformers via Speculative Decoding

↑ Contents

9. Estimate a serving memory budget for a 70-billion-parameter model, explicitly choosing precision, context and concurrency assumptions.

Applied scenario · Inference, serving and cost · Checklist

I would state a hypothetical architecture before budgeting: 70 billion parameters, 80 layers, eight KV heads and head width 128. BF16 weights occupy about 140 GB, which is about 130 GiB; int8 roughly 70 GB and four-bit roughly 35 GB, excluding quantisation metadata. For BF16 KV, bytes equal 2 times concurrency times layers times tokens times KV heads times head width times two. Choosing 8,192 tokens and four concurrent sequences gives 10 GiB of KV cache. On two hypothetical 80 GiB accelerators, tensor parallelism could split weights and KV heads, leaving roughly 9.8 GiB per device for activations, buffers, fragmentation and headroom. I would verify actual usable memory and peak allocations, then lower concurrency if needed. This is a capacity estimate, not evidence that the configuration meets a latency SLO.

Read: Efficient Memory Management for Large Language Model Serving with PagedAttention

↑ Contents

10. You are asked to reduce serving cost by an order of magnitude. Rank the levers and explain how you would test whether that target is achievable.

Applied scenario · Inference, serving and cost · Checklist

I'd rank levers by where the money goes, starting from a breakdown in cost per successful task at the required quality and latency SLO: input versus output tokens, the repeated share of input, and accelerator utilisation. Each output token needs its own decode step, so if output dominates, prompt caching barely helps; the levers are a smaller or distilled model for the traffic it can handle, and shorter answers. If long repeated prompts dominate, prefix caching and trimmed context come first. If utilisation is low, batching and right-sized capacity can beat any model change. Quantisation makes each step cheaper but removes no work, and its ceiling is a few-fold gain, not tenfold. Levers interact: a smaller model shrinks what caching and quantisation can still save, so multiplying separate gains overstates the total. I'd measure each lever alone, then prove the combined stack on replayed production traffic against the quality bar and p99. If cost is mostly output from a model that can't shrink without failing the bar, tenfold may not exist, and I'd say so upfront.

Read: Optimizing inference · Site Reliability Engineering

↑ Contents

11. After a deployment, p99 latency doubles although model weights are unchanged. How do you isolate the cause?

Applied scenario · Inference, serving and cost · Checklist

With weights unchanged I'd search around the model, starting with where slow requests spend their time: queue wait, prefill, decode and dependency calls, read from traces of tail requests before and after the release, because stage p99s don't add up to the end-to-end p99. Each component points somewhere different. Longer queueing means capacity fell or load rose: a lower batch limit, a missing replica or a new co-located workload pushes utilisation toward one, where waiting grows nonlinearly, so a small loss can double the tail. Longer prefill suggests a changed system prompt or template broke the shared prefix and collapsed the prefix-cache hit rate. Longer decode points to longer outputs or KV evictions forcing recomputation. Longer dependency time is a retrieval or tool tail. I'd also compare the traffic mix, since a new client brings longer prompts without any code change. Then one hypothesis at a time, confirmed by rollback or canary; a canary proves causality only under comparable traffic and neighbours, otherwise I'd replay recorded traffic against both versions.

Read: The Tail at Scale

↑ Contents

12. Choose a chunking approach for technical documentation, including how you preserve meaning across section boundaries.

Applied scenario · RAG and retrieval · Checklist

I'd split along the document's structure, because an embedding is one vector per chunk: a chunk that straddles two topics averages them and matches neither, and one cut mid-procedure loses what this flag or the step above refers to. The unit is a section; sections longer than a few hundred tokens split recursively at paragraphs with a small overlap, while code blocks and tables stay whole, a table keeping its header row. Each chunk carries its heading chain, document id and position, so it's intelligible away from its page and links back precisely. The trade-off is size: small chunks match precisely but lose context, large ones keep it but blur the vector and fill the prompt. My baseline is a fixed-size window with overlap, compared on representative questions with known source passages: recall at k for retrieval, then answer quality, with tables and answers spanning two sections checked separately. The boundary: this needs real structure, and on scanned PDFs or inconsistent headings I'd fall back to recursive size-based splitting.

Read: Contextual Retrieval in AI Systems

↑ Contents

13. When would you combine lexical and dense retrieval, and how would you merge their rankings?

Knowledge · RAG and retrieval · Checklist

It's worth combining when queries contain both exact strings and free wording. Lexical search is irreplaceable for identifiers, error codes, function names, versions and rare terms that an embedding doesn't distinguish; dense search pulls in paraphrases and synonyms where there's no word overlap at all. You can't add their scores together directly: a lexical score isn't normalised and isn't comparable with cosine similarity. It's more practical to fuse by rank, for instance reciprocal rank fusion, or to normalise within a single query, or to take the union of candidates and rerank with a cross-encoder, at which point the original scales stop mattering entirely. I'd test on a fixed query set with labelled relevance, switching off each branch and the reranking in turn to see what each part contributes. The price is a second index, an extra latency stage and cost. If queries are homogeneous, hybrid adds complexity without a gain.

Read: Contextual Retrieval in AI Systems

↑ Contents

14. How would you separately evaluate document retrieval and answer generation in a RAG application?

Applied scenario · RAG and retrieval · Checklist

For representative questions I label answer-bearing passages and define acceptable answers separately. Retrieval is then measured by its own metrics, whether the needed passage appeared in the results and at what rank, and generation by groundedness, whether every statement rests on the context it was given, and by factual correctness. The key move is running generation with the gold passages as input: if the answer is right with them, retrieval or context assembly needs investigation; if not, the problem is the model, the prompt or the context format. Without that split, an end to end metric mixes two different defects and the team fixes the wrong stage. I'd keep the set small but alive, topping it up with real production failures. Limits: relevance labelling is subjective, gold passages don't reproduce the noise of real retrieval, and a judge model itself needs checking against human agreement.

Read: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · Demystifying evals for AI agents

↑ Contents

15. Design retrieval that enforces the source system's access rights, including permission changes and shared caches.

System design · RAG and retrieval · Checklist

The source system defines document permissions; retrieval must enforce them before content reaches the model. I pass the user's identity into search and filter candidates by access, then check authoritative permissions if indexed ACL freshness cannot be established. Post-filtering can damage recall, and applying controls only after generation is too late. Permission changes and deletions need measurable propagation, revoked grants must invalidate caches, and cached answers must respect the current caller's access. A shared embedding cache need not be per user if it cannot expose restricted content and retrieval remains authorised. I test revoked access, deleted documents and users without rights, including citations, suggestions and logs. Stale content can be labelled; uncertain authorisation must fail closed rather than merely display a warning.

Read: Document-level access control

↑ Contents

16. Compare exact search, HNSW and IVF-PQ for an embedding index under memory, recall and latency constraints.

Knowledge · RAG and retrieval · Checklist

I'd start with exact search over a sample of queries: it gives the baseline against which the word recall means anything at all. HNSW builds a multi layer graph of neighbours and descends it greedily: search is fast and accurate and inserts need no retraining, but deletes are usually only marked, and the graph lives entirely in memory and costs dearly there. IVF-PQ first divides the space into coarse clusters and looks at only the nearest few, storing vectors compressed by product quantisers: it cuts memory by an order of magnitude or more, but quantisation itself loses accuracy and the clustering has to be trained on representative data. I'd choose by measuring recall against the exact baseline at the target latency, and separately measure memory footprint, the cost of updates and deletions, and behaviour under metadata filters. A small collection needs no approximate index at all, and no index will fix bad embeddings.

Read: Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs · Faiss indexes

↑ Contents

17. What does interleaving reasoning with tool observations add to a language-model agent compared with reasoning alone?

Knowledge · Agents, tools and protocols · Checklist

Reasoning alone can't check its own premises: if the model assumes a wrong fact in step two, every later step builds on it, and the chain gets more fluent, not more correct. Interleaving adds the one channel that can contradict the model. It states a thought, takes an action, and the observation that comes back, not the plan, decides the next step, so an empty search or an unexpected value forces a revision before the final answer. The reasoning, in turn, tracks progress and chooses the next action, which acting alone lacks. The cost is latency, another model call and more context per round, plus tool errors, empty results and junk the model must not mistake for fact. I'd run the same tasks three ways, reasoning only, acting only and interleaved, and compare success rate, cost per task and how often the agent recovers after a failed call. The boundary: when the model already knows the answer, uninformative results can derail it, and a silently wrong tool makes it confidently wrong.

Read: ReAct: Synergizing Reasoning and Acting in Language Models

↑ Contents

18. Design recovery from tool errors and timeouts, including cases where a timed-out call may already have changed external state.

System design · Agents, tools and protocols · Checklist

Before retrying, I classify the outcome: transient failure, permanent failure or ambiguous completion. A timeout on a write is ambiguous because the operation may still be running or already committed. For writes, I persist a client-generated idempotency key with the task and reuse it under the provider's documented guarantees. Without that guarantee, a state read alone is insufficient: it may be stale while the original operation remains in flight. I require an authoritative terminal outcome or reconciliation; otherwise I stop automated retries. Safe retries have an attempt budget, overall deadline and exponential backoff with jitter. When that budget expires, I escalate with the operation history and unresolved status. Idempotency depends on provider semantics, retention and key scope; merely attaching a key does not make a write safe.

Read: Designing robust and predictable APIs with idempotency

↑ Contents

19. How would you select, name and document tools so that a model chooses the right operation and arguments?

System design · Agents, tools and protocols · Checklist

The model chooses a tool only from its name, description and schema, so every pair of overlapping tools is a decision it can get wrong, and every free-text argument is a place to invent a value. I'd shape tools around the agent's tasks, not the API's endpoints: a search instead of a list that dumps every record, names prefixed by service, enumerations wherever options are finite, and a strict schema so validation rejects a bad argument before the downstream system sees it. The real trade-off is granularity: a few high-level tools cut selection errors but make arguments complex, while many thin ones keep arguments simple but make selection ambiguous and chains longer. Results are paginated and capped, and an error says what to change. On a fixed set of realistic tasks I'd measure the share of calls with the right tool and valid arguments and the tokens per task, before and after each merge or rename. The boundary: once the set grows into dozens of tools, selection degrades even with good descriptions, so I'd load tools per task.

Read: Writing effective tools for AI agents—using AI agents

↑ Contents

20. Define completion and stopping conditions for an agent loop so it cannot spend indefinitely on an unfinished task.

System design · Agents, tools and protocols · Checklist

The loop has to end in an explicit terminal state, and there are several: success, blocked, cancelled by the user, and budget exhausted. They cannot be merged, because the response differs: success closes the task, blocked needs a human, and budget exhausted says honestly that we never learned whether the task was doable. I define progress from observable state in the environment, not from the model's confidence, and alongside it I keep hard limits on steps, money and time; without them an agent will cycle through options forever on an unsolvable task. The cost is obvious: some genuinely long tasks get cut off, so on stopping you save state, what was done and which side effects are already applied, so a resume or handoff does not repeat them. The limit: budgets bound spend, not quality, and an agent can stop on time with a formally complete but useless result.

Read: Building Effective AI Agents

↑ Contents

21. Design human approval for consequential agent actions, including how approval remains bound to the exact action being executed.

System design · Agents, tools and protocols · Checklist

Approval works only if the person sees the concrete action, not the intent: recipient, amount, object and how reversible the consequences are, exactly as they will go to the external system. So permission binds not to the conversation or the session but to the validated arguments of one specific call: we freeze them, show them, and at execution recheck the initiator's rights and that fields are unchanged. If the model changed even one material argument after approval, the old permission is void and a new one is requested. I make execution idempotent so a retry after a network failure is not a second charge, and I log the decision, its author, time and result. The limit: this protects against mistaken and externally prompted actions, but under a stream of requests it degenerates into a formality: people approve without looking, so only actions whose cost of error is real deserve it.

Read: Building Effective AI Agents · Designing robust and predictable APIs with idempotency

↑ Contents

22. Walk through a preference-based RLHF pipeline and explain the roles of the reward model and reference-policy penalty.

Knowledge · Fine-tuning and post-training · Checklist

RLHF splits into three stages that differ by data. First demonstrations: examples of desired answers, on which the base model is finetuned with supervised learning. Then preference pairs: a person sees several answers to one prompt and ranks them, and on those comparisons you train a reward model, a separate model predicting which answer a person picks. In the third stage the policy is optimised against that reward on its own rollouts, with a penalty for diverging from the reference model added to the objective, usually a KL divergence; without it the policy drifts to where reward is high and the text has degraded. That is the trade-off: push harder on reward and you reach its overoptimisation sooner. Check on held-out human preferences, not on the reward itself, and watch how divergence from the reference tracks the real gain. The reward model inherits annotator bias, and optimisation amplifies it.

Read: Training language models to follow instructions with human feedback

↑ Contents

23. Compare DPO with PPO-based RLHF, including the assumptions behind offline preference learning and reasons to collect fresh rollouts.

Knowledge · Fine-tuning and post-training · Checklist

DPO starts from the same KL-regularised objective as PPO-based RLHF but uses its closed-form optimum: the reward becomes implicit, β times the log ratio of policy to reference probability, plus a term depending only on the prompt. In the Bradley–Terry preference model that term cancels, so the loss is written directly on the policy: a logistic loss on β times the gap between chosen and rejected log ratios. β plays the KL penalty's role, and there's no reward model, value network or sampling loop. The price is an assumption: fixed pairs must cover what the policy will generate, since the loss shapes only responses in pairs and can lower the likelihood of both answers or drift toward length. PPO scores fresh rollouts from the current policy, catching errors where they occur, at the cost of a reward model that can be exploited. I'd compare held-out win rate against the reference, KL and length drift. A simpler loss isn't better alignment: when the change needed is large or the reward is verifiable, fresh rollouts earn their cost.

Read: Direct Preference Optimization: Your Language Model is Secretly a Reward Model

↑ Contents

24. Derive the low-rank weight update used by LoRA and propose an experiment for choosing its rank.

Knowledge · Fine-tuning and post-training · Checklist

LoRA freezes the original weight matrix and trains a low-rank correction instead: ΔW = BA, where B is d by r, A is r by k, and r is much smaller than both d and k. Instead of d times k trainable parameters you are left with r times the sum of d and k, orders of magnitude fewer at small rank, and at inference the correction can be merged into the base weights with no extra latency. The constraint is that rank sets capacity: too small an r simply can't hold the behaviour change you need. The experiment is straightforward. Fix the data and the set of target modules, vary only r over a grid, and compare quality on held-out tasks while checking for regressions in general capability. Merging the adapter is convenient, but it gives up keeping several adapters on one base and switching them per request.

Read: LoRA: Low-Rank Adaptation of Large Language Models

↑ Contents

25. How does QLoRA reduce finetuning memory, and which tensors remain trainable or require higher precision?

Knowledge · Fine-tuning and post-training · Checklist

QLoRA saves memory by keeping base weights quantised to four bits and frozen while only low-rank adapters, kept in 16-bit, are trained: gradients flow back through the dequantised base weights, but weight gradients and optimiser states exist for the adapters alone, and those were exactly what ate most of the memory in full finetuning. Three mechanisms work together: NF4, a quantisation format designed for normally distributed weights; double quantisation, which compresses the quantisation constants themselves; and paged optimisers, which move optimiser states into host memory during spikes. You pay in compute, because weights are dequantised to the compute precision on every matrix multiply, which can slow training depending on implementation. Activation memory does not go anywhere and grows with sequence length and batch size, so on long context it becomes the binding constraint again. The limit: this is a way to fit finetuning into less memory, not a way to guarantee full-finetuning quality on any task.

Read: QLoRA: Efficient Finetuning of Quantized LLMs

↑ Contents

26. Choose between prompt changes, retrieval and finetuning for a failing AI feature; include data freshness, latency and total cost.

Applied scenario · Fine-tuning and post-training · Checklist

I'd let a diagnostic pick the intervention: rerun a sample of real failures with the right document pasted into the prompt. If they now succeed, the model lacked knowledge, and retrieval fits: facts stay fresh without retraining and answers can cite sources, paid for with a search hop in latency and an index to keep current. If they still fail with the evidence in front of the model, it's instructions or behaviour, so I tighten the prompt with explicit rules and examples; that's cheap and becomes the baseline everything else must beat. Finetuning is for what persists after that, a format, tone or decision pattern the prompt can't hold reliably, once I have enough representative examples to train and still hold some out. Its cost is the lifecycle: labelling, training, evaluation and redoing it when the base model changes, though shorter prompts can cut per-request tokens and latency. I'd compare the options on one held-out set by cost per successful task. Finetuning won't keep facts fresh, and retrieval can't fix a model that ignores what it found.

Read: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · LoRA: Low-Rank Adaptation of Large Language Models

↑ Contents

27. Design an LLM judge and a calibration process that exposes order, verbosity and self-preference biases.

System design · Evaluation and observability · Checklist

A judge starts not with a prompt but with criteria: what counts as a good answer in this task, written so that two people would agree. Then you need a held-out calibration set with human labels; it measures the judge itself, otherwise we are simply trusting a model to grade a model. The known biases are checked by experiment: run each pair in both orders and see whether the verdict flips; pad answers with repetition that adds no content and see whether they start winning, to catch a reward for verbosity; compare how often the judge prefers its own family's answers with how often people do on the same pairs, since a judge can favour its own style even when authorship is hidden. I would track agreement between judge and people, the share of pairs where swapping order changes the decision, and send unreliable cases to experts. The limit: a judge measures conformance to criteria, not usefulness to the user, and wherever an answer can be checked deterministically an ordinary check is cheaper.

Read: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

↑ Contents

28. Design a release gate for prompt and model updates, including what happens when aggregate gains hide a critical regression.

System design · Evaluation and observability · Checklist

I'd version everything, prompt, model and parameters, dataset and grader code, and compare the candidate with a pinned baseline on the same items. The rule isn't one average: the aggregate must not drop, and each critical slice, safety, money, rare but expensive cases, has its own rule. The hard part is statistics on small slices. With 50 items at a 90 percent pass rate, the 95 percent interval is about plus or minus eight points, so a three-point regression is invisible. So I compare paired, item by item, with several samples per item, and count items that flipped from pass to fail; for the most severe failures I keep must-pass items where any flip blocks. Thresholds and severity weights are fixed before the run, because a strict gate rejects good releases and a loose one ships regressions, and that trade is a product decision. Then a canary on a small share of traffic with numeric rollback triggers. The boundary: the gate only sees failure modes the dataset covers.

Read: Demystifying evals for AI agents

↑ Contents

29. Benchmark results improve but users report a worse product. What hypotheses would you test first?

Applied scenario · Evaluation and observability · Checklist

I'd order the hypotheses by how cheaply each can be ruled out. First, causes outside the model: latency, answer format, truncation or a changed system prompt, visible in logs within a day. Second, a mix mismatch: classify a sample of live traffic by task type, language and length, and re-weight the benchmark gain by that mix; if the gain vanishes, the model improved where users aren't. Third, a regression on a slice the average hides, from per-slice deltas. Fourth, the direct test of the complaint: a blind side-by-side of the old and new model on sampled production prompts, judged by people or by a judge calibrated against them. If the new model wins there, the complaints are about something else, often changed expectations. Fifth, contamination: n-gram overlap with training data, and reworded benchmark items, where a real gain survives rewording. Reproduced complaints become test cases, but I'd hold part of them out, because optimising straight at them detaches the metric from usefulness again.

Read: Rules of Machine Learning

↑ Contents

30. Design observability for a production LLM workflow: which spans, versions, costs and feedback should be linked?

System design · Evaluation and observability · Checklist

Every user task gets one trace id that all spans attach to, model calls, retrieval and tool calls, each tagged with the prompt, model, index and code versions, so a trace answers not only what broke but which change broke it. Each span records latency, tokens, cost, outcome and error code, and user feedback carries the same id. Content is the sensitive part, so structure and metrics are kept for every task, while full payloads are kept by tail-based sampling: every failed task, low judge score or thumbs-down, plus a small fixed share of successes as a baseline, with masked fields and short retention. Then a failure reads directly: a retrieval span that returned nothing followed by a confident answer goes into an ungrounded-answer bucket, not a model bucket. I'd measure the share of failures attributed to a stage, time to diagnose and payloads that escaped masking, and recurring buckets become eval cases. The boundary: a wrong but fluent answer looks like a success, so its payload survives only if feedback or a judge flags it.

Read: Site Reliability Engineering · Demystifying evals for AI agents

↑ Contents

31. How should evaluation of a tool-using agent differ from grading one generated response?

System design · Evaluation and observability · Checklist

The main difference: an agent has a result in the environment, and that is what you check, not the text in which it reported success. Saying it is done costs nothing, so evaluation reads the final state: was the record created, was the mail sent, what status is the ticket in. The second dimension is the trajectory: a task can be completed at the cost of extra calls, wasted money, or a dangerous action like deletion, and that is graded separately from completion, otherwise a bad path is masked by a good outcome. The third is repeatability: one lucky run proves nothing; you need several trials from the same clean environment state and a measured success rate, because the agent is non-deterministic. The limit: all this needs isolated environments with reproducible state; they are expensive and always poorer than reality, so some failure modes appear only in production.

Read: Demystifying evals for AI agents

↑ Contents

32. Compare projection layers, cross-attention and token-based integration for giving a language model access to images.

Knowledge · Multimodal and voice · Checklist

I'd separate the encoder that turns pixels into features from the interface that fuses them with the language model; the three options differ in the interface. A projection maps each patch feature into the model's embedding space, and the results sit in the sequence like text: cheap to train, decoder unchanged, but tokens scale with resolution, and a 336 × 336 image at 14-pixel patches is 576 tokens of context, prefill and KV cache. Cross-attention, as in Flamingo, keeps features outside the sequence, compresses them to a fixed set of vectors and inserts gated cross-attention layers into a frozen model, gates starting at zero so text behaviour is initially untouched; the sequence stays short, but new layers need training. Discrete tokens quantise the image into a learned codebook added to the vocabulary, so the model can also generate images, but quantisation loses fine detail. All three need alignment training on image-text pairs. I'd compare them on text-in-image and spatial tasks, plotting accuracy against visual tokens per image: on dense documents, fixed compression drops exactly what matters.

Read: Flamingo: a Visual Language Model for Few-Shot Learning

↑ Contents

33. How would you improve and evaluate speech recognition for mixed-language utterances and different accents?

Applied scenario · Multimodal and voice · Checklist

I'd start with evaluation, because an average WER hides exactly the groups this work is for. I'd build slices by language, accent, noise and code-switching, each with enough distinct speakers, and report WER per slice with confidence intervals bootstrapped over speakers, since one speaker's utterances are correlated. Code-switching needs its own care: a model like Whisper predicts one language token per segment and decodes conditioned on it, so a switched phrase can come out transliterated or even translated, and scoring is ambiguous too, because a loanword written in the other script counts as an error unless normalisation rules settle it. Beside WER I'd track errors in names, amounts and other entities an action depends on, since one wrong digit outweighs many small slips. The cheap lever is biasing decoding with a term list or prompt, which helps only known words; fine-tuning on accented and code-switched audio reaches further but can regress other languages. I'd validate any adaptation on held-out speakers and domains, and treat a slice with only a few speakers as no evidence either way.

Read: Robust Speech Recognition via Large-Scale Weak Supervision

↑ Contents

34. Design a knowledge assistant over ten million enterprise documents with per-user permissions and a continuously changing corpus.

System design · AI system design · Checklist

I'd pin down query volume, freshness and permission semantics, then size. At an assumed twenty chunks per document that's two hundred million vectors, about 800 GB as 1,024-dimensional float32, so the index is sharded or product-quantised, with recall checked against exact search. Ingestion is event-driven: changed documents are re-chunked and re-embedded, deletions tombstoned at once, behind a versioned index. The decisive point is that permissions must propagate faster than reindexing. ACLs live on each chunk as group ids, updated without re-embedding, and the user's groups are resolved at query time, so a membership change applies on the next query. The filter belongs inside the ANN query: post-filtering a top-k can leave almost nothing for a user who sees a small slice of the corpus, while filtering during graph search keeps recall until the allowed set gets tiny, where I switch to exact search over that subset. I'd measure revocation lag at p99, filtered recall and answer grounding, and show a shard outage as partial coverage. Stale text can be shown with its date; uncertain authorisation fails closed.

Read: Document-level access control · Contextual Retrieval in AI Systems

↑ Contents

35. Design an LLM gateway with provider routing, failover, caching, rate limits and enforceable spending budgets.

System design · AI system design · Checklist

A gateway exists for one contract over different providers, so first I describe their capabilities, streaming output, tool calling, context limits, behaviour on failure, and unify only genuinely compatible semantics; the rest I expose explicitly rather than emulate, otherwise the client quietly gets different behaviour. Quotas and budgets I check before dispatch, per client, and they must account for retries, or failover eats the budget itself; when a budget is exhausted I return a clear error saying when to try again. I cache whole responses only where returning the same answer twice is acceptable, and the key includes the client, the model and its parameters, otherwise the cache serves another client's or another model's answer. The subtlest part is the failover boundary: once part of a response has gone to the client you cannot switch providers without breaking the stream, and a side-effecting call cannot be repeated without idempotency. The limit: a gateway levels interfaces and failures, but not the quality gap between models.

Read: Scaling your API with rate limiters · Designing robust and predictable APIs with idempotency

↑ Contents

36. Design a customer-support agent that can execute service actions and transfer the case to a human when needed.

System design · AI system design · Checklist

I'd make authority a property of the tools, not of the prompt. Each supported intent maps to a tool whose backend enforces the limits: the agent reads order status freely, issues a refund only below a threshold and only to the verified customer, and anything else isn't callable at all. Duplicates are prevented by the case record, not the model's memory: each action is written there as an intent with an idempotency key and a status before it executes, and the agent reads the record before acting, so a retry or a resumed conversation can't refund twice. Escalation triggers are observable: the customer asks for a person, the request is out of scope, a tool returned an ambiguous outcome, or the same issue comes back. The handoff carries the history, the actions and their results. I'd compare against human-only handling on resolution without a human, wrong-action rate, repeat contacts and escalation precision. The real boundary is the threshold: lower it and more cases escalate, raise it and the agent's mistakes become direct losses.

Read: Building Effective AI Agents · Writing effective tools for AI agents—using AI agents

↑ Contents

37. Design a streaming chat service for hundreds of millions of users, including capacity, conversation storage and graceful overload behaviour.

System design · AI system design · Checklist

KV-cache memory, not compute, caps concurrent streams, so I'd size from that. Suppose 100 million daily users, ten replies each, 20 seconds of streaming per reply: about 230,000 concurrent streams on average, say half a million at peak. A 70B-class model with 80 layers and eight 128-wide KV heads needs a third of a megabyte of bf16 cache per token, so a 4,000-token conversation holds 1.3 GB, and an assumed 80 GB of free cache per replica fits about 60 streams, far too few to saturate compute. That's roughly 8,000 replicas before headroom. So admission counts free cache blocks: a request starts only when its expected length fits, otherwise it waits in a bounded queue or is rejected at once with a retry hint. A closed connection cancels generation and frees its blocks. Conversations, two billion messages a day, go to an append-only store partitioned by conversation, so a dropped stream resumes from the last saved message. The boundary: with short chats or a quantised cache, decode throughput becomes the limit and admission counts tokens per step.

Read: Site Reliability Engineering · Efficient Memory Management for Large Language Model Serving with PagedAttention

↑ Contents

38. Implement scaled dot-product attention with a causal mask and tests that would expose attention to future tokens.

Coding · Practical coding · Checklist

I'd start with an explicit contract on dimensions: batch, heads, length, head width, with a shape check at the entry, because most bugs here are swapped axes and silent broadcasting. Then scores as Q times K transposed on the last axis, division by the square root of head width, minus infinity at future positions j > i, allowing the diagonal and past, a stable softmax that subtracts the row maximum, and multiplication by V. The mask has to come before softmax: zeroing the weights after normalisation is wrong, the sum is spoiled by future positions. I'd write two kinds of test: a tiny example with numbers worked out by hand, and a causality test that changes tokens after a position and confirms the outputs before it didn't move. A causal mask alone never empties a row, since the diagonal is allowed, but with a padding mask a row can be fully masked and softmax gives NaN, so I'd test that case. This is a teaching implementation: no memory saving like flash attention, and no inference caching.

Read: Attention Is All You Need

↑ Contents

39. Implement a token-bucket limiter and explain what must change when several workers enforce the same limit.

Coding · Practical coding · Checklist

The bucket stores a token count and last refill time. On each call I measure elapsed time on a monotonic clock, add elapsed time times the refill rate, clip at capacity, and deduct what was asked for if there's enough. Capacity sets the burst you allow and the rate sets the average pace, the main trade-off. Avoid wall-clock time: a clock change or a sync will give a negative interval or hand out a full bucket. Check and deduct must be one atomic operation under a lock, otherwise two threads see the same last token. I'd test a burst on a full bucket, refusal on an empty one, and recovery after the right interval, injecting a fake clock instead of waiting. A shared limit across workers needs an external counter with a script that refills and deducts atomically, or the quota split per worker: cheaper, less exact at the boundaries.

Read: Scaling your API with rate limiters

↑ Contents

40. Write an asynchronous API batch processor with bounded concurrency, retry jitter and isolated per-item failures.

Coding · Practical coding · Checklist

I'd use a fixed pool of workers pulling from a queue, not one task per item behind a semaphore, so a million items don't mean a million pending tasks. Each item runs in a wrapper that turns its own failures into a result, value, permanent error or cancelled, stored by input index, so one failure can't sink its siblings, unlike a task group that cancels everything on the first error. Each attempt has a timeout inside an overall deadline, and cancelling the batch cancels in-flight requests too, so the wrapper mustn't swallow cancellation. I retry only timeouts, connection errors, rate-limit responses and server errors, with capped exponential backoff and full jitter, a uniform random sleep up to the current backoff, so clients don't retry in waves, and a Retry-After header wins. Unsafe writes retry only with an idempotency key. Tests use a fake clock and API: partial success, a permanent error, the deadline firing mid-retry and cancellation halfway. The boundary: per-item backoff doesn't bound the aggregate rate, so against a shared quota I'd put a token bucket in front.

Read: Designing robust and predictable APIs with idempotency · Site Reliability Engineering

↑ Contents

41. How does sparse expert routing increase parameter capacity, and what costs remain despite activating only some experts?

Knowledge · LLM fundamentals · Checklist

Routing buys capacity, not free scale: the model holds far more parameters while any one token is computed by only a few of them. A router scores experts per token and sends it to the top few, so arithmetic grows with the experts activated rather than the total, and quality improves as experts specialise. Memory does not shrink: every expert's weights stay resident, and in distributed serving tokens travel to the device holding their expert, putting an all-to-all exchange on the critical path. I would measure tokens per expert to expose imbalance, the share dropped when an expert hits its capacity limit, and the all-to-all share of step time, judging quality against a dense model of the same active size. The argument holds only while routing stays balanced and batches are large enough to amortise that exchange.

Read: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

↑ Contents

42. Explain how latent attention compresses the KV representation and what must be reconstructed at inference.

Knowledge · LLM fundamentals · Checklist

Latent attention changes what you store, not how precisely you store it. Keys and values are projected down to a single low-rank latent vector per token, and that latent is what the cache holds; during attention it is projected back up to per-head keys and values, or the up-projections are folded into the query and output projections so attention runs on the latent directly, with the positional part cached as a small separate key. Cache bytes per token therefore fall by about the compression ratio, paid for with extra arithmetic on every decode step and more intricate kernels. I would measure cache bytes per token, the context length and batch size reachable within the same memory, decode throughput, and quality against a grouped-query baseline at an equal cache budget. The distinction matters because quantisation lowers the precision of the same tensors: the two compose, so each needs its own baseline or one saving gets counted twice. The gain is real only when the KV cache, not the weights, is what bounds you.

Read: DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

↑ Contents

43. What makes a prompt prefix reusable, and which changes invalidate or reduce the value of a prefix cache?

Knowledge · Inference, serving and cost · Checklist

Reuse is exact and token-level, not semantic: the cache holds attention state for a prefix, and a request reuses it only while its tokens match from position zero under the same model version, precision and parallel layout. One changed character near the start invalidates everything after it, which dictates prompt layout — system rules, tool definitions and long documents first; the user turn, timestamps, retrieved snippets and per-request identifiers last. Cache memory competes with the KV cache of live requests, and entries expire, so I would measure hit rate, the share of prefill tokens served from cache, time to first token with and without it, and cost per request. Two limits: caching removes prefill work only, so a short-prompt long-answer workload gains almost nothing, and entries must be scoped per tenant, because a shared cache turns time to first token into a side channel: a fast response confirms that someone else sent the same prefix, which lets an attacker test guesses about another tenant's prompt.

Read: Prompt caching

↑ Contents

44. How does paging the KV cache reduce waste, and what remains for the scheduler to solve?

Knowledge · Inference, serving and cost · Checklist

Paging attacks fragmentation, not scarcity. If each request reserves one contiguous span sized for the longest answer it might produce, most of that reservation sits idle, and the free memory left over is unusable because it is scattered, so concurrency collapses well before the device is full. Paging stores the cache in fixed-size blocks with a per-request table mapping logical positions to physical blocks, so growth is incremental, internal waste is bounded by one partly filled block, and identical prefixes or parallel samples can point at shared blocks copied only on write. What it leaves untouched is policy: who is admitted when blocks run out, whose blocks are evicted or recomputed, and how long a request may wait. I would measure achieved batch size, the share of memory actually holding live tokens, preemption rate and p99 latency — paging widens the pipe, and a poor scheduler can still starve requests.

Read: Efficient Memory Management for Large Language Model Serving with PagedAttention

↑ Contents

45. A service can spend more on one reasoning run or generate several candidates and verify them. How would you allocate compute under a latency deadline?

Applied scenario · Inference, serving and cost · Checklist

I would treat this as an empirical allocation question, not a preference. On the same held-out tasks I would run one long reasoning pass and a sample-and-verify strategy at several budgets, then compare success per unit of spend and per unit of wall-clock time, counting the verifier's own tokens and latency as part of the cost: a verifier that doubles latency can lose on a deadline it would win on cost. Sampling parallelises, so it often fits a deadline that a single long pass misses, but it only helps when a cheap check can tell candidates apart; without that, extra candidates add spend and no accuracy. Because the gain varies with difficulty, small where one pass already succeeds or nothing does and largest in between, I would route by an observable feature available before the answer exists, never by correctness. Then a hard token and time ceiling with a defined fallback, reporting spend per successful task and deadline misses. Where no strategy succeeds at any budget, extra test-time compute is wasted and a stronger model is the lever.

Read: Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters · Evaluation best practices

↑ Contents

46. Where should a cross-encoder reranker sit in a retrieval pipeline, and when does its quality gain justify latency?

Knowledge · RAG and retrieval · Checklist

A cross-encoder belongs after a cheap retriever and before generation, scoring query-document pairs for a shortlist rather than the corpus: it reads the query and passage together, which is what makes it more accurate than comparing two independent embeddings, and also what makes it too slow to run over everything. The pipeline's ceiling is therefore the first stage's recall — a reranker cannot promote a document that was never fetched, so I measure recall at the candidate depth separately from final answer quality, otherwise a first-stage miss looks like a reranker failure. Tuning is a sweep: candidate count and final context size against held-out answer quality and end-to-end latency, watching the point where extra candidates stop changing the top results. The gain justifies the latency when relevance is genuinely the bottleneck; if answers fail because the corpus lacks the fact, reranking buys nothing.

Read: Contextual Retrieval in AI Systems

↑ Contents

47. How would you connect each factual claim in an answer to the retrieved passage that supports it?

System design · RAG and retrieval · Checklist

The point is that attaching a source is not evidence of support, so I design for claim-level attribution and then verify it. Generation emits each factual sentence with the identifier of the passage it used, ideally with character offsets, which keeps the binding checkable instead of a bibliography appended at the end. A separate verification step then asks whether the cited passage actually entails the claim — an entailment model or a second-pass judge, calibrated against a human-labelled sample, because a judge that rubber-stamps is worse than no check. Claims that fail get dropped, marked unsupported, or trigger abstention, and I measure the share of sentences with a verified citation, precision of citations on a labelled set, and how often the system abstains. The limit is that entailment only proves the passage says it; if retrieval surfaced a stale document, a perfectly supported claim can still be false.

Read: Patterns for Building LLM-based Systems & Products

↑ Contents

48. What does MCP standardise between an AI application and an external service that a model's function-call format does not?

Knowledge · Agents, tools and protocols · Checklist

A function-call format describes only the shape of the arguments a model emits. It says nothing about where the tool lives, how the application learns the tool exists, or how the result comes back. MCP standardises that surrounding layer — a client-server handshake, capability discovery, and typed primitives for tools, resources and prompts over a defined transport — so the same server works with any compliant application, within the capabilities both sides declare at the handshake, and a tool can be added without changing the model or redeploying the client. Execution and authority stay outside the model: the host application still decides which servers are connected and what each may do. I would judge it by how much integration code disappears per new tool. The trade-off is the trust boundary it creates: a connected server contributes tool descriptions and returns content, both of which reach the context as untrusted input, so a third-party server is a supply-chain and prompt-injection surface.

Read: Model Context Protocol specification · Security Best Practices

↑ Contents

49. Design memory for an agent that works across many sessions: what is persisted, retrieved, revised and forgotten?

System design · Agents, tools and protocols · Checklist

I'd keep two stores with different lifecycles: the session's working context, filled and discarded, and durable memory, where only confirmed things graduate: stable preferences, decisions with their rationale, task outcomes. The real choice is what gets written. I extract at write time into structured records, subject, attribute, value, source and timestamp, rather than embedding raw transcripts, because a keyed record can be superseded when the user changes their mind, while two contradictory transcript chunks both stay retrievable. The cost is lost detail, so raw episodes are kept briefly as a fallback, not as memory. Retrieval returns a handful of records relevant to the current task, never the whole history, which would recreate the context problem. Forgetting is explicit: expiry for time-bound facts, supersession by key, deletion on request. I'd replay multi-session tasks with memory off and on, measuring success, the share of recalled records actually used and errors caused by stale ones. The boundary: memory the model inferred rather than was told reinforces its own mistakes, so inferences get lower trust or aren't stored.

Read: Effective context engineering for AI agents

↑ Contents

50. When does splitting work among agents improve results, and when do coordination costs dominate?

Knowledge · Agents, tools and protocols · Checklist

Splitting helps when the work genuinely decomposes: subtasks that can run without seeing each other's intermediate state, and whose results a synthesis step can combine — broad search over many sources is the clean case, because breadth is parallel and each branch is independent. It stops helping when agents must agree as they go. Every handoff is a lossy summary through natural language, so shared mutable state produces conflicting edits, duplicated effort and contradictory findings that someone has to reconcile. I would define delegation boundaries and completion criteria per subagent, then make synthesis an explicit step that surfaces disagreement rather than averaging it away. The honest comparison is against one agent given the same total token budget, counting duplicate work, coordination failures, cost and wall-clock latency. Multi-agent often wins latency and loses cost, and if a task is inherently sequential, coordination overhead is pure loss.

Read: How we built our multi-agent research system

↑ Contents

51. How would you measure unsupported claims in a deployed RAG application without treating every fluent answer as correct?

System design · Evaluation and observability · Checklist

Fluency is not the signal, so I measure faithfulness against the evidence the system actually retrieved, not against the world. The unit is a claim: decompose the answer into separate statements and label each supported, contradicted or unsupported by the retrieved passages. That judgement scales with an LLM judge, but only after I calibrate it against a human-labelled sample, because an uncalibrated judge inherits the same plausibility bias I am trying to detect. Attribution matters as much as the rate: if the supporting passage was never retrieved, that is a retrieval defect, and improving the prompt will not fix it. On live traffic I would sample by segment rather than uniformly, since failures cluster in thin or ambiguous parts of the corpus. The limit is that a claim faithful to a retrieved passage can still be wrong if the source is stale, so faithfulness measures grounding, not truth.

Read: Your AI Product Needs Evals

↑ Contents

52. A reasoning summary looks sound, but the answer follows a misleading hint. What should an evaluation and monitoring system conclude?

Applied scenario · Evaluation and observability · Checklist

The conclusion is that the explanation is not evidence about the computation. A summary that reads well while the answer tracks a planted hint tells you the account is unfaithful, not that the model reasoned well and slipped. So the system should score what it can observe — final answers, tool calls, actions taken — independently of how coherent the narration looks, and never let a plausible summary raise confidence in a wrong result. The diagnostic I would build is paired: run otherwise identical tasks with and without the misleading hint, then measure two things separately, how often the answer moves with the hint, and how often the explanation admits the hint influenced it. A wide gap is the faithfulness signal. In monitoring, treat traces and summaries as useful, incomplete hints for triage. They are not a window into hidden reasoning, so a clean trace is not a safety guarantee.

Read: Reasoning Models Don't Always Say What They Think

↑ Contents

53. Design a secure developer API for model access, including authentication, quotas, streaming responses and backwards compatibility.

System design · AI system design · Checklist

This endpoint is a contract other people build on, so every decision must survive clients I cannot upgrade. Authentication is per key with scopes and rotation; quotas meter tokens and concurrency, not just request counts, because two requests can differ in token cost by orders of magnitude and a long stream holds capacity for its whole duration; a rejection returns remaining budget and a retry hint rather than a bare error. Streaming needs its own contract: incremental events carrying a request id, an explicit terminal event, and defined behaviour when the stream dies mid-answer, since a truncated reply is not safe to retry blindly. Compatibility comes from additive change — new optional fields, versioned model names, deprecation windows backed by usage telemetry. The hard part is that model behaviour is not versionable the way a schema is: identical requests drift over time, so the promise must cover the interface and documented limits, never identical output.

Read: Scaling your API with rate limiters

↑ Contents

54. Build an end-to-end latency budget for a voice agent, from detecting the end of speech to playing its reply.

System design · Multimodal and voice · Checklist

I budget to the first audible sound, because that is what a caller perceives as responsiveness, and I enumerate every stage before it: end-of-turn detection, final transcription, model time to first token, enough text to synthesise a first chunk, synthesis, network transport and client buffering. The least obvious term, and often one of the largest, is turn detection — waiting for silence to be confident the user finished — and it trades directly against interruptions: shorten it and the agent talks over people, lengthen it and the agent feels slow. Streaming is what makes the sum survivable, since transcription, generation and synthesis overlap instead of queueing, so I track each stage's contribution separately and set a median and a p95 target, because the tail is what callers remember. A budget has to hold under load as well, so I also define degradation: a shorter reply, a filler acknowledgement, or a cheaper voice when a stage overruns.

Read: The Tail at Scale

↑ Contents

55. Implement a minimal agent runner with validated tool dispatch, error handling and a hard step limit.

Coding · Practical coding · Checklist

I would make the loop's state an explicit object — message history, step counter, deadline, budget and a termination reason — because state hidden in control flow is what makes a runner hard to stop, test and debug. Each iteration asks the model for one action, and dispatch goes through a registry: resolve the tool by name, reject anything unregistered, validate arguments against its schema, then call it. An unknown tool or bad argument is not a crash but an observation returned to the model so it can correct itself, while genuine infrastructure failures retry under a bounded policy. Termination must be reachable several ways — a final answer, the step limit, the deadline, the budget, an unrecoverable error — and the runner records which one fired. I would test the limit path and an always-failing tool: an agent that cannot stop is the expensive failure. It omits approval for consequential actions and persistence across restarts.

Read: Building Effective AI Agents

↑ Contents

56. Distinguish time to first token, time per output token, inter-token latency and throughput when comparing serving systems.

Knowledge · Inference, serving and cost · Checklist

Three of these describe what one user experiences and throughput describes capacity, so under load they pull in opposite directions. Time to first token runs from request arrival to the first token: queueing plus prefill, so it grows with prompt length and queue depth. Time per output token is the average decode pace after that, end-to-end time minus TTFT divided by the remaining tokens. Inter-token latency is that pace as a distribution of individual gaps, exposing stalls an average hides, for example when the scheduler admits a long prefill and every running decode pauses. Throughput is aggregate tokens or requests per second. Larger batches raise throughput and slow each user's decode, so a throughput figure means nothing without the latency it was bought at. I'd compare systems at matched request rate and length distributions, report p50 and p99 for each latency metric, and quote throughput as the highest load still meeting the latency targets. Token counts aren't comparable across tokenisers, and offline batch jobs care only about throughput and cost.

Read: vLLM documentation

↑ Contents

57. Why can an exact attention kernel run faster without changing the quadratic attention computation?

Knowledge · LLM fundamentals · Checklist

Because on a GPU standard attention is limited by memory traffic, not arithmetic. The usual kernel writes the N-by-N score matrix to high-bandwidth memory, reads it back for the softmax, writes the probabilities, then reads them again to multiply by V. FlashAttention tiles Q, K and V into blocks that fit in on-chip SRAM and keeps a running maximum and sum per row, the online softmax, rescaling the partial output as each key block arrives, so the full matrix never lands in HBM. FLOPs stay quadratic; the backward pass even adds some, recomputing scores from saved softmax statistics rather than storing them, and still wins because far fewer bytes move. It's exact, differing from standard attention only by floating-point summation order, unlike sparse or low-rank methods that change what's computed. I'd benchmark the real sequence lengths, head widths, masking and dtype on the target GPU, checking agreement within tolerance. At short sequences attention isn't the bottleneck, and in single-token decode attention's cost is reading the KV cache, a different problem.

Read: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

↑ Contents

58. Design natural-language querying over a warehouse with thousands of tables, from schema selection to safe query execution.

System design · AI system design · Checklist

With thousands of tables the hard part is choosing the schema, not writing SQL, and the real risk is a query that runs and returns a wrong number. So I'd retrieve a small candidate set from the catalog, using table and column descriptions, governed metric definitions, known join paths and verified example queries, and generate against that, preferring a semantic layer where metrics like revenue are already defined, because raw joins invite fan-out double counting. Permissions aren't the model's job: the query runs read-only under the requesting user's own warehouse identity, so row and column security apply, after parsing confirms a single SELECT over catalogued objects, with a cost estimate, timeout and row limit. Ambiguity gets a clarifying question, and the answer shows the SQL and its assumptions. I'd evaluate on real questions with verified queries, comparing result sets rather than SQL text, and measure schema-retrieval recall separately, since a missed table caps everything downstream. Where definitions are contested or undocumented, generation can't fix that; curated metrics come first.

Read: OWASP Top 10 for LLM Applications · Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

↑ Contents

59. A customer reports worse answers after a model upgrade. How do you verify the regression and decide what to restore or change?

Applied scenario · Evaluation and observability · Checklist

A complaint is a lead, not yet a confirmed regression, so I'd start from the customer's concrete failing requests rather than our aggregate evals, which didn't flag it. First I'd establish what actually changed: the model version, but also default parameters, system prompt, tool schemas, output limits and retrieval, any of which can ship with an upgrade. Then I'd replay the same inputs on old and new configurations with everything else pinned, several samples each because outputs vary, and grade them side by side, blind, against criteria the customer agrees describe the failure. If the gap survives, I'd search traffic for similar requests to size it. The remedy follows the cause: a configuration bug gets fixed; a prompt tuned to the old model's quirks gets adapted, often with the customer; a genuine capability loss on their slice justifies pinning the previous version for a bounded window while a fix lands. Either way the cases join the release gate. Pinning is temporary because old versions retire, and if paired replay shows nothing beyond sampling noise, the regression isn't confirmed.

Read: Your AI Product Needs Evals

↑ Contents

60. How would you make an agent's production actions reversible where possible and auditable where reversal is impossible?

System design · Agents, tools and protocols · Checklist

Reversibility has to be designed into each tool before the agent can call it, so I'd classify every action as reversible, compensable or irreversible. Where possible I'd shape the tool so undo is native: drafts instead of sends, soft deletes with retention, scheduled actions with a cancel window, versioned configuration. Compensation is weaker than rollback, since a refund after a charge is a new action that can fail and leaves traces, so each compensable action is registered with its compensating action and the window in which it still works. Before executing, the runner writes an intent record to an append-only log the agent can't modify: exact arguments, an idempotency key, who authorised it, policy and model versions; afterwards, the external system's result. Irreversible actions get stricter gates: previews, limits or human approval. I'd rehearse compensations regularly and test whether the log alone reconstructs any action. The boundary: an audit trail enables accountability and repair, not undo, so for irreversible actions with serious consequences the real control is not letting the agent act alone.

Read: Designing robust and predictable APIs with idempotency · Compensating Transaction pattern

↑ Contents

61. Budget memory for full bf16 finetuning of a 7B model with Adam, then recalculate for LoRA and explain your optimiser assumptions.

Applied scenario · Fine-tuning and post-training · Checklist

I'd state the precision assumption first, because the total depends on it. With bf16 weights and gradients, an fp32 master copy and Adam's two fp32 moments, it's 2 plus 2 plus 12, so 16 bytes per parameter: about 112 GB for 7 billion parameters before any activation, the accounting the ZeRO paper uses. Lower-precision moments or master weights change that, so I'd name my choice. LoRA removes most of it: the frozen base needs only 2 bytes per parameter, about 14 GB, with no gradients or optimiser state, while adapters of tens of millions of parameters at modest rank carry the full 16 bytes and stay under a gigabyte. What LoRA doesn't remove is activations: backpropagation still runs through every frozen layer to reach the adapters, and activation memory grows with batch, sequence length and depth, so at long contexts it can exceed the weights. I'd budget activations separately, add checkpointing if needed, keep allocator headroom, and confirm peak memory on a representative batch at the longest sequence.

Read: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models · LoRA: Low-Rank Adaptation of Large Language Models

↑ Contents

62. Design training for a model that cannot fit on one accelerator; explain how you partition state and keep communication affordable.

System design · Fine-tuning and post-training · Checklist

I'd start by sizing what doesn't fit, because each kind of state has a different fix. With mixed-precision Adam, parameters, gradients and optimiser state cost about 16 bytes per parameter; activations scale separately with micro-batch, sequence length and depth. ZeRO shards model state across data-parallel ranks: stage one splits optimiser state and stage two adds gradients, both at roughly ordinary data-parallel communication, since the all-reduce becomes a reduce-scatter plus an all-gather; stage three also shards parameters and gathers each layer just in time, at about one and a half times the traffic. That divides state, not computation. If one layer's working set still doesn't fit, I need tensor parallelism, which splits matrix multiplications and communicates inside every layer, so it belongs on fast intra-node links; across nodes, pipeline parallelism splits layers into stages that exchange only activations, paying in idle bubbles. Activations get checkpointing. Before scaling I'd measure peak memory per device, the share of step time spent communicating, and per-device throughput from one node to the target, since on slow interconnect stage three's gathers can dominate.

Read: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

↑ Contents

63. Explain group-relative policy optimisation and the trade-offs of estimating advantages without a learned value model.

Knowledge · Fine-tuning and post-training · Checklist

GRPO replaces PPO's learned critic with the group itself as the baseline. For each prompt it samples a group of completions from the current policy, scores them, and sets each one's advantage to its reward minus the group mean, divided by the group's standard deviation; with outcome rewards that value applies to every token. The update is PPO's clipped objective, with KL to the reference model added to the loss rather than folded into the reward. Dropping the value model, usually comparable in size to the policy, saves its memory and training, but rollouts remain the main cost: several samples per prompt, reward computation, reference forward passes. The baseline is noisy for small groups, and when every completion earns the same reward all advantages are zero, so that prompt gives no gradient beyond the KL term; dividing by a tiny spread also amplifies near-unanimous groups. I'd log the share of zero-variance groups, filter prompts by difficulty, and compare held-out success, KL, length and reward stability. A learned critic earns its cost when credit must be assigned across long multi-step trajectories.

Read: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

↑ Contents

64. Policy reward keeps rising while humans prefer its outputs less. How would you diagnose and limit reward hacking?

Applied scenario · Fine-tuning and post-training · Checklist

Rising reward with falling human preference is the signature of overoptimisation: the policy has moved where the reward model's errors outweigh its signal. I'd confirm it with blinded, held-out human comparisons across checkpoints, plotted against proxy reward and KL from the reference policy, to find where human preference peaked. Then I'd look at what the reward newly pays for, such as length, confident tone, formatting or agreeing with the user, and test those features directly on the reward model. The quick limit is rolling back to the peak checkpoint and rerunning with a stronger KL coefficient or fewer steps. The durable fix is coverage: fresh preferences on the current policy's outputs, where the reward model never trained, then retraining it. Gao, Schulman and Hilton saw gold reward rise then fall with distance from the initial policy, but their gold was a synthetic reward model, not people, so the shape is a hypothesis to check. KL control and early stopping limit drift; they don't make the proxy right.

Read: Scaling Laws for Reward Model Overoptimization

↑ Contents

65. Explain how Constitutional AI uses principles, self-revision and AI-generated preferences, and identify what human judgement it still depends on.

Knowledge · Fine-tuning and post-training · Checklist

Constitutional AI replaces human harmlessness labels with model judgements guided by written principles, in two stages. In the supervised stage a helpful model answers harmful prompts, critiques its answer against a principle drawn from the constitution and revises it, and a model is finetuned on the revisions. In the RL stage the model produces answer pairs, a feedback model picks the better one under a sampled principle, and those AI preferences train the preference model used as reward: the RLHF pipeline with machine labels for harmlessness. So the principles shape every critique, every comparison and what the preference model learns. What stays human is substantial: people wrote and selected the principles, helpfulness preferences in the original work were still human labels, the starting model was trained with human feedback, and evaluation was human comparison. I'd measure agreement between AI and human labels on held-out pairs and track over-refusal on benign prompts. It cuts labelling effort; it moves value choices into a document rather than removing them.

Read: Constitutional AI: Harmlessness from AI Feedback

↑ Contents

66. Explain hypothetical-document retrieval and how you would establish whether it improves your query distribution.

Knowledge · RAG and retrieval · Checklist

HyDE bets that an answer-shaped text lands closer to real answers in embedding space than a short question does. A language model writes a plausible document answering the query, an encoder embeds it, and retrieval returns the real documents nearest that vector; the original paper used an unsupervised contrastive encoder without relevance labels, relying on the embedding to discard invented specifics. The hypothetical text is only a search key: it never enters the answer's context or citations, because its details may be fabricated. The cost is a generation call before every retrieval. Whether it helps is an empirical question for my traffic, so I'd label representative queries and compare recall and ranking against direct-query retrieval and a hybrid baseline, by query type, then check answer quality and p95 latency. It tends to help short, underspecified queries worded unlike the documents, and to hurt when the model doesn't know the domain and invents the wrong vocabulary, or when queries hinge on exact identifiers that lexical search already matches.

Read: Precise Zero-Shot Dense Retrieval without Relevance Labels

↑ Contents

67. In an interview where you are expected to use AI tools, how do you stay accountable for code you did not write by hand, and where would you decline what the model produced?

Applied scenario · Practical coding · Checklist

Delegating the typing doesn't delegate ownership: whatever I submit, I should be able to explain line by line, and that judgement is what's really being assessed. So before prompting I'd state the interface, constraints and edge cases myself, which gives me a specification to check output against instead of reading generated code and nodding along. I'd ask for pieces small enough to review in a minute, read each diff before accepting it, and say aloud what I'm checking. I'd decline code I can't explain, calls to library functions I can't confirm exist, error handling that swallows exceptions, tests weakened until they pass, and changes beyond what I asked for; with input handling or anything security-sensitive I'd slow down whatever the clock says. Verification means running tests on the edge cases I named before generating. Under time pressure I'd spend review in proportion to risk: boilerplate gets a glance, core logic gets traced through a case. Rewriting everything by hand is the opposite failure, throwing away the leverage the format expects.

Read: Designing AI resistant technical evaluations · AI Interview Success: An Interviewer's Inside Guide

↑ Contents

68. A team attributes better results to a reasoning model, but it also changed prompts, tools and compute budget. Design a comparison that identifies what improved.

Applied scenario · Evaluation and observability · Checklist

Three changes shipped together, so a before-and-after number credits the model with all of them by default. I'd freeze what isn't under test: the task set, the grader and the tool environment, with tool responses recorded so a live API can't drift between runs. Then I'd vary one factor at a time plus the combinations that matter: old model with the new prompt and tools, new model with the old prompt, both models at the same compute budget. A full factorial over old and new versions of model, prompt, tools and budget is only sixteen configurations. Reasoning models respond strongly to budget, so an unmatched comparison often measures extra test-time compute rather than the model. Each cell gets repeated trials, since a few points on a stochastic system can be noise, and I'd report intervals, hard slices and failure examples, not one average. Separately, I'd measure the deployable system under its real cost and latency limits: the matched-budget winner may be unaffordable. If the factors interact strongly, the honest answer is the combination.

Read: Evaluation best practices · Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

↑ Contents

69. Distinguish outcome rewards, process supervision and inference-time verification. What evidence would justify each for a new reasoning task?

Knowledge · Evaluation and observability · Checklist

The first two are training signals; the third isn't. An outcome reward scores only the end result, an exact answer or passing tests, like the rule-based rewards behind DeepSeek-R1, and reinforcement learning on it changes the weights: cheap when a checker exists, but the credit is sparse and a right answer reached by wrong steps still gets paid. Process supervision labels individual steps and trains a reward model on them; Lightman et al. found it selected maths solutions better than outcome supervision, at the price of step-level human labels. Inference-time verification leaves the weights alone: sample several candidates and let a verifier or a vote choose, paying latency and tokens per request. The same step-level reward model can serve either role, so the distinction is where the signal is applied. I'd want to know whether a reliable outcome check exists, whether step labels are affordable and consistent, and the verifier's false-accept rate, because any rule gets exploited. I'd judge all three on held-out outcome checks independent of that signal, and wouldn't carry maths results to workflows without a checker.

Read: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (v1) · Let's Verify Step by Step

↑ Contents

70. A reasoning API request consumes tokens but returns no usable answer. How would you diagnose it and design a bounded recovery policy?

Applied scenario · Inference, serving and cost · Checklist

Billed tokens and a usable answer are separate facts, so I'd read the completion status first. A reasoning model's hidden tokens count against the same output limit and are billed, so a request can hit that limit mid-reasoning and come back incomplete with little or no visible text. That's a different failure from a refusal, a content-filter stop, a tool error inside an agent loop, or a client timeout where the server may still finish and charge. So I'd log the status and its reason, reasoning versus visible tokens, input size, and the configured limit and effort. Recovery follows the cause: for exhaustion, one retry with a larger output allowance, lower effort or a smaller context, never an identical replay that mostly reproduces the failure. Everything sits under one cap on attempts, spend and elapsed time, and tool calls carry idempotency keys so a retry can't repeat a side effect. I'd track recovered-answer rate and spend per recovery. A task that exhausts generous budgets needs decomposition or a human, not more tokens.

Read: Reasoning models

↑ Contents

71. Plan a product with the interviewer, then build it in about two hours with the AI coding tools of your choice and defend its design, code and path to production.

Applied scenario · Practical coding · Checklist

I would first agree what the user must be able to do and demonstrate the smallest complete flow, rather than treat the time limit as a target for generating code. I would split that flow at clear interfaces, give the assistant only the relevant context and acceptance checks, and inspect each change before integrating it. My review would concentrate on the data model, error handling and assumptions that could invalidate the approach. A short decision log would preserve what I rejected and why. In the demonstration I would show an ordinary case and a failure case, then explain what is measured, what is assumed and what remains unfinished. If a dependency or model stalls, I would cut scope while preserving the core flow. A working prototype is evidence of a bounded result; production still needs appropriate access controls, operational checks and validation.

Read: The AI-native interview · AI Interview Success: An Interviewer's Inside Guide

↑ Contents

72. Review and improve a colleague's draft pull request that adds a cross-cutting feature to an unfamiliar codebase, working alongside coding agents.

Applied scenario · Practical coding · Checklist

I would first reconstruct the intended behaviour and trace the entry points affected by the change. An agent's summary can help navigate, but I would verify it against callers and existing invariants before trusting its recommendations. For a cross-cutting feature, the main risk is inconsistent application: one endpoint may check permission while another, a cache or a background job bypasses it. I would separate observed defects from hypotheses and seek a small counterexample for each important concern. The review decision would cite the behaviour demonstrated by a test or trace, rather than the assistant's confidence. I would also inspect error paths, retries and shared state, and reject fixes that weaken tests or broaden scope without justification. A clean diff is not enough for approval when critical paths remain unexamined; I would explain the residual uncertainty and respect the interview's actual limits on AI use.

Read: The AI-native interview · AI Interview Success: An Interviewer's Inside Guide · Demystifying evals for AI agents

↑ Contents

73. Inside a clone of the product's repository, implement a data structure the product actually uses, asking for AI help only on targeted syntax questions.

Coding · Practical coding · Checklist

I would begin with the callers: which operations they require, how often they run, and what behaviour existing tests promise. That contract determines the representation. A fast lookup alone is not enough if updates, ordering or expiry dominate the workload. I would state the invariants and time and memory costs, then trace small counterexamples before committing to the design. For example, updating an expiring entry must not let an old queued expiry delete its replacement. Empty input, duplicate keys and boundary times make useful probes because they expose assumptions that normal examples hide. I would follow the repository's conventions and keep AI assistance to targeted syntax questions, as this reported task specifies. The final explanation would connect the chosen structure to product behaviour and identify the workload change that would justify replacing it; a structure is not universally best because its average lookup is constant time.

Read: Working Inside Existing Systems · collections: OrderedDict objects · Algorithms, fourth edition: Directed Graphs

↑ Contents

74. In an AI-assisted coding session, explain which models you use and why, how you watch the token budget, and how you give the agent its context and task.

Applied scenario · Practical coding · Checklist

I would choose among the models permitted in the session using the task's difficulty, observed reliability, latency and remaining budget. There is no universal rule that a named model should perform every planning or editing step. Before delegating, I would inspect the relevant code and give the agent a bounded objective, interfaces, invariants, exclusions and a concrete done criterion. Repository text and tool results are evidence to inspect, not authority to change those constraints. I would track context size, output and retries, because repeated failed attempts can dominate both cost and time. If the session drifts, I would return to the last verified state and supply a compact summary of decisions and open questions. A context reset is useful only if it preserves what matters. I remain responsible for accepting the result and would stop delegation when I cannot review the proposed change reliably.

Read: Effective context engineering for AI agents · AI Interview Success: An Interviewer's Inside Guide

↑ Contents

75. How do schema-constrained responses differ from function calls, and which component actually executes an action?

Knowledge · Agents, tools and protocols · Checklist

A schema-constrained response controls the shape of returned data; a tool call requests an operation with arguments. Neither one executes an action or proves that it is authorised. The application must inspect completion status, validate the payload and enforce domain rules before dispatch. A response can be refused or truncated, and even a structurally valid object can contain a wrong account, an invented evidence identifier or an impossible amount. I would distinguish a recoverable formatting failure from a denied action, cap retries and return an explicit failure when recovery is exhausted. For consequential operations, permissions and any required confirmation remain bound to the actual arguments in application code. I would test a valid-looking request that crosses a tenant boundary as well as malformed output. Structured generation reduces parsing failures, but it does not establish truth, business validity or permission.

Read: Tool use with Claude · Structured outputs

↑ Contents

76. A long-running agent confidently pursues the wrong objective. How do you locate where its task state diverged and recover?

Applied scenario · Agents, tools and protocols · Checklist

I would compare the current plan with the last verified objective, constraints and completion criteria, then find the first decision that introduced a different goal. The cause might be an ambiguous handoff, stale memory, a misleading tool result or an instruction embedded in retrieved content. I would not simply ask the same drifting session to try harder. I would preserve useful verified work, discard unsupported assumptions and restart from a concise task state that distinguishes facts, decisions and open questions. Any external effects need reconciliation before replaying steps. To reduce recurrence, I would checkpoint the objective at meaningful transitions and keep a bounded budget and explicit stop reasons. The check is whether subsequent actions advance the agreed outcome without violating constraints. Summarisation helps with context pressure, but a summary that removes a critical restriction can itself cause the drift.

Read: Effective context engineering for AI agents

↑ Contents

77. For a coding agent, how would you separate the model's contribution from the harness's contribution to reliable task completion?

System design · Agents, tools and protocols · Checklist

I would define success through behaviour checked outside the coding agent, then hold the tasks and resource budget fixed while changing the model or the harness separately. The harness includes context assembly, tool permissions, execution isolation, tests, retries and stopping rules; an improvement in any of these can look like a smarter model. I would version those components, repeat uncertain cases and inspect failed traces rather than report only an aggregate pass rate. A patch that passes because the agent weakened the tests is a harness failure, so acceptance checks need protection from the submitted change. Conversely, a broken dependency or timeout can make capable model output fail for environmental reasons. I would report those categories and total cost per accepted task separately. This comparison identifies contributions within the tested setup; it does not establish a universal ranking across repositories or budgets.

Read: Demystifying evals for AI agents

↑ Contents

78. How would you build a useful evaluation set when reference answers are missing and expert time is scarce?

Applied scenario · Evaluation and observability · Checklist

I would begin with a small set of representative requests and a failure taxonomy, then spend expert time where a wrong judgement changes the product decision. Not every case needs a single reference answer: some can be checked through required evidence, forbidden claims, valid state transitions or an explicit abstention rule. Experts should define those criteria and adjudicate ambiguous examples before I automate grading. I would separate development cases from a held-out check set and keep their provenance and revisions visible. Synthetic cases can probe boundaries, but they do not establish the frequency of real user failures. A model judge can help prioritise review only after comparison with human labels; disagreement remains an inspection queue. I would report coverage gaps and uncertainty instead of treating a small, convenient dataset as proof of quality. More labels are valuable when they change a decision, not merely increase a count.

Read: Your AI Product Needs Evals · Demystifying evals for AI agents

↑ Contents

79. How would you change evaluation when an LLM workflow moves from suggesting actions to executing them in production?

System design · Evaluation and observability · Checklist

Once a workflow executes actions, evaluating fluent advice is insufficient: I need to inspect the resulting state and whether the action was permitted. I would test tool selection, arguments, authority, confirmation where required, and the consequences of partial failure. A timeout after a write is especially important because retrying may duplicate an operation that already succeeded. The evaluation should reconcile by operation identity and distinguish safe recovery from a repeated effect. I would use isolated synthetic services for destructive cases and check tenant boundaries, revoked access, exhausted budgets and stopping behaviour. A high average task-success rate cannot compensate for an unauthorised action, so critical failures block the release separately. Logs should identify the versions and external effects without retaining unnecessary sensitive content. These tests provide evidence for the scenarios covered; they do not make every future model-driven action safe.

Read: Demystifying evals for AI agents · OWASP Top 10 for LLM Applications

↑ Contents

80. How would you version prompts and dependent configuration so an observed regression can be rolled back reproducibly?

System design · Evaluation and observability · Checklist

I would version a release bundle, not just a prompt string: model identifier, decoding settings, tool schemas, retrieval configuration, relevant index version and application code all affect behaviour. Each trace should identify that bundle so a reported regression can be reproduced as closely as the provider allows. I would compare the candidate and previous bundle on held-out cases, inspect critical failures separately and define a rollback trigger before rollout. Returning traffic to a known bundle is useful only if its dependencies still exist and remain authorised. Index changes may need a retained compatible version, while an external action cannot be undone merely by restoring a prompt. I would therefore separate configuration rollback from reconciliation of side effects. The operational check is a rehearsed switch and recovery path; perfect reproducibility cannot be promised when a provider changes an unpinned model or service.

Read: Site Reliability Engineering

↑ Contents

81. Design a pipeline that extracts structured fields from ten million scanned documents and makes uncertain results reviewable.

System design · AI system design · Checklist

I would separate ingestion, extraction, validation and review so uncertain fields do not silently become trusted records. Documents need stable identifiers, versions and durable processing state; workers can then retry bounded stages without creating duplicate results. Each extracted field should retain its source page or region and extraction version. Schema checks catch malformed output, while domain checks and sampled expert review address plausible but wrong values. I would calibrate any confidence threshold on representative documents instead of assuming a vendor score has the same meaning for every form. Low-confidence or high-consequence fields enter a review queue with the original evidence and a correction history. Capacity planning includes that queue, not only model throughput. I would measure field-level errors, review load, delay and cost by document class. Ten million processed documents are not success if the rare critical errors become invisible.

Read: Interpret and improve model accuracy and confidence scores · Structured outputs

↑ Contents

82. Design continuous index updates, including changed documents, deletions and queries during a rebuild.

System design · RAG and retrieval · Checklist

I would treat indexing as a versioned projection of the source system, with stable document IDs, change ordering and explicit deletion records. Incremental workers need idempotent updates and a checkpoint so a retry neither skips a change nor resurrects deleted content. For a rebuild, I would create a separate index, load a consistent snapshot, catch up subsequent changes and validate counts, representative queries and permissions before switching the read alias. An atomic alias change helps the cutover, but it does not by itself solve stale writes or access revocation. Those need their own propagation and enforcement policy, including caches. I would monitor source-to-search lag and test updates, deletes and a query during cutover. A retained old index can support rollback only if deleted or newly restricted documents cannot become visible again; freshness includes authority as well as content.

Read: Aliases · Document-level access control

↑ Contents

83. An agent's context fills up over a long task. How would you decide what stays in the window, what is summarised and what moves out of it?

System design · Agents, tools and protocols · Checklist

I would treat the window as a budget, not a container. The model attends to everything in it, so stale history costs money, latency and accuracy: retrieval of material placed mid-context degrades, and measured effective context falls well short of the advertised limit. I would hold a small working set — the goal, the current plan, the last few observations and whatever the next tool call needs — and write finished results to an external store the agent can search, so the record survives without riding in every request. Compaction happens at boundaries I choose, such as a completed sub-task, and preserves identifiers, open questions and decisions verbatim, since those are what a later step reads back. I would track tokens and latency per step and watch task success as the window fills. This stops holding when the task genuinely needs global comparison across everything retrieved; then I would split the work rather than compress it.

Read: Effective context engineering for AI agents · Lost in the Middle: How Language Models Use Long Contexts · RULER: What's the Real Context Size of Your Long-Context Language Models?

↑ Contents

84. Several product teams share one internal AI gateway. How would you keep one team's traffic, data and caches from affecting another's?

System design · AI platform and operating model · Checklist

I would decide isolation per shared resource rather than once for the gateway. The resources are provider quota, the gateway's own capacity, prompt and embedding caches, request logs and evaluation data. Each tenant gets its own credentials, quota and rate limit, so a retry storm in one team degrades only that team; without per-tenant limits one incident consumes the shared quota. Caches are keyed by tenant by default. A shared prefix cache does not hand one tenant another's text, but a cache hit is faster than a miss, so timing reveals whether a prefix has been seen before — worth closing where prompts are themselves sensitive. Logs carry prompts and completions, so I would scope read access per tenant and set retention, while aggregate cost and latency telemetry stays central. I would test isolation by loading one tenant and watching the others. Strict isolation costs cache hit rate and money; where every tenant is one legal entity under one data policy, I would say so and relax it deliberately.

Read: Architect multitenant solutions on Azure · Prompt caching · Scaling your API with rate limiters

↑ Contents

85. A provider announces that a model version your products depend on retires in three months. How would you run the migration across the teams that use it?

Applied scenario · AI platform and operating model · Checklist

The deadline is fixed and the risk is schedule, not technology, so I would start with an inventory: which services, prompts, evaluation baselines and tuned models name that version, and what each depends on. A model version is a dependency, and treating it as a configuration value is how a quiet regression ships. Each owning team re-runs its evaluation against the replacement before anything moves, because a new version changes behaviour: prompts, output formats and thresholds usually need work, and that work is part of the migration rather than a follow-up. I would move services one at a time behind a flag while the old version still answers, so rollback is real rather than theoretical, and I would keep the riskiest and the highest-traffic cases apart. What I would leave behind matters as much: pinned versions, an inventory that stays current and an owner per integration. This is harder where a tuned model has no equivalent on the new version; then retraining, not migration, is the real plan.

Read: Managing technical lock-in in the cloud · Rules of Machine Learning

↑ Contents

86. Your AI platform's latency and availability depend on an external model provider. What would you promise internal teams, and how would you keep that promise?

System design · AI platform and operating model · Checklist

I would promise an end-to-end objective the platform can actually hold, stated in what a caller feels: latency at a high percentile and the share of requests that finish usefully, measured at our edge rather than taken from the provider's uptime page. Availability I do not control is not a promise I can make, so the objective rests on the controls I own — a timeout that matches the caller's deadline, a bounded retry budget, a smaller fallback model, a degraded but honest answer, and a queue for work that can wait. Retries deserve care: naive retries during provider overload add load to a system already failing. The fallback changes answer quality, so it needs its own evaluation and the caller needs to know which path served them. I would publish an error budget, say what happens when it is spent, and name the failure modes that stay outside my control. Where a team needs a stronger guarantee than that, the answer is a second provider or capacity we reserve, and both cost money.

Read: Site Reliability Engineering · Scaling your API with rate limiters

↑ Contents

87. Before launching an assistant that can use tools, how would you organise adversarial testing, and what would a clean result let you claim?

System design · Safety, security and governance · Checklist

I would start from the harms rather than the techniques, and from two different attackers: a user trying to make the product do something it should not, and untrusted content the model reads — a web page, a document, a tool result — that carries instructions. The second matters more once the assistant has tools, because the damage is an action, not a sentence. Coverage comes from mixing automated generation of many variants with people who did not build the system, and from logging every attempt rather than the successes alone, so the suite can be re-run against the next version and coverage can be argued. Findings go to named owners with fixes, and I would distinguish blocking a class of attack from one phrasing that stopped working. The honest claim at the end is narrow: the attacks we tried, at this version, mostly failed. That is not a safety proof — prompt injection has no complete fix today — so it has to sit on top of least privilege and confirmation for consequential actions.

Read: OWASP Top 10 for LLM Applications · AI Risk Management Framework · Prompt injection (series)

↑ Contents