Skip to content

Evaluation and observability

English · Русский · AI Interview Atlas › Themes

Knowing whether an AI system works: eval sets, LLM judges, regression gates, online measurement and production traces.

On this page: Both tracks (8) · AI Engineering (13)

Both tracks

  • A team attributes better results to a reasoning model, but it also changed prompts, tools and compute budget. Design a comparison that identifies what improved.
    • Applied scenario · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether a model claim is separated from changes to the surrounding system.
    • A strong answer covers:
      • Freeze task inputs, scoring and tool state; vary model, prompt and budget in controlled comparisons.
      • Repeat trials and report uncertainty, difficult slices and failures rather than only aggregate wins.
      • Compare both a matched-budget baseline and the deployable system under its actual cost and latency limits.
    • Read: Evaluation best practices (OpenAI API documentation) · Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Snell et al., arXiv)
  • A reasoning summary looks sound, but the answer follows a misleading hint. What should an evaluation and monitoring system conclude?
    • Applied scenario · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether observable explanations are distinguished from correctness and faithful accounts of computation.
    • A strong answer covers:
      • Score final results and observable actions independently of explanation quality.
      • Compare otherwise identical tasks with and without misleading hints; measure answer changes and disclosures separately.
      • Treat summaries and available traces as incomplete diagnostic signals, not access to hidden reasoning or a safety guarantee.
    • Read: Reasoning Models Don't Always Say What They Think (Chen et al., arXiv)
  • How would you build a useful evaluation set when reference answers are missing and expert time is scarce?
    • Applied scenario · ✍ Answer · Suggest a fix
    • Tests: Whether sampling and annotation effort target consequential uncertainty.
    • A strong answer covers:
      • Start from error analysis of real traces to learn which failures matter, before labelling anything.
      • Spend expert time on uncertain, high-stakes cases, judged by pass-fail criteria rather than one answer to match.
      • Calibrate cheaper graders against expert labels and track agreement before trusting them at scale.
    • Read: Your AI Product Needs Evals (Hamel Husain) · Demystifying evals for AI agents (Anthropic)
  • Design a release gate for prompt and model updates, including what happens when aggregate gains hide a critical regression.
    • System design · Asked at: Anthropic † · ✍ Answer · Suggest a fix
    • Tests: Whether release decisions account for failure severity and uncertainty.
    • A strong answer covers:
      • Version the candidate, baseline, dataset and graders.
      • Set acceptance rules for critical slices and repeated stochastic trials.
      • Combine offline results with a monitored rollout and explicit rollback criteria.
    • Read: Demystifying evals for AI agents (Anthropic)
  • Benchmark results improve but users report a worse product. What hypotheses would you test first?
    • Applied scenario · ✍ Answer · Suggest a fix
    • Tests: Whether evaluation coverage is connected to actual user tasks.
    • A strong answer covers:
      • Compare benchmark tasks and traffic with actual user workflows.
      • Check leakage, slice regressions, latency and changes in user expectations.
      • Reproduce complaints and add representative failures without overfitting the test set.
    • Read: Rules of Machine Learning (Google for Developers)
  • Design online evaluation for an AI feature: choose what to log, review manually and compare in an experiment.
  • A customer reports worse answers after a model upgrade. How do you verify the regression and decide what to restore or change?
    • Applied scenario · Asked at: OpenAI † · ✍ Answer · Suggest a fix
    • Tests: Whether concrete customer failures drive reproduction and remediation.
    • A strong answer covers:
      • Collect the customer's exact failing requests and list every change shipped with the upgrade, not only the model.
      • Replay old and new configurations on identical inputs with repeated samples and blind side-by-side grading.
      • Choose the remedy by cause: fix configuration, adapt the prompt or pin the version; add cases to the release gate.
    • Read: Your AI Product Needs Evals (Hamel Husain)
  • How would you change evaluation when an LLM workflow moves from suggesting actions to executing them in production?
    • System design · Asked at: Palantir † · ✍ Answer · Suggest a fix
    • Tests: Whether action consequences change the acceptance criteria.
    • A strong answer covers:
      • Grade outcomes in the environment, not the proposal text: what changed, and was it permitted.
      • Weight errors by consequence and reversibility; irreversible actions need near-zero tolerance or human approval.
      • Test in sandboxed replicas, then use shadow mode and staged rollout with action logs and a kill switch.
    • Read: Demystifying evals for AI agents (Anthropic) · OWASP Top 10 for LLM Applications (OWASP Gen AI Security Project)

AI Engineering

  • Distinguish outcome rewards, process supervision and inference-time verification. What evidence would justify each for a new reasoning task?
    • Knowledge · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether training signals are distinguished from deployment-time selection and independently measured quality.
    • A strong answer covers:
      • Explain which feedback changes model weights and which selects among outputs at inference.
      • Assess label availability, verifier errors and ways to exploit the reward or selection rule.
      • Use independent held-out outcome checks; do not generalise mathematics results to every workflow.
    • Read: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (v1) (DeepSeek-AI, arXiv) · Let's Verify Step by Step (Lightman et al., arXiv)
  • Design an LLM judge and a calibration process that exposes order, verbosity and self-preference biases.
    • System design · Asked at: Perplexity † · ✍ Answer · Suggest a fix
    • Tests: Whether model judgements are validated against independent labels.
    • A strong answer covers:
      • Define a task-specific rubric and a held-out human-labelled calibration set.
      • Randomise answer order and test sensitivity to length and model identity.
      • Measure disagreements and use expert review where automated judgement is unreliable.
    • Read: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., arXiv)
  • How would you measure unsupported claims in a deployed RAG application without treating every fluent answer as correct?
    • System design · Asked at: Anthropic †, OpenAI † · ✍ Answer · Suggest a fix
    • Tests: Whether faithfulness to evidence is separated from fluency and general plausibility.
    • A strong answer covers:
      • Make the unit a claim checked against the passages actually retrieved for that answer.
      • Scale labelling with a judge calibrated against a human-labelled sample.
      • Separate a retrieval failure from an unfaithful generation when attributing the error.
    • Read: Your AI Product Needs Evals (Hamel Husain)
  • How does benchmark contamination distort evaluation, and how would you reduce leakage into training or model selection?
    • Knowledge · Suggest a fix
    • Tests: Whether repeated tuning on test data is recognised as leakage.
    • A strong answer covers:
      • Distinguish test items seen in pre-training from selection leakage through repeated tuning on the test set.
      • Keep a private test set used once per decision, and tune only on a separate development split.
      • Detect contamination with n-gram overlap and canary strings, and compare against items created after the data cutoff.
    • Read: Proving Test Set Contamination in Black Box Language Models (Oren et al., arXiv)
  • Design observability for a production LLM workflow: which spans, versions, costs and feedback should be linked?
    • System design · ✍ Answer · Suggest a fix
    • Tests: Whether a failed user task can be reconstructed without indiscriminate sensitive logging.
    • A strong answer covers:
      • Link model, retrieval and tool spans to one task identifier and version set.
      • Record latency, token use, outcomes and sampled feedback with retention controls.
      • Use traces to diagnose stage failures and convert recurring failures into eval cases.
    • Read: Site Reliability Engineering (Google) · Demystifying evals for AI agents (Anthropic)
  • How would you version prompts and dependent configuration so an observed regression can be rolled back reproducibly?
    • System design · ✍ Answer · Suggest a fix
    • Tests: Whether rollback restores the actual system configuration.
    • A strong answer covers:
      • Version the full configuration as one immutable bundle: prompt, model snapshot, parameters, tools and retrieval index.
      • Pin model snapshots rather than aliases, since a floating alias changes behaviour without a deploy.
      • Log the bundle id on every request, and confirm a rollback by rerunning the failing cases on it.
    • Read: Site Reliability Engineering (Google)
  • How should evaluation of a tool-using agent differ from grading one generated response?
    • System design · ✍ Answer · Suggest a fix
    • Tests: Whether final state, trajectory and repeated-run reliability are all inspected.
    • A strong answer covers:
      • Check the final environment state, not just the agent's success message.
      • Grade task completion separately from unsafe or wasteful trajectories.
      • Repeat trials and report reliability across runs with clean, controlled environments.
    • Read: Demystifying evals for AI agents (Anthropic)
  • Design an experiment for a suspected emergent capability or bias in a large model, with controls against misleading task framing.
    • Applied scenario · Asked at: Anthropic † · Suggest a fix
    • Tests: Whether a capability claim is testable and alternative explanations are considered.
    • A strong answer covers:
      • Turn the claim into a prediction with a metric, a threshold and a result that would refute it.
      • Control framing: paraphrased prompts, shuffled options, counterbalanced order, and continuous metrics beside exact match.
      • Rule out contamination and scoring artefacts with fresh items, smaller-model baselines and repeated samples.
    • Read: Are Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al., arXiv)
  • Build the evaluation harness for a new model release, covering reproducibility, execution isolation and comparable results.
    • System design · Asked at: Google and Google DeepMind † · Suggest a fix
    • Tests: Whether evaluation infrastructure can support trustworthy comparisons.
    • A strong answer covers:
      • Pin everything that shapes a score: dataset version, prompt template, decoding settings, grader and seeds.
      • Run model-generated code in isolated, network-restricted sandboxes with resource limits and clean state per task.
      • Report means over repeated samples with confidence intervals, and rerun a baseline model to detect harness drift.
    • Read: Demystifying evals for AI agents (Anthropic) · Evaluation best practices (OpenAI API documentation)
  • Two qualified clinicians produce different good notes from the same consultation. How would you evaluate a generated note fairly?
  • In a short unscripted call, propose testable ideas about a language model's behaviour that need no insider access, only careful observation.
    • Knowledge · Asked at: Anthropic 🗣 · Suggest a fix
    • Tests: Whether you generate concrete, testable hypotheses about model behaviour quickly.
    • A strong answer covers:
      • State each idea as a behaviour, a controlled prompt change and the measurable outcome you expect.
      • Draw on observable levers: position of information, option order, length, persona, language and repeated sampling.
      • Say how many samples and which control would separate a real effect from noise or prompt wording.
    • Read: Lost in the Middle: How Language Models Use Long Contexts (Liu et al., arXiv)
  • Frame a business problem as a machine-learning task: choose and justify the quality metric, and explain how you would collect and validate the data.
    • Knowledge · Asked at: T-Bank ✅ · Suggest a fix
    • Tests: Whether the metric and data follow from the business goal rather than habit.
    • A strong answer covers:
      • Translate the business decision into a prediction target, the action it triggers and the cost of each error.
      • Pick the offline metric from those error costs, and name the online business metric it should move.
      • Validate the data: label delay and leakage, time-based splits, and drift between training and serving.
    • Read: Rules of Machine Learning (Google for Developers)
  • Modify a prompt so that a target share of the model's outputs matches the desired result, and show how you measured it.
    • Applied scenario · Asked at: Anthropic 🗣 · Suggest a fix
    • Tests: Whether prompt changes are driven by measurement on a set of cases rather than single examples.
    • A strong answer covers:
      • Define the desired result as a pass-fail check, and build a representative case set first.
      • Change one thing at a time and sample each case several times, since outputs vary between runs.
      • Report the share with a confidence interval on held-out cases, so the gain is not overfitting.
    • Read: Evaluation best practices (OpenAI API documentation) · Your AI Product Needs Evals (Hamel Husain)

← Fine-tuning and post-training · Safety, security and governance →