Applied and customer scenarios¶

English · Русский · AI Interview Atlas › Themes

Open-ended situations with customers and messy problems: the first 48 hours, rescuing a pilot, decomposing a business problem.

On this page: Both tracks (16) · AI Engineering (4) · AI Leadership (5)

Both tracks¶

A customer wants to automate claims processing with AI. What would you do in the first two weeks?
- Applied scenario · Senior · Asked at: OpenAI † · ✍ Answer · Suggest a fix
- Tests: Whether you turn a broad request into a bounded, measurable workflow.
- A strong answer covers:
  - Map the current process, decisions, exceptions and accountable users.
  - Check data access, error consequences and which steps need human judgement.
  - Choose a narrow pilot with a baseline, acceptance criteria and an owner.
- Read: Building Effective AI Agents (Anthropic) · AI Risk Management Framework (NIST)
A customer executive wants to cancel an AI pilot because it keeps producing wrong results. What would you do over the next 48 hours?
- Applied scenario · Senior · Asked at: Palantir † · ✍ Answer · Suggest a fix
- Tests: Whether you combine incident triage with a credible customer recovery plan.
- A strong answer covers:
  - Acknowledge impact and collect concrete failing workflows with the customer.
  - Contain harmful behaviour and separate data, system and expectation problems.
  - Agree on a short recovery experiment and explicit continue-or-stop criteria.
- Read: Site Reliability Engineering (Google) · Demystifying evals for AI agents (Anthropic)
A customer insists on fine-tuning their own model on support tickets, while you suspect retrieval would be enough. How do you proceed?
- Applied scenario · Senior · Asked at: Databricks † · ✍ Answer · Suggest a fix
- Tests: Whether you investigate the need without turning architecture preferences into a sales argument.
- A strong answer covers:
  - Ask what ownership is meant to achieve: behaviour, knowledge, cost or control.
  - Build a representative task set and compare plausible approaches.
  - Explain data governance, maintenance and migration costs before recommending.
- Read: Rules of Machine Learning (Google for Developers) · Building Effective AI Agents (Anthropic)
A deployed agent's base model will be retired in 60 days. How would you migrate it without losing quality?
- Applied scenario · Senior · Asked at: Databricks † · ✍ Answer · Suggest a fix
- Tests: Whether provider lifecycle risk is handled through reproducible evaluation and controlled change.
- A strong answer covers:
  - Inventory affected workflows, tool contracts and model-specific behaviour.
  - Compare candidates against representative recorded tasks and failure cases.
  - Plan staged migration, rollback within the available window and customer communication.
- Read: Demystifying evals for AI agents (Anthropic) · Site Reliability Engineering (Google)
A customer says their eight-GPU chatbot is too slow and too expensive. You have a week with them; how do you use it?
- Applied scenario · Senior · ✍ Answer · Suggest a fix
- Tests: Whether you diagnose bottlenecks and evaluate changes against useful customer outcomes.
- A strong answer covers:
  - Measure real workload, latency distribution, utilisation and cost per successful task.
  - Profile before choosing serving, model, retrieval or interaction changes.
  - Validate improvements on the same workload with quality and reliability guardrails.
- Read: Site Reliability Engineering (Google) · Rules of Machine Learning (Google for Developers)
An enterprise wants document Q&A over two million internal files within four weeks, with no data leaving its VPC. How would you scope the pilot?
- Applied scenario · Senior · ✍ Answer · Suggest a fix
- Tests: Whether you negotiate scope around access controls, deployment constraints and measurable value.
- A strong answer covers:
  - Clarify corpus priority, permissions, permitted services and the most valuable queries.
  - Choose a representative subset and a deployable end-to-end workflow.
  - Agree acceptance tests, operational ownership and what the pilot cannot establish.
- Read: Architect multitenant solutions on Azure (Microsoft Learn) · AI Risk Management Framework (NIST)
An enterprise customer says Claude hallucinates in their retrieval-based knowledge assistant. How would you investigate the first failures?
- Applied scenario · Senior · Asked at: Anthropic † · ✍ Answer · Suggest a fix
- Tests: Whether you separate retrieval failures, unsupported generation and misunderstood user expectations.
- A strong answer covers:
  - Collect concrete failing cases with full traces: question, retrieved passages, assembled prompt and answer.
  - Sort each into missed retrieval, an answer unsupported by retrieved text, or a question the corpus cannot answer.
  - Report the split with counts, fix the dominant cause first, and keep the cases as a regression set.
- Read: Your AI Product Needs Evals (Hamel Husain)
A freight railway loses substantial revenue to unexpected locomotive downtime. Turn the problem into an engineering plan.
- Applied scenario · Senior · Asked at: Palantir † · Suggest a fix
- Tests: Whether you connect operational decisions, usable data and intervention value.
- A strong answer covers:
  - Ask which failure modes cost most and which decision an early warning would change.
  - Check whether sensor and repair logs label failures, then price false alarms against missed breakdowns.
  - Close on a pilot fleet, comparing unplanned downtime and cost with matched locomotives.
- Read: Rules of Machine Learning (Google for Developers) · Site Reliability Engineering (Google)
How would you design a system to improve traffic in New York City?
- System design · Senior · Asked at: Palantir † · Suggest a fix
- Tests: Whether you decompose an ambiguous city-scale objective into measurable interventions.
- A strong answer covers:
  - Ask what improving traffic means and for whom: travel time, safety, emissions, buses or freight.
  - Choose levers the system actually controls, such as signal timing or bus lanes, and their data.
  - Evaluate one corridor against a comparable control, watching for traffic displaced onto nearby streets.
- Read: User Needs + Defining Success (Google PAIR) · Data and AI Ethics Framework (Government Digital Service)
Design synchronisation between two systems that hold employee records.
- System design · Senior · Asked at: Palantir † · Suggest a fix
- Tests: Whether you identify record ownership, conflicts and recovery semantics.
- A strong answer covers:
  - Assign an owner per field, not per record, and a stable identifier linking both systems.
  - Propagate changes idempotently with version ordering, and define which side wins each conflict.
  - Reconcile regularly, replay safely after outages, and route unmatched or deleted records to review.
- Read: Designing robust and predictable APIs with idempotency (Stripe)
Design an application for recording species while exploring an unfamiliar environment.
- System design · Senior · Asked at: Palantir † · Suggest a fix
- Tests: Whether you clarify field constraints and support trustworthy data collection.
- A strong answer covers:
  - Ask about field constraints first: connectivity, battery, gloves, light and who records.
  - Design capture to work offline with location, time and photo, and resolve sync conflicts later.
  - Store the model's suggestion apart from the observer's call, and audit a sample against experts.
- Read: Build an offline-first app (Android Developers)
A customer wants to replace a proprietary frontier-model API with an open model. How would you run the engagement?
- Applied scenario · Senior · Suggest a fix
- Tests: Whether the migration plan tests quality, operating costs and ownership assumptions.
- A strong answer covers:
  - Ask why they want to switch: cost, data control, customisation or vendor dependence.
  - Compare quality on their own tasks and full operating cost, including serving, upgrades and staff.
  - Close with a shadow run against agreed thresholds and named owners for serving and upgrades.
- Read: Evaluation best practices (OpenAI API documentation) · vLLM documentation (vLLM project)
A robotics customer requests 50,000 hours of demonstrations across twelve tasks and three robot types. How would you organise collection and decide what data to keep?
- Applied scenario · Senior · Suggest a fix
- Tests: Whether collection targets are tied to coverage and usable demonstrations rather than hours alone.
- A strong answer covers:
  - Ask what the policy must be able to do, and how success on each task is judged.
  - Allocate collection by coverage of tasks, robots and conditions, not equal hours per cell.
  - Reject failed or sloppy demonstrations, and train on early batches to see which data raises success.
- Read: Open X-Embodiment: Robotic Learning Datasets and RT-X Models (Open X-Embodiment Collaboration, arXiv)
Estimate the cost and turnaround of AI due diligence over 5,000 documents. Which improvement would you investigate first?
- Applied scenario · Senior · Suggest a fix
- Tests: Whether estimates expose workload assumptions and the dominant cost driver.
- A strong answer covers:
  - State workload assumptions: pages per document, tokens per page, passes and questions per document.
  - Compare model cost with lawyers' review time, which may dominate, and find the turnaround bottleneck.
  - First test what cuts the dominant driver, such as triaging documents for review, on a labelled sample.
- Read: Cost Optimization Pillar — AWS Well-Architected Framework (Amazon Web Services)
A lawyer reports that the assistant missed a change-of-control clause during contract review. How would you investigate?
- Applied scenario · Senior · Suggest a fix
- Tests: Whether you trace the failure through ingestion, retrieval, reasoning and review.
- A strong answer covers:
  - Reproduce the exact case: document version, query, retrieved passages and the output the lawyer saw.
  - Locate the failing stage: parsing, chunking, retrieval, reasoning over indirect wording, or human review.
  - Turn it into a regression test, check similar contracts for the same miss, and warn affected users.
- Read: Contextual Retrieval in AI Systems (Anthropic) · Your AI Product Needs Evals (Hamel Husain)
After an introduction to insider trading, explain how you would detect it: what data would you need, what would you ask the customer and what patterns would you look for?
- Applied scenario · Asked at: Palantir 🗣 · Suggest a fix
- Tests: Whether you turn a vague, high-stakes problem into a scoped first system that can be tested.
- A strong answer covers:
  - Ask which instruments and data exist, and what an analyst must decide from an alert.
  - Scope one pattern first: unusual trading by connected accounts before price-moving announcements.
  - Backtest on confirmed past cases, and cap alert volume at what analysts can review.
- Read: Insider Trading Detection: FINRA’s Vital Role in Ensuring Market Integrity (FINRA)

AI Engineering¶

Instead of a coding round, build a proof of concept for a customer scenario inside the company's own agent product and show how the customer would use it.
- Applied scenario · Asked at: Cognition † · Suggest a fix
- Tests: Whether you understand how customers actually use the product, not only how to write code.
- A strong answer covers:
  - Ask who the customer's user is and which task in their day the product takes over.
  - Build with the product's own features as a customer would, not custom code that bypasses them.
  - Demo the flow end to end on realistic input, including a failure and how the user recovers.
- Read: How the alpha phase works (Government Digital Service)
Lead a timed case-study call with a simulated customer: uncover the real problem, propose an approach and agree on next steps.
- Applied scenario · Asked at: Cognition † · Suggest a fix
- Tests: Whether you run a customer conversation under time pressure and leave with a concrete plan.
- A strong answer covers:
  - Open with questions on the outcome they need, the current workflow and who decides.
  - Propose the smallest approach that tests their biggest risk, and say what you would not do.
  - Close on dated next steps, owners, success criteria and the data they will send.
- Read: User Needs + Defining Success (Google PAIR) · How to recruit UX Research participants (GitLab Handbook)
Integrate a library you have never used into an existing system within minutes, then design the surrounding system for scale, storage and failures.
- Applied scenario · Asked at: Palantir 🗣 · Suggest a fix
- Tests: Whether you learn an unfamiliar tool quickly and carry it into sound design decisions.
- A strong answer covers:
  - Read the core abstraction and one minimal example, then wrap the library behind a thin interface.
  - Carry what you learned about its limits, state and errors into the scale and storage design.
  - Name how the design fails, plan retries, timeouts and backpressure, and test the integration path.
- Read: Working Inside Existing Systems (Palantir Careers)
Write a prompt that extracts a date range from a customer's request in Russian and tells apart near-identical phrasings such as «с первого числа» (from the first), «по первое число» (through the first) and «за первое число» (for the first).
- Applied scenario · Asked at: Tochka ✅ · Suggest a fix
- Tests: Whether you handle language-specific ambiguity and test a prompt on hard cases.
- A strong answer covers:
  - Define the output: start and end dates, inclusive or exclusive bounds, and the reference date.
  - Spell out how the prepositions «с», «по» and «за» set each bound, with contrasting examples.
  - Test on a set of near-identical phrasings and relative dates, and ask back when it is ambiguous.
- Read: Evaluation best practices (OpenAI API documentation)

AI Leadership¶

How would you explain a complex AI research result to a nontechnical audience?
- Applied scenario · Senior · Asked at: Anthropic † · Suggest a fix
- Tests: Whether you communicate implications and limitations without replacing evidence with hype.
- A strong answer covers:
  - Ask who the audience is and which of their decisions the result should inform.
  - Cut detail, never accuracy: one honest analogy, what was measured and against which baseline.
  - State what the result does not show, and check the audience can restate its limits.
- Read: Data and AI Ethics Framework (Government Digital Service)
A prospective customer already runs on H100 GPUs. When would you recommend that they keep their current setup?
- Applied scenario · Senior · Suggest a fix
- Tests: Whether you can reject a migration when customer economics do not justify it.
- A strong answer covers:
  - Ask what prompted the conversation: cost, capacity, latency or pressure from a vendor.
  - Compare cost per useful output after migration effort, porting risk and remaining hardware commitments.
  - Recommend staying when the gain does not clear switching costs, and name what would reopen it.
- Read: Cost Optimization Pillar — AWS Well-Architected Framework (Amazon Web Services) · Managing technical lock-in in the cloud (Government Digital Service and Central Digital and Data Office)
A hospital group schedules outpatient visits manually by phone. What would you build for them?
- Applied scenario · Senior · Suggest a fix
- Tests: Whether you design around workflow, integration and safe exception handling.
- A strong answer covers:
  - Ask how a booking happens today: who calls, which rules apply, which system holds the schedule.
  - Write into the existing scheduling system, automate routine bookings, and route urgent or clinical requests to staff.
  - Pilot in one clinic, measuring completed bookings, no-shows and correct escalations against the phone baseline.
- Read: NHS service standard (NHS Digital Service Manual)
A contact centre wants voice agents to replace its phone menus. How would you lead the engagement?
- Applied scenario · Senior · ✍ Answer · Suggest a fix
- Tests: Whether you scope user journeys, handoff and operational readiness together.
- A strong answer covers:
  - Take the highest-volume intents first and define containment honestly, counting transfers.
  - Design the handoff to a human as a first-class path that carries context and never loops.
  - Plan operational readiness: latency targets, interruption handling, monitoring and a rollback.
- Read: Building Effective AI Agents (Anthropic)
A public agency wants a multilingual assistant for a paper-and-call-centre welfare service, hosted on premises. How would you scope delivery?
- Applied scenario · Senior · Suggest a fix
- Tests: Whether access, language coverage and deployment constraints shape the service.
- A strong answer covers:
  - Ask which languages, channels and literacy levels users have, and who cannot use a chat at all.
  - Weigh on-premises limits on model size against quality per language, and keep phone and paper routes.
  - Accept on per-language task success with native-speaker review, leaving eligibility decisions to caseworkers.
- Read: AI Risk Management Framework (NIST)

← Programmes and delivery · Payments and regulated domains →