Skip to content

Answers: AI Leadership

English · Русский · AI Interview Atlas › Tracks and roles

Written answers to the priority questions of this track, in the same order and numbering as Start here. Answer each question yourself first: what follows is one good answer, not the only correct one, and an interviewer is listening to your reasoning rather than checking your wording.

Answers written: 82 of 82.

Contents

1. An assistant reads external email, searches internal documents and sends replies. Where could an attacker redirect it, and how would you constrain the damage?

System design · Safety, security and governance · Checklist

Incoming mail lands in the same context as instructions, while the assistant's authority is wider than the sender's rights: a message asks it to find an internal document and forward it outside, and the model obliges. You do not fix that with prompt wording, you fix it by separating levels. Untrusted mail content stays data, and the authority to search or send lives in a tool layer that never reads a message as a command. I would narrow the tool set, restrict recipients to the current thread and known addresses, and require explicit human approval to go beyond that or to attach files. I would test against a corpus of indirect injections and exfiltration attempts, measuring the share of outbound messages carrying data above the requester's access level. The risk does not disappear: injection filters miss things, and data can still leak through an allowed channel.

Read: OWASP Top 10 for LLM Applications · The lethal trifecta for AI agents: private data, untrusted content, and external communication · Prompt injection (series)

↑ Contents

2. How would you organise safe deployment of an AI model into production?

System design · Safety, security and governance · Checklist

Safe deployment means every step of exposure is a decision a named person makes against criteria written before the results arrive. I'd turn intended use and the outcomes we won't accept into evals with thresholds, so the release bar exists before anyone is attached to the date. Ownership is split on purpose: one owner runs the evaluation, someone answerable for the product's consequences, not the person measured on the launch date, accepts the residual risk, and the on-call owner can halt rollout without a meeting. Exposure goes internal, then a small share of traffic, then wider, with halt thresholds on a labelled sample of live traffic: harmful responses, complaints, escalations. The non-obvious cost is that small stages see rare harms slowly, so a stage ends when enough traffic has passed for the rare failure to show, not after a fixed number of days. The boundary: rollback doesn't recall sent messages or undo actions taken, and a prompt edit or an index refresh changes behaviour as much as a new model, so it passes the same gate.

Read: AI Risk Management Framework · Site Reliability Engineering · Anthropic's Responsible Scaling Policy

↑ Contents

3. How would you approach safety for a generative-AI consumer product?

Product case · Safety, security and governance · Checklist

Safety requirements should come from who actually uses the product and what harm is plausible for them; a generic list of prohibitions over-refuses ordinary users and still misses the specific risk. So I'd write an abuse model per population: a teenager looking for self-harm methods, an adult using role-play to extract dangerous instructions, someone generating content to harass a real person. Each layer stops something different: a product limit removes a capability outright, certain but costly for legitimate use; model behaviour covers the grey zone and is probabilistic; reports surface patterns the eval set hasn't seen; human escalation handles crisis signals where a wrong automatic reply is expensive. I'd measure harmful completions and unnecessary refusals on a double-labelled sample stratified by segment and language, oversampling small groups so their rates are visible at all. A severe new pattern in reports gets a blunt product block at once, then a model fix and an eval update. The boundary: the product rarely knows a user's age or intent, and averages hide harm concentrated in a small group.

Read: AI Risk Management Framework · OWASP Top 10 for LLM Applications

↑ Contents

4. What safeguards should surround an AI system that acts on a user's behalf?

System design · Safety, security and governance · Checklist

The key distinction is that the model proposes an action and the system authorises it, and those are different subsystems. Authority is scoped to the user, the resource and the operation: the agent acts no wider than the rights of the person it acts for, and a token is issued for a specific operation rather than for the whole account. Consequential and irreversible actions are shown to a human in plain wording: what exactly, to whom, for how much. Policy is checked at execution time on the service side, because a check inside the prompt can be talked around. You need an action log with an end to end identifier, spending and rate limits, and a recovery plan for partial completion, where a compensating operation matters more than a blind retry. The boundary is confirmation fatigue: ask about everything and people stop reading, so confirm what is expensive and irreversible.

Read: OWASP Top 10 for LLM Applications · Designing robust and predictable APIs with idempotency

↑ Contents

5. Describe a situation where delivery pressure conflicted with security or safety concerns. How did you decide what to do?

Behavioral · Safety, security and governance · Checklist

Answer this as a risk decision, not as a story about saving the day. Take a real episode and start with the hazard: what specifically could have happened, who it would have affected, and what evidence you had at the time, including what you did not know. Then name the alternatives you considered seriously, such as narrowing the launch, enabling it for a subset of users, adding manual review, or moving the date, and say who had the authority to accept the remaining risk and how you escalated. That is what the interviewer is listening for: whether you separate your own opinion from someone else's decision, and whether you know who owns the risk. The common failure is the hero who saved everyone, or a decision with no author. Finish with an honest result, including what went wrong, and the control that changed afterward.

Read: AI Risk Management Framework

↑ Contents

6. How would you improve ChatGPT for enterprise customers?

Product case · AI product strategy and metrics · Checklist

Improving an enterprise assistant starts with a segment and one workflow, not a feature list. I would look for a recurring task where the answer already exists inside the company but is expensive to find and assemble, for example drafting a response against internal policies, and describe a specific user and the point where they get stuck. Then an honest comparison: if search, a template or rules solve it, the model is not needed. For enterprise customers demos do not decide anything, integration and trust do: sign in through corporate accounts, permissions inherited from the source, logging, and requirements on data storage and processing. I would bound the pilot with an outcome announced in advance: time on task, share of tasks finished without rework, quality guardrails, and the condition for expanding. The weakness of pilots is that enthusiasts do not predict behaviour under mandatory rollout.

Read: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications · Rules of Machine Learning

↑ Contents

7. A model offers much greater capability but costs ten times as much. How would you decide what product to build with it?

Product case · AI product strategy and metrics · Checklist

The question is whether gains make previously untrusted tasks acceptable: where mandatory human review was required before and spot checks are now enough. You count not the price of a token but the cost of a successfully completed workflow, including retries, review and the cost of error. Three attempts of a cheap model still cost less in tokens than one of a model ten times the price, so the expensive one wins where the cheap one can't reach the bar at any number of retries, or where human review time and the cost of a mistake dominate the bill. You pay in latency, complexity and dependence on a single provider. I'd measure that cost on my own task set, test willingness to pay on a narrow segment, and turn on selective routing: the expensive model only for hard cases or cases where an error is costly, the cheap one as the base. The limit is that if users don't notice the difference in outcome, no quality gain pays for itself, and prices and capabilities move faster than product plans.

Read: Rules of Machine Learning · Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications

↑ Contents

8. How do you decide whether an LLM is useful for a proposed product problem?

Product case · AI product strategy and metrics · Checklist

An LLM earns its place where inputs are too messy for rules and a mistake is cheap or caught before it does harm, and I establish that from the job, not the model: what the user is trying to do, how it's done today, what an error costs and who sees it. The non-obvious trade-off is coverage against variability: the model handles free text and odd documents that rules can't parse, but the same input can give different outputs, and its failures can't be listed in advance the way a rule's can. Where the value is exact repeatability, the model is at most an interface over a deterministic system. Then I check data access, whether correct can be defined and labelled, and the latency budget. The comparison is the LLM against the best simple baseline on labelled real cases, scoring the error that matters weighted by its cost, plus cost and latency per task, with a rejection rule stated up front. If correct can't be defined or labelled, the answer is no LLM yet: build the measurement first.

Read: Rules of Machine Learning · Building Effective AI Agents

↑ Contents

9. Design a Gemini-based assistant for university students.

Product case · AI product strategy and metrics · Checklist

I would drop the universal helper and pick one segment with one recurring need, say first year students preparing for an exam in a specific subject, and make the course materials the source rather than the whole internet. The flow is built around understanding: the assistant works through the mistake, asks questions back and points to the place in the materials instead of producing the finished assignment; uncertainty is stated out loud, and copying is treated as the main abuse model and limited in the product, with academic integrity rules in mind. I would measure task success, learning value through a delayed knowledge check, and only then retention. That is exactly where the boundary runs: retention can grow from harm, because students love an assistant that does the work for them even though it makes their learning worse, and without a learning metric you will not see it.

Read: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications · AI Risk Management Framework

↑ Contents

10. Users say Gemini gives incorrect answers with excessive confidence. What would you change?

Product case · AI product strategy and metrics · Checklist

First I would stop treating this as one problem and split the complaints by cause: the model does not know the fact, retrieval brought back the wrong thing or nothing, or the answer is presented so that the confidence does not match the grounds. Those are three different fixes. Grounding in sources reduces invention but adds latency and moves quality onto retrieval; abstention can reduce exposure to wrong answers but irritates people when it fires without reason; interface changes like showing citations and marking unverified claims are cheap and shift trust without touching the model. So I would compare those options on the same task set rather than by feel, and watch the share of claims with no source support, useful task completion, and regressions by segment. Calibration aligns confidence with observed correctness; it does not fix factual errors. Abstention and interface choices change exposure, and warnings may be ignored.

Read: AI Risk Management Framework · Demystifying evals for AI agents

↑ Contents

11. How would you prioritise features for an AI writing assistant in Microsoft Word?

Product case · AI product strategy and metrics · Checklist

Prioritisation starts by splitting writing into distinct tasks: drafting from scratch, rewriting an existing passage, shortening, fixing tone and grammar, and merging several documents. For each I'd find the point where the person gets stuck today and compare expected value, quality risk, engineering cost and dependencies such as access rights to the document. The risk here is asymmetric: a suggestion next to the text is cheap when it is wrong, while a sweeping edit of someone else's document is expensive, so tasks with a visible, reversible result on a selected fragment come first. The first release should be narrow, and I'd measure not the volume generated but the assistance a user accepted: the share of inserted text still in the document later, and the share of undos. The limit is that acceptance is not quality, people accept an edit to avoid arguing, so text quality needs a separate check.

Read: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications · Rules of Machine Learning

↑ Contents

12. How would you measure success for an AI feature in a Microsoft product?

Product case · AI product strategy and metrics · Checklist

Success is an observable completed task, and model metrics explain why it moves but never prove value. Take a feature that drafts replies: success is a draft that gets sent with light edits. Then I'd keep the layers apart: adoption as the share of eligible users who tried it, retention as the share still using it weeks later, task quality as the share of drafts sent and how much was edited or discarded, p95 latency, and cost per sent draft, with eval-set scores underneath as diagnosis. The real trade-off is the comparison: a holdout group is the only clean read on cause, but it withholds the benefit and needs weeks to show retention, while before and after is cheap and confounded by novelty and season. Against gaming, every volume metric gets a paired quality metric that falls when volume is inflated: count an impression as usage and adoption rises while the share sent drops. The limit: even a clean experiment misses trust effects that take months, and outputs used outside the product need sampled human rating.

Read: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications

↑ Contents

13. What North Star metric would you choose for an AI search product, and what would constrain it?

Product case · AI product strategy and metrics · Checklist

I would make the headline metric the share of sessions with a usefully completed search task, not clicks and not time on page, because both rise when the answer is bad. Completion has to be observed indirectly: the user did not rephrase, took the next action, and if they opened a source, did not come straight back. In an answer product a click-out alone is ambiguous: it can mean checking the answer or that the answer fell short, and a satisfied user may not click at all, so these signals are calibrated against human labelling. Any single metric pushes you to an extreme, so you add guardrails around it: the share of claims with no source support, safety, latency and cost per query, and a regression in any of them cancels the gain. I would explain the denominator separately: do we count sessions or queries, what we do with bots and repeats, how we segment by language and task type. For navigational and entertainment queries completion is poorly defined, and there the metric misleads.

Read: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications

↑ Contents

14. Notification engagement rises while total time spent remains flat. How would you interpret that result?

Product case · AI product strategy and metrics · Checklist

The first hypothesis is substitution: notifications may not create activity but move it out of organic visits, so total time does not change. But before explaining behaviour I would check the measurement: do the definitions of engagement and time match, did impressions simply grow, are cohorts of new and existing users mixed together, and was this an experiment with a control or a before and after comparison. Then the substantive question: do notifications lead to actions that would not have happened otherwise, or to the same actions just earlier. Look at opt-outs and unsubscribes, complaints, satisfaction and retention over a long horizon, because a short-term rise in clicks is often paid for by losing the channel. Finally, flat time does not by itself mean there is no value: the product may have become more useful in less time, and that is a fine outcome.

Read: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications

↑ Contents

15. Choose an AI agent or product you use and propose its most valuable improvement.

Product case · AI product strategy and metrics · Checklist

Pick a product you use often enough to have watched it fail repeatedly in real work, and choose the failure where your own workaround costs the most. The interviewer is scoring whether your judgement rests on observed friction with a frequency and a cost rather than on a feature you'd like, so say how often it happens, what it costs in time or redone work, and what you do instead. Then show prioritisation: name two or three other improvements and rank them on frequency, cost and how many users plausibly share the problem, admitting where your own usage is unusual. Close with the cheapest test of the hypothesis, often a mock-up or a person simulating the behaviour behind the interface, with one success criterion fixed in advance and the side effect you'd watch, since most fixes cost latency, privacy or extra steps. The usual mistake is a wish list or invented internal metrics. From outside you can't measure the effect directly, so say so and name a proxy you could observe, such as retries or abandoned sessions.

Read: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications

↑ Contents

16. Define a strategy for an internal AI platform, including who owns its success metrics.

Product case · AI platform and operating model · Checklist

An internal platform strategy starts from noticing that several teams are doing the same work: getting access to models, assembling evaluation, setting up logging, dealing with secrets and personal data. The platform takes what repeats and leaves teams what is unique to them, otherwise it becomes a bottleneck. I would split the metrics by level: the platform owns reliability and demand, meaning gateway availability and SLO, time from idea to working prototype, the share of teams that chose it voluntarily, and cost per request; the business outcome stays with the consuming teams, because otherwise nobody owns it. In practice, pick one workflow and support it end to end, say explicitly what the platform does not do, and set a date in advance to review continued investment. If internal teams have no right to choose an alternative, the adoption number stops proving anything.

Read: Site Reliability Engineering · Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications

↑ Contents

17. How would you support multiple model providers when some deployment environments permit only a subset of models?

System design · AI platform and operating model · Checklist

The environment's allowed list must govern model selection. The route is computed from data class, jurisdiction and the policy of the specific environment: the application asks for a capability such as long context or image input, and the platform picks a permitted model and can explain why that one. A portable interface over providers hides the shape of the call, but it must not hide the differences: context length, tool support, structured output format and guardrail behaviour diverge, and believing models are interchangeable turns into regressions in production. So capabilities are declared explicitly and checked with a shared eval set on every allowed route. Fallback is permitted only inside the allowed set, and when no compliant route exists the request is rejected with a clear error instead of quietly going to another provider. The price is less flexibility and constant upkeep of the environment matrix.

Read: Architect multitenant solutions on Azure · AI Risk Management Framework

↑ Contents

18. How would you handle an engineer who is not meeting performance expectations?

Applied scenario · Leading engineering teams · Checklist

I would first establish the expected outcomes for the role and collect specific examples of the gap, rather than treating reputation or one missed deadline as a diagnosis. In a private conversation I would ask for the engineer’s perspective and check whether unclear priorities, missing access, workload or skill gaps are contributing. We would agree what improvement looks like, the support I will provide and a realistic review date. I would record the agreement and give timely feedback against the same criteria, including when performance improves. If the gap persists, I would work with the relevant people partner through the organisation’s established process, considering role fit as well as continued support. I would protect confidentiality and avoid making promises about outcomes I cannot control. The team still needs clear delivery expectations; supporting one person should not mean silently transferring an unlimited workload to colleagues.

Read: [re:Work] Google's New Manager Training Slides

↑ Contents

19. How would you handle a high-performing engineer whose behaviour repeatedly creates conflict with teammates?

Applied scenario · Leading engineering teams · Checklist

I would separate the engineer’s valuable output from the behaviours causing harm, then gather concrete examples and hear the people involved. A technical disagreement is not automatically misconduct; repeated interruptions, withheld information or personal attacks need a different response from a well-founded objection. In private I would describe the observed impact and make clear that collaboration is part of performance, including for the strongest contributor. We would agree specific changes, support such as coaching, and a review date. I would check whether team incentives or unclear decision rights are rewarding the behaviour and correct those conditions too. Follow-up should use observable evidence from delivery and collaboration, without turning colleagues into a public jury. If harmful behaviour persists, I would use the organisation’s established people process. Protecting one person’s output cannot justify accepting an indefinite cost to the rest of the team.

Read: [re:Work] Google's New Manager Training Slides

↑ Contents

20. Describe how someone you mentored progressed in their career.

Behavioral · Leading engineering teams · Checklist

Pick someone whose growth shows in evidence that doesn't rest on your opinion: scope they now own, decisions they make without you, an assessment by people outside your line. Start with their goal and the gap to the next level as it showed in their work, not with your role. Then say what kind of help it was, because mentoring and sponsoring differ: questions and advice that helped them choose their own actions, or putting their name forward for visible work. The interviewer is scoring the trade-off underneath: stretch work can fail in public, and growth comes from handing over the decision, not just the task, so name where you chose not to step in and what that cost, perhaps a slower or rougher result. Split the credit cleanly, their outcomes in their name, your part narrower, and test it with one question: would they describe your support the same way? The usual mistake is a mentor who decided everything. Close with what they handle now that they couldn't then, and what you do differently since.

Read: [re:Work] Google's New Manager Training Slides

↑ Contents

21. What do you do when your team disagrees with your proposed direction?

Applied scenario · Leading engineering teams · Checklist

First I work out what the disagreement is about: facts, a forecast, or priorities. Otherwise the discussion goes in circles. I ask people to name the assumptions my proposal rests on, then take the strongest objection and state it better than its author did: if it falls apart, the direction is firmer; if it doesn't, I'm the one who has to change the decision. Where the argument turns on an unknown quantity, it's cheaper to run a bounded experiment or prototype with an agreed criterion than to keep arguing; the price is time, so I fix the deadline up front. Then I say plainly whose call it is: the discussion is open, the responsibility is mine. The decision, reasons and rejected options go into a design doc with the conditions for revisiting it. This is overkill when a decision is cheap and reversible: then trying is faster than aligning.

Read: Design Docs at Google

↑ Contents

22. Tell me about a difficult organisational change you managed that involved high-performing engineers leaving. How did you respond?

Behavioral · Leading engineering teams · Checklist

Choose a real organisational change where strong engineers actually left, and state what you personally controlled. Explain the purpose of the change and the evidence you had about why people departed; distinguish exit feedback from your interpretation. Describe the effect on the remaining team, including workload, trust and delivery commitments. Then explain your response: conversations you held, commitments you renegotiated, responsibilities you clarified or practices you changed. A retention payment or a new meeting is an action, not proof of success. Give the follow-up evidence you genuinely have, such as subsequent departures, workload indicators or feedback, and acknowledge other factors that could explain it. End with what you would now do earlier or differently. If you have not managed this situation, say so and offer the closest relevant experience, clearly naming the difference rather than inventing departures or claiming authority you did not hold.

Read: Organizational Change Management

↑ Contents

23. How would you stabilise an engineering team after a change in management?

Applied scenario · Leading engineering teams · Checklist

I spend the first weeks listening rather than reforming: one on ones with the team and with key partners, asking what gets in the way today and what people are afraid of tomorrow. A change of manager by itself lowers predictability, so any big early change reads as a threat, even a sensible one. Then I put clarity back: which priorities hold right now, who owns which decisions, which meetings and reports stay and which go away. Urgent risks to delivery and to people I close immediately, without waiting for the full picture. After that I make a small number of changes, each with a stated reason, and check the effect after a fixed period against dates, incidents, attrition and direct feedback. This pace is wrong if the team is already in crisis: there you stop the fall first and listen as you go.

Read: Organizational Change Management

↑ Contents

24. Describe how you identified a valuable opportunity and persuaded a group to deliver it.

Behavioral · Leading engineering teams · Checklist

Build the answer around the opportunity, not around how persuasive you were. Start with what you noticed and why it mattered: which signal in the data, the support tickets or the metrics, what size of effect you expected and on what basis. Then the important part. Everyone you needed already had a plan for the quarter, so say whose priorities you were pushing against, what objections came back, and how the proposal changed because of other people's input; a coalition holds together through co-authorship, not through persuasion. A written document with options, risks and open questions helps here, because people argue with a text rather than with rumours. Then give what was actually delivered and what effect it had, and mark the limits of your influence honestly: where other people decided, and what did not work. The mistake is a story where you convinced everyone at once, with no losses.

Read: Design Docs at Google · Site Reliability Engineering

↑ Contents

25. Walk through the most difficult product launch you led.

Behavioral · Programmes and delivery · Checklist

Pick a launch with a decision you drove that could have gone the other way, hard because of dependencies you didn't control, not volume or long hours. Give the frame briefly, the goal, hard constraints on date, quality or regulation, the critical path, and cut team history, tooling and the full timeline. The interviewer is separating your contribution from the team's, so speak in the first person singular for your own decisions and the plural for the team's work, and expect probing there. Then unpack the one decision: the options, what you knew, what you gave up, and how the people it cost came to accept it. Say how readiness was defined before the date, with launch criteria, a rollback plan and on-call cover. Close with evidence: which criteria were met or missed in the terms promised beforehand, and the single process change you carried into later launches, with a sign it worked. The usual mistakes are an overview where you never decide anything and a victory report where no bet went wrong.

Read: Site Reliability Engineering · Design Docs at Google

↑ Contents

26. Three weeks before launching an LLM feature, evaluation reveals substantial hallucinations on edge cases. What happens next?

Applied scenario · Programmes and delivery · Checklist

The first thing I do is stop discussing the date and establish the shape of the risk: how severe the consequences of this error are, how often it appears on the real distribution of requests, who it hits, and how trustworthy the evaluation itself is, since a small or unrepresentative eval set can both exaggerate a problem and hide it. I compare the options on one scale of harm and cost: narrow the feature to the cases where it's reliable, add human review or an honest refusal, fix the cause in retrieval and prompts, or move the launch. Each option pays something: narrowing costs value, a human in the loop costs speed and money. I name an owner for the launch decision, written exit criteria, and a communication plan for users and support. None of this helps if the edge-case error is irreversible: then there's no launch to discuss.

Read: AI Risk Management Framework · Demystifying evals for AI agents

↑ Contents

27. Describe a large cross-functional project, including ambiguity, roadblocks and delays.

Behavioral · Programmes and delivery · Checklist

What's tested here isn't stamina, it's how you change the way you work as the picture clears. Start with the map: what you promised to deliver, what those deliverables depended on, and who personally owned each dependency, because people are accountable, not teams. Then take one blockage and show the mechanics: which signal let you see it rather than hear about it afterwards, what you changed in the plan, the order of work or the escalation path, and what you gave up to do it. Keep three things apart: what was delivered, what effect it had, and how that compares with the original target; if the target was missed, say so. The common mistake is a long coordination story with no decision in it. Finish with what you now do earlier, which is usually an early test of the most fragile dependency.

Read: Design Docs at Google · Site Reliability Engineering

↑ Contents

28. How would you roll out updates to Alexa devices already deployed in people's homes?

System design · Programmes and delivery · Checklist

Updating home devices means operating in someone else's environment: different models and firmware versions, weak Wi-Fi, a socket switched off halfway through the install. I assess: changes, affected devices, whether protocol compatibility with older versions holds, how large the package is and when to download it so the user isn't disturbed. Then I ship in waves: our own devices, a small group, then wider cohorts with a health check after install and a metric of devices that came back online, not just ones that received the package. The install has to be atomic, with a backup image and an automatic return to the previous version after a failed boot or health check. I set the thresholds for pausing the rollout, the owner of the rollback call, and what support will see. None of this saves you from a defect in the update mechanism itself: that may require a service visit.

Read: Site Reliability Engineering

↑ Contents

29. A customer wants to automate claims processing with AI. What would you do in the first two weeks?

Applied scenario · Applied and customer scenarios · Checklist

Two weeks buy one bounded workflow and a go or no-go, not a prototype of automated claims. The first week I'd spend with handlers, mapping intake, document extraction, coverage check, routing, adjudication and payment, with each step's volume, handling time and exception rate, to see where time actually goes. The first candidate is usually intake and document triage, not adjudication: an extraction error gets caught by the handler downstream and is cheap to reverse, while a wrongful denial lands on the policyholder with legal exposure, so denials stay with a person, and wrongful denial, overpayment and delay each get their own threshold. The second week builds the baseline from past claims: handling time, extraction accuracy on a double-checked sample, the share of simple claims that could go straight through. The pilot then has acceptance criteria per claim type and an owner in claims operations. Without a decision history or access to the claims system, two weeks yield an assessment and a data plan, and saying so early is the result.

Read: Building Effective AI Agents · AI Risk Management Framework

↑ Contents

30. A customer executive wants to cancel an AI pilot because it keeps producing wrong results. What would you do over the next 48 hours?

Applied scenario · Applied and customer scenarios · Checklist

In the first hours I acknowledge the problem plainly, with no explanation of how it happened: trust is gone, and an argument about wording finishes it. I ask for specifics, not "it works badly" but a dozen real cases with the request, the answer they got, and the answer it should have been. That is the only way to separate three different illnesses: data that's missing or dirty, defects in the system itself such as retrieval and prompts, and expectations that never matched what the feature can do. I contain the damage: I switch off the riskiest scenarios and put human review or an honest refusal where confidence is low. By the end of the second day I propose a short recovery stage with a fixed set of cases, a deadline, and explicit continue-or-stop criteria. If the root cause is expectations, closing the pilot is more honest than fixing symptoms.

Read: Site Reliability Engineering · Demystifying evals for AI agents

↑ Contents

31. A customer insists on fine-tuning their own model on support tickets, while you suspect retrieval would be enough. How do you proceed?

Applied scenario · Applied and customer scenarios · Checklist

I don't start with an architecture argument. I start by asking what the customer is trying to solve with their own model: different behaviour and output format, knowledge that isn't in the context, a different cost per request, or control over the weights and where they run. Those four needs are met differently. Fine-tuning covers the first one well and can serve the third by teaching a smaller model the narrow task, but it freezes knowledge at training time, ages along with the support tickets and learns the mistakes support staff made in them, while retrieval over their data updates when someone edits the source. Then I build a representative task set from their own tickets with expected answers, and compare the approaches on it honestly, hybrid included. Before recommending anything I talk through the cost of ownership: labelling, data quality, retraining when the base model changes, evaluation, migration. If the requirement comes from a regulator or from data isolation, arguing about quality is beside the point.

Read: Rules of Machine Learning · Building Effective AI Agents

↑ Contents

32. A deployed agent's base model will be retired in 60 days. How would you migrate it without losing quality?

Applied scenario · Applied and customer scenarios · Checklist

Sixty days is fixed; the hidden dependency is that the evaluation must exist before any candidate is chosen, so the first week builds the measuring instrument, not model trials. An agent's quality belongs to the model together with prompts, tool schemas and parsers tuned to its habits, so a successor regresses not because it's weaker but because it formats tool calls differently, refuses elsewhere or follows instructions more literally. I'd freeze an eval from recorded traffic and known failures and replay it in shadow against candidates, with thresholds per slice rather than one average. The trade-off is depth against a fallback: porting prompts carefully to one successor gives the best quality; keeping a second candidate alive costs more but is the only rollback once the old model is gone. Rollout goes shadow, small share, wider, in staffed hours, with customers told in advance. The boundary: if a slice can't reach parity by an agreed date, I'd decide what to cut and say so, and tasks with no objective check need human review time planned in.

Read: Demystifying evals for AI agents · Site Reliability Engineering

↑ Contents

33. A customer says their eight-GPU chatbot is too slow and too expensive. You have a week with them; how do you use it?

Applied scenario · Applied and customer scenarios · Checklist

I'd start the week with measurements, not ideas: what the real workload profile looks like, how latency is distributed across percentiles rather than averaged, what the response time is made of — retrieval, prefill, decode, external calls — what the utilisation of those eight GPUs is, and what one successfully completed user task costs. The customer's complaint is usually about that last number, not about the price of a token. I profile first and only then pick the lever: batching and serving parameters, a smaller model or quantisation, shorter context and more precise retrieval, streaming and caching for perceived speed. That order holds because serving is often underloaded and tuning it leaves the answers unchanged, while a smaller or quantised model changes them and needs a full quality re-evaluation. Every change gets validated on the same recorded workload with quality and reliability guardrails. If the bottleneck is in the data sources or in the scenario itself, optimising inference won't help.

Read: Site Reliability Engineering · Rules of Machine Learning

↑ Contents

34. An enterprise wants document Q&A over two million internal files within four weeks, with no data leaving its VPC. How would you scope the pilot?

Applied scenario · Applied and customer scenarios · Checklist

Four weeks and two million files is a conversation about choosing, not about coverage. First I narrow it down: which document collections are genuinely needed, who has rights to them today, which services are allowed inside the perimeter, and which fifteen or so employee questions are worth the most. Permissions are not post-processing: the index has to carry labels and retrieval has to filter by the asker's rights, or the pilot becomes a leak path of its own. Then I take a representative slice of the data rather than a random one: it needs current and stale documents, duplicates and mixed formats, because those are what break answer quality. I build the end-to-end path in full, even on a narrow slice. Up front I agree acceptance tests, an owner for operations, and what the pilot will not prove: load, full coverage, and cost at the full volume.

Read: Architect multitenant solutions on Azure · AI Risk Management Framework

↑ Contents

35. A shopping agent can initiate payments. How would you prove that each payment matches the user's actual authorisation?

System design · Payments and regulated domains · Checklist

A payment has to rest on a separate verifiable authorisation, not on what the agent understood the user to mean. Intent is expressed as an explicit object with constraints, amount or limit, payee, currency, validity period and conditions, and it's signed so it can be produced later; that's what AP2 is built on, where signed mandates tie the user's consent to a specific purchase and payment and give a consistent trail. Authority is validated at execution time on the payment side: if the amount or the payee falls outside the grant, you need a new confirmation, not the agent's guess. The audit trail keeps what the user approved, what the agent proposed and what executed. Authorisation evidence and settlement are different things: an authorisation doesn't mean the money arrived. A signature authenticates a signer and unchanged mandate; genuine consent also requires a trusted approval flow. Limits and revocation still matter.

Read: AP2 - Agent Payments Protocol Documentation · Agentic Commerce Protocol · Universal Commerce Protocol (UCP)

↑ Contents

36. An AI support agent initiates a refund, then the payment API times out. How should the system recover without issuing the refund twice?

System design · Payments and regulated domains · Checklist

A timeout does not mean the refund failed, so I'd treat the unknown outcome as a reconciliation problem rather than a reason to repeat the call. Every operation gets a stable identifier created before the first request, and the intent is persisted in our own database before the request goes out, otherwise a retry has nothing to attach to. The retry goes with the same idempotency key under the provider's contract, with exponential backoff and jitter; if the response is ambiguous again, the status is resolved with a status query or by reconciling reports, not with a second refund. You have to account for the finite lifetime of keys, concurrent attempts under one key, and a changed amount, which makes it a different operation. I'd test parallel retries and the manual escalation path. There is no end to end exactly once: this is retry with deduplication and later reconciliation.

Read: Designing robust and predictable APIs with idempotency

↑ Contents

37. A bank wants to roll out an AI assistant that proposes case decisions to employees. What evidence and controls would you require before expansion?

Applied scenario · Payments and regulated domains · Checklist

Before expanding you have to fix which decisions the assistant touches, which customers they affect, and where its authority ends: it proposes, the employee decides, and that has to be true in the interface, not just in a document. I would collect two kinds of evidence: a set of real cases with known correct outcomes, including rare and expensive errors, and the behaviour of employees, the share of suggestions accepted and rejected, and reversals and corrections after acceptance. The mechanism is uncomfortable: a suggestion shifts attention, and agreement quickly becomes a formality, so suggestion quality is measured together with the quality of human decisions. Then you name owners, monitoring, the escalation path and a staged expansion by segment with a stop ready, and the applicable jurisdiction requirements are verified separately with lawyers. Quality on historical cases does not predict behaviour with new products and unfamiliar fraud schemes.

Read: AI Risk Management Framework · Supervisory Letter SR 11-7 on guidance on Model Risk Management

↑ Contents

38. Why do you want to work at Anthropic, and where do you disagree with its approach?

Self-presentation · Behavioral and values · Checklist

Build the motivation on verifiable specifics: name a paper, a product or a company decision you actually studied, and explain how it connects to your work and to what you want to do next. Generic mission statements tell the interviewer little about you. The second half of the question matters more than the first. The disagreement has to be real, with a clear basis, and it must come with an answer to what evidence or results would change your position; that is what separates judgement from a pose. Take the subject from the substance of the work, not a safe detail and not an accusation. Finish with how you would work while holding that disagreement: what you contribute, how you raise the objection internally, and why you carry out a decision that went against you. Refresh the facts from the company's public material before the conversation.

Read: Careers

↑ Contents

39. Describe a mistake you made, its consequences and what you changed afterward.

Behavioral · Behavioral and values · Checklist

Pick a mistake that genuinely cost money, time or trust, or the answer turns into hidden boasting. Start with the decision: what you chose, what you knew at the time, and why it looked reasonable, so the interviewer sees your thinking and not just the outcome. Then the consequences and the recovery, without spreading the blame across circumstances and colleagues: a blameless postmortem means an honest analysis of causes, not a way of dropping your own responsibility. The most common failure in this answer is the line about taking on too much, which dresses a virtue up as a flaw. Finish with a change that stayed with you, a different rule for deciding, a check before release, the habit of writing the postmortem, and with a sign that it worked, for example a similar situation that went differently the next time.

Read: Postmortem Culture: Learning from Failure

↑ Contents

40. Tell me about a time you disagreed with someone and could not persuade them.

Behavioral · Behavioral and values · Checklist

What matters in this answer is what happened after the argument, not the argument itself. Put both positions fairly and in their strongest form: if the other side sounds stupid, the interviewer decides you were listening badly. Give your own evidence and how you put it across, in conversation, in a written document, through data. Then say clearly who made the decision and why it was theirs to make, and describe how you backed the agreed action: you object before the decision, and after it you deliver, without quiet sabotage. Name the result, even if it proved you right, but without any triumph. Mark the limit as well: what you would not have done, and in what situation you would escalate. Finish with the conclusion, usually about what your argument was missing or who you should have gone to earlier.

Read: Leadership Principles

↑ Contents

41. Explain the product trade-offs in choosing a model.

Knowledge · AI product strategy and metrics · Checklist

I'd fix the task and a pass criterion before running anything, on real product examples, because leaderboard gaps can shrink or reverse on your own distribution. For each candidate I'd measure pass rate, cost per passing output including retries and the human time spent fixing failures, and p95 latency end to end with retrieval and tool calls, since that's what the user waits for. The non-obvious trade-off is with reasoning models: they can lift the pass rate and still lose on cost and latency, because their reasoning tokens are billed as output and spent before the answer starts. That can win a batch task and disqualify an interactive one, which is why routing only the hard cases to them often beats choosing one model. Then the constraints act as filters, not weights: where the model may run, what data may be sent and whether the provider keeps it, and how often versions change or retire. If only one model clears them, there's nothing to compare, and the real work is keeping the evaluation runnable for its next version.

Read: Reasoning models

↑ Contents

42. Which business use cases suit AI agents, and how would you compare them?

Product case · AI product strategy and metrics · Checklist

I rank candidates on a few properties rather than on enthusiasm. Volume and repetition, because the payoff is per execution; a verifiable outcome, so success can be checked without a human reading everything; tolerance for error, since an agent will be wrong sometimes and the question is what that costs; and whether the actions are reversible. Support triage, back-office data work and research-and-summarise tasks usually score well; anything irreversible and regulated usually does not. The comparison I insist on is against the cheaper thing: a single prompt, a fixed workflow, or ordinary automation. Autonomy is worth paying for only when the path genuinely varies per case, because agents cost more per task and add supervision, audit and incident work that never appears in the demo. So I would pilot with the hardest measurable slice, and count the human minutes spent correcting it, not just the tasks completed.

Read: Building Effective AI Agents

↑ Contents

43. How would you price an enterprise AI product, which costs must you count, and how does customising it for each client affect its margin?

Product case · AI product strategy and metrics · Checklist

Enterprise AI pricing is a margin question disguised as a value question, so I start with the cost stack per account: inference, including retries and evaluation tokens; retrieval, indexing and storage of their corpus; the human cost of onboarding, support and incidents; and engineering time spent keeping their configuration alive. Then the price metric: seats are predictable for the buyer but decouple revenue from usage, while consumption tracks cost and frightens procurement, so a common answer is a platform fee plus metered usage with a committed floor. Customisation is where margin actually disappears. A bespoke integration or a client-specific eval set is not a one-off but a recurring obligation that every model upgrade reopens, so I would price it separately or refuse it until it can be productised. I would watch gross margin per account rather than blended, because averages hide the accounts that lose money.

Read: What We Learned from a Year of Building with LLMs (Part I)

↑ Contents

44. Where does an AI answer engine outperform traditional search, and where does it fall short? What would you improve first?

Product case · AI product strategy and metrics · Checklist

The comparison only works per task. An answer engine wins where the user's job is synthesis: a question whose answer is spread across several sources, a comparison, an unfamiliar topic they cannot yet phrase as keywords; one pass replaces five tabs. Classic search still wins navigation, where the user knows the destination and a link is faster than a paragraph; queries needing the freshest state; and anything where the user must judge the source themselves, because an answer flattens away who said it and when. The failure mode is confident synthesis over weak evidence, which search never claimed to do. So I would improve attribution first: claim-level citations the user can verify at a glance, and visible recency, measured by the share of sampled claims their citation actually supports, and how often people still reformulate. If the answer is right but unverifiable, enterprise and research users will not trust it.

Read: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

↑ Contents

45. How would you evaluate an enterprise AI assistant when your team cannot inspect customer data?

System design · AI platform and operating model · Checklist

If I cannot see the data, I move the measurement to where the data lives and bring back numbers. The evaluation harness runs inside the customer's environment, computing scores, failure categories and counts, and exports aggregates with a minimum cell size — never examples or transcripts. Labels come from their people, not mine: I supply the rubric and the sampling plan, and I audit agreement between their reviewers, because a rubric applied inconsistently is worse than no rubric. Alongside that I build a proxy set from public or synthetic documents shaped like their corpus, which is what I iterate against daily. The trade-off is honest: proxy data misses exactly the distribution that breaks in production, and aggregates tell me a category is failing without showing me why. So I also negotiate a narrow escalation path — a redacted or customer-approved example when a defect needs diagnosis, rather than standing access.

Read: Demystifying evals for AI agents

↑ Contents

46. An engineering team adopts AI coding tools, but reviewers struggle to verify the resulting changes. How would you build the missing capability?

Applied scenario · Leading engineering teams · Checklist

The gap is not tool operation, it is that review was designed for changes a human drafted incrementally and now receives large, plausible diffs nobody wrote by hand. So I would fix ownership first: the author is accountable for the change regardless of what produced it, and a pull request nobody can explain does not get reviewed. Concretely the author states intent, the tests that pin the behaviour, and what they verified themselves; reviewers are trained to read tests before implementation, to check edge cases and error paths where generated code is weakest, and to reject a change too large to reason about. I would keep changes small and invest in tests as the durable check, since a human cannot out-read the generator; but tests generated with the code can encode the same misreading, so they count only once someone has checked them against the stated intent. I would measure escaped defects, revert rate and review latency, never tool adoption, which says nothing about understanding.

Read: Designing AI resistant technical evaluations

↑ Contents

47. Describe your most recent strategic engineering hire. How did you define the hiring bar and assess whether the hire met the need?

Behavioral · Leading engineering teams · Checklist

Use your most recent real strategic hire and explain the capability the organisation needed, rather than presenting a generic staffing philosophy. State your authority: you may have owned the role, designed the assessment or contributed an interview without making the final decision. Explain why hiring was preferable to training, borrowing expertise or reducing scope in that situation. Describe the hiring bar in terms of work the person needed to perform and the evidence used to assess it consistently. Include a meaningful tradeoff, such as immediate domain knowledge versus the ability to build a new capability, without disclosing candidate details. Then describe onboarding and what you observed after the hire. If the person has only recently joined, say that the strategic outcome is not yet known. If you have no such hire, be explicit and discuss the closest staffing decision without recasting participation as ownership.

Read: [re:Work] Google's New Manager Training Slides

↑ Contents

48. How do you establish engineering quality standards across a team?

Applied scenario · Leading engineering teams · Checklist

A standard that lives in a wiki page is a preference; a standard that lives in the default path is real. So I start from outcomes the team can observe — escaped defects, revert rate, time to restore, review latency, test coverage of the risky paths — and agree what good looks like, otherwise review becomes an argument about style. Then I make the requirement proportionate to risk: a copy change and a change to the payments path should not face the same gate, or people will route around the gate. Whatever we agree goes into tooling, templates, linters and CI, so the easy path is the correct one. I would write it with the senior engineers rather than for them, since an imposed standard gets performed rather than followed, and revisit it against the numbers: a gate that never catches anything is pure cost.

Read: Site Reliability Engineering

↑ Contents

49. Researchers want more exploration while product engineers need a reliable release. How would you structure ownership and handoffs?

Applied scenario · Leading engineering teams · Checklist

These are two different bars, not two levels of rigour, so I separate them structurally. Exploration is judged on how fast the team learns and may be messy, undocumented and abandoned; production is judged on reliability and has an owner who is paged. The boundary is a handoff contract: a stable interface, evaluation evidence on an agreed set, and a named owner who accepts it; nothing enters the release path by being interesting. Behind that boundary the product team keeps versioning, gating and a rollback path, so a research change is a candidate, never a fait accompli. I would keep one shared evaluation suite both sides trust, because these arguments are usually about evidence. The pattern I watch for is researchers on permanent call for their own prototype, which quietly ends exploration; that is the signal to productionise it or retire it.

Read: Rules of Machine Learning

↑ Contents

50. How would you plan an AI programme whose central research hypothesis may fail?

Applied scenario · Programmes and delivery · Checklist

I plan the uncertainty, not the timeline. The first step is to write the hypothesis down precisely enough that it could be false, then order the work so the cheapest falsifying experiment comes first, because a programme that learns its core assumption is wrong in month nine has spent eight months buying nothing. Milestones are therefore defined by which unknown they retire, not by artefacts delivered, and each carries a decision rule agreed in advance: what result continues, what pivots, what stops. Agreeing the stop condition before anyone is invested is the whole trick, since afterwards it becomes a referendum on people's judgement. I would also fund a fallback path in parallel, so there is something to ship if the hypothesis fails, and report progress as uncertainty removed. A programme that reports only percentage complete is not being managed.

Read: Design Docs at Google

↑ Contents

51. Engineering, operations and risk teams disagree about launch readiness. How would you reach a traceable decision?

Applied scenario · Programmes and delivery · Checklist

Disagreement about readiness is usually disagreement about criteria that were never written down, so I stop collecting approvals and start converting objections into testable statements. Each team names what must be true to launch — an error rate below a stated threshold, a runbook exercised, an audit trail retrievable — with the evidence that shows it. Most conflict dissolves here, because operations and risk are not disputing the numbers, they are worried about different failure modes. Then I make decision rights explicit before the room meets: who decides, who must be consulted, and who genuinely holds a veto — one bounded by policy, not by discomfort. Whatever remains unresolved is written as accepted residual risk with a named owner, a mitigation and a review date, so the decision is traceable. A veto with no criterion behind it is the thing to resolve.

Read: AI Risk Management Framework

↑ Contents

52. A contact centre wants voice agents to replace its phone menus. How would you lead the engagement?

Applied scenario · Applied and customer scenarios · Checklist

I would scope by intent rather than replace the menu wholesale. Pull the call-reason distribution and take the few intents carrying most volume with a verifiable resolution; leave the long tail on the existing path, because a voice agent that handles everything badly is worse than a menu. The success measure has to be honest containment — calls genuinely resolved without a human, not calls that ended — and I would track repeat calls within a day, where fake containment shows up. Transfer to a human is a first-class design problem, not a failure branch: it carries the transcript and what was attempted, and never loops the caller back. Then operational readiness before launch: latency targets, interruption handling, monitoring on abandonment, and a rollback to the menu. I would pilot on one intent in one region and expand on evidence.

Read: Building Effective AI Agents

↑ Contents

53. A product team requests an exception to an AI safety control for one enterprise customer. How would you assess, authorise and time-limit the exception?

Applied scenario · Safety, security and governance · Checklist

First I ask what the control actually protects against, because some requests target a control whose purpose nobody remembers, and some are reasonable asks blocked by a clumsy implementation. I also ask who bears the harm if it's relaxed, since that's rarely the team asking. If the underlying risk is real, the answer is usually not a binary exception but a narrower path: a compensating control, a tighter scope, human review in place of the automated block, logging that makes the residual risk observable. Whatever is granted is scoped to that customer and that use, never to the code path generally, because an exception in shared code silently becomes the new default. It gets a named owner on the business side rather than the engineer who implemented it, an expiry date, and a review able to refuse renewal. The failure mode to avoid is the permanent temporary exception, so I would report open exceptions as a standing number.

Read: AI Act

↑ Contents

54. Tell me about conflicting stakeholder priorities and how you established alignment.

Behavioral · Behavioral and values · Checklist

Choose a conflict where the parties were both right, because a story where one side was simply wrong tests judgement about people, not alignment. Name what each stakeholder was measured on — that is usually the real source of the disagreement, and saying it aloud is most of the work: a sales commitment, a reliability target and a compliance deadline can be genuinely incompatible without anyone behaving badly. Then show how you made the trade-off visible: the options, their costs, and whose goal each damaged, rather than hunting for a compromise nobody asked for. Be concrete about the decision and what was given up, including your own priority if that happened. Finish with evidence it held — a written agreement people referred to later, a metric both sides accepted — and avoid the ending where everyone simply agreed once you explained.

Read: Organizational Change Management

↑ Contents

55. Describe feedback that was difficult for you to hear and what you did with it.

Behavioral · Behavioral and values · Checklist

The test is whether you can still hear it, so state the feedback in the words it was actually given. Softened versions — I was told to communicate more — signal that you have not really accepted it. Admitting your first reaction is fine and makes the answer credible, because most people are defensive before they are reflective. What matters is the next step: that you went and checked, by asking others whether they saw the same thing or looking for it in your own record, instead of accepting or rejecting it on one person's word. Then name the specific change, the narrower the better, and a later situation where you can tell it stuck. Avoid two endings — the one where the feedback turned out to be unfair after all, and the one where you simply tried harder — because neither shows anything changed.

Read: [re:Work] Google's New Manager Training Slides

↑ Contents

56. Present a major initiative you led: what worked, what did not, and what you learned.

Self-presentation · Programmes and delivery · Checklist

Pick an initiative where you owned the outcome, not just a workstream, and open with its original goal as it was set — scope, date, a metric — because an honest assessment compares the result with that, not with whatever eventually shipped. Then assess it on two axes. Planning: which assumptions proved wrong, about demand, an estimate or a dependency. Execution: where the team carried out a sound plan badly, or rescued a weak one. The interviewer is listening for whether you can tell the two apart, because they lead to different lessons. Make what didn't work substantive: one decision of your own that you'd now reverse, and what it cost. The usual mistake is a failure section made of things that happened to you, such as a reorganisation or a vendor. If you present with slides, give the failures as much room as the successes, since follow-up questions go wherever you were vague. Close with the result against the original goal, gap included, and a practice you changed afterwards.

Read: Postmortem Culture: Learning from Failure

↑ Contents

57. Tell me about a technical misjudgement that delayed a project.

Behavioral · Programmes and delivery · Checklist

Choose a misjudgement that was technical and genuinely yours: an assumption that a component would scale, that an integration was a week's work, that a data format was stable. What's tested is whether you can locate the faulty assumption precisely, so state it as the sentence you believed at the time and why it was reasonable on what you knew. Then the part most candidates skip: when did the first evidence against it appear, and how long did you take to act on it? That gap is often where most of the delay came from, rather than the original error. Be specific about the cost in time and who absorbed it. The weak ending is adding buffer to estimates, because padding doesn't correct a wrong assumption. A strong one names a change in how you plan: testing the riskiest technical assumption first with a short spike, or writing assumptions into the design doc with a date to check them, and a later project where that caught a problem early.

Read: Postmortem Culture: Learning from Failure · Design Docs at Google

↑ Contents

58. An upstream data team will miss a milestone on your AI programme. How do you revise the plan and communicate the impact?

Applied scenario · Programmes and delivery · Checklist

I'd first pin down what actually slipped, because late data covers very different situations: the whole dataset or one source, labels or raw records, a week or an open-ended delay, and how far I trust the new date given how the team did against the last one. Then I trace it through the critical path. A slip only moves our date if the work that needs the data sits on that path; the evaluation harness, pipeline code and a pilot on a partial or synthetic sample can often proceed, so the real impact may be smaller than the headline, or larger if labelling and training are strictly sequential. I'd bring sponsors options, each with its cost: re-sequence, launch on partial data with a named quality risk, cut scope, or move the date, plus who owns the choice. And I'd replace the single handover date with intermediate checkpoints, such as a schema and a sample delivered early. The boundary: if the slip comes from conflicting priorities, re-planning won't fix it; it goes to whoever sets both teams' priorities.

Read: Dependency Mapping

↑ Contents

59. What is the largest team you have managed, and how did your responsibilities change at that scale?

Self-presentation · Leading engineering teams · Checklist

Give the number precisely and with its structure: how many direct reports, whether some of them were managers, the total headcount, and what the organisation owned. Inflating it with dotted lines or contractors falls apart at the first follow-up. The real question is the second half: what changed in your job as it grew. With a handful of people you can know every design decision; with managers under you, you can't, so name what you stopped doing yourself — reviewing every change, running every incident, interviewing every candidate — and what replaced it: delegated decision rights, written standards, an operating rhythm, skip-level conversations, a few metrics you actually read. The interviewer is listening for whether your mechanisms matched the span, and the common failure is a large team run like a small one, with you as the bottleneck. If your largest team was small, say so and talk about complexity instead: several functions, time zones or stakeholders. Close with a signal that showed the model working or failing, and what you changed in response.

Read: [re:Work] Google's New Manager Training Slides

↑ Contents

60. Describe a team disagreement that you resolved without discarding the valid concerns on either side.

Behavioral · Leading engineering teams · Checklist

Choose a disagreement where both sides were defending something real, say delivery speed against operability, rather than one where somebody was simply wrong, because the question is about keeping valid concerns, not about picking a winner. State each position in its strongest form. The interviewer is listening for the step from positions to needs: what each person feared would happen if the other prevailed. Positions conflict far more often than needs do, and once the needs are visible there's often an option that meets both — a staged rollout, a guardrail with a deadline, a reversible first step. Make clear this isn't splitting the difference, which usually leaves both concerns half addressed; that's the superficial compromise being tested. Say who decided, and what you'd have done if the needs had genuinely conflicted. The usual mistake is casting yourself as the wise mediator whose idea solved everything. Close with evidence: what was built, whether both concerns stayed addressed, and how the two people worked together afterwards.

Read: [re:Work] Google's New Manager Training Slides

↑ Contents

61. Two product teams need shared AI infrastructure, but neither can spare an engineer. How would you establish ownership without creating an unstaffed platform promise?

Applied scenario · Leading engineering teams · Checklist

The thing to prevent is a platform announced before anyone is staffed to run it: teams start depending on it, it degrades, and its failures belong to no one. I'd first define what's actually shared — model access, an evaluation harness, logging with redaction — and what each team would build alone, because a quarter of duplication can be cheaper than a premature shared dependency that couples two roadmaps. If it's genuinely shared, capacity must come from somewhere, and the honest options are few: fund headcount, take an explicit share out of both teams' plans, or make one team the owner, with the other as a customer on written terms. A volunteer working group isn't one of them. Then a narrow charter: the owner, what's in and out of scope, response expectations, how changes are requested, and the trigger for a dedicated team, such as a third consumer or growing support load. If neither team will give up roadmap capacity, that's a priority decision for their common manager, not an org-design problem.

Read: Team Topologies

↑ Contents

62. A clinical AI service handles recordings, transcripts and notes containing protected health information. How does that constrain architecture and use of third-party model APIs?

System design · Payments and regulated domains · Checklist

Protected health information changes the order of work: before choosing a model I'd map every place the data goes, because each copy carries the same obligations. That means the audio, transcripts and generated notes, but also logs, traces, caches, retry queues, evaluation sets and support tickets, which is where it usually leaks. A third-party model API is acceptable only as permitted processing: a contract that explicitly covers health data, limits retention, excludes training on it, and fixes where it's stored and who can access it. Without that the data doesn't go, which may mean a different vendor or a self-hosted model at higher operating cost. Then minimisation: each step receives only what it needs, identifiers are removed where the task allows, and prompts and outputs stay out of general-purpose logging. Redaction tools miss things, so I'd measure their misses on a labelled sample, and test deletion and access auditing before launch. Consent to record and the obligations that apply depend on jurisdiction and each party's role, so counsel confirms them; the architecture can't.

Read: OWASP Top 10 for LLM Applications

↑ Contents

63. Refactor a small, poorly structured program with passing tests. Explain the first change and how you preserve observable behaviour.

Coding · Practical coding · Checklist

Before touching anything I'd settle what observable behaviour means here — return values, error messages, side effects like written files or output order — and whether the passing tests actually assert it. Passing tests only protect what they check, so where coverage is thin I'd first add characterisation tests that record what the code does now, including the odd cases I'd be tempted to fix. The first change should be small, mechanical and easy to revert: usually extracting a function from the longest block or naming a magic value, then running the tests. Each step goes in its own commit, and I never mix a refactoring with a behaviour change, since a failing test then can't say which one broke it; a bug I find gets fixed separately, with its own test. I'd justify each step by a cost it removes, such as a rule duplicated in three places or logic you can't test in isolation, not by taste. If the code is about to be replaced or rarely changes, the cheapest refactoring is none.

Read: Working Inside Existing Systems · Refactoring

↑ Contents

64. Design an agent that creates and updates work orders in a live ERP without duplicating or silently corrupting business operations.

System design · Agents, tools and protocols · Checklist

I'd keep the model off the ERP's write path: it produces a typed command — create or update a work order, with explicit fields — and deterministic code validates it against business rules before any write: the asset exists, the status transition is allowed, no open order already covers this job. Duplicates are prevented by a stable business key derived from the triggering event, not from the model's wording, used as an idempotency key, so a retried tool call or a re-run agent finds the existing order instead of creating a second. Updates carry the record version they read and fail on conflict, because a planner may have edited the order in between, and silent last-write-wins is how corruption goes unnoticed. After a timeout the agent looks up that key before retrying. Each write logs the proposal, the validated command and the ERP's response, and I'd test duplicate events, concurrent edits and injected timeouts. Validation stops illegal states, not plausible wrong ones like a valid order on the wrong asset, so high-impact changes need approval.

Read: Designing robust and predictable APIs with idempotency

↑ Contents

65. What do you do when an MVP of an enterprise AI product is too expensive or too slow to build?

Product case · AI product strategy and metrics · Checklist

An MVP that's too expensive is usually being treated as a small product rather than as an experiment, so I'd go back to what we need to learn. I'd write down the assumptions — the problem is costly enough to pay for, the model reaches the needed quality on this customer's documents, we can actually get the data, the workflow will absorb the output, the buyer can get it through security review — and rank them by uncertainty and cost of being wrong. In enterprise AI the riskiest are rarely the interface; they're data access and quality on real data. Each gets the cheapest test that could kill it: running the model offline on a sample of the customer's data, a concierge pilot where people do the part that isn't built, a design partner committing to a paid pilot. I'd agree beforehand what result justifies the build. The trade-off: these tests prove neither scale nor integration cost; when integration is the main risk, a thin end-to-end slice at one customer is the cheapest honest test.

Read: Rules of Machine Learning

↑ Contents

66. In discovery for an enterprise AI product, who is the persona, where do you find B2B respondents, and how will you tell that a problem is worth solving?

Product case · AI product strategy and metrics · Checklist

In enterprise there's no single persona: there's the user who does the work, the sponsor who owns the budget and the target metric, and the people who can block the purchase — security, legal, IT, the data owner. They want different things, and an AI product often dies at the blockers rather than with users, so discovery has to cover all of them, ideally several roles inside the same account, to see how decisions actually get made. Respondents come from wherever trust already exists: current customers and their account teams, sales conversations, partners and integrators, former colleagues. Cold outreach mostly reaches people with spare time. To judge whether it's worth solving I look past stated pain to behaviour: what they already spend on workarounds — people, spreadsheets, a vendor — how often it occurs, and whether a sponsor will commit something scarce: data access, expert time or a paid pilot. Interest without commitment isn't evidence. And a few large accounts can make a problem look universal, so I'd check it repeats across segments.

Read: How to recruit UX Research participants

↑ Contents

67. Design a moderation system that combines specialist classifiers and LLMs while controlling delay and review workload.

System design · AI system design · Checklist

I'd start from the policy rather than the models: risk classes defined by harm, and for each the relative cost of wrongly removing content against missing a violation, because those tolerances drive every later choice. Severity also decides where moderation sits against the delay budget: the gravest classes are held until checked, while low-harm ones can be published and reviewed asynchronously. Then a cascade. Fast specialist classifiers score everything; confident cases on either side are decided automatically; the uncertain band goes to an LLM that reads the policy text and context, slower and costlier but better on nuance and on policy changes that have no training data yet; only severe cases that stay ambiguous reach people. The trade-off sits in the thresholds: widening the uncertain band improves accuracy but raises latency, cost and review load, so I'd set them per class from precision and recall on a labelled set, and feed appeals and reviewer overturns back as labels. The LLM stage reads content written by adversaries, so it can be manipulated, and the labelled set goes stale as they adapt.

Read: A Holistic Approach to Undesired Content Detection in the Real World · Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

↑ Contents

68. How would you weigh a performance improvement against reduced model interpretability?

Applied scenario · Safety, security and governance · Checklist

I wouldn't weigh it in the abstract, because interpretability isn't one property: the question is what we need explanations for in this use. If a decision has to be explained to the person it affects, contested by them, or validated by an independent reviewer — lending, hiring, clinical triage — losing that can make the better model unusable whatever its accuracy. If it ranks recommendations, explanations mostly serve debugging, and other tools can cover that. So I compare the gain with the consequences of an error and with controls that don't rely on the model's internals: outcome monitoring by segment, human review of high-impact cases, a simpler challenger as fallback. Post-hoc explanations help, but they describe an approximation, not the model's actual computation, so they're weaker evidence than they look. I'd check that the improvement holds on the slices that matter, not just on average, because an opaque model's gain can hide a subgroup getting worse that we can no longer diagnose. Where explanation is a requirement rather than a preference, no performance gain buys it back.

Read: Supervisory Letter SR 11-7 on guidance on Model Risk Management · AI Risk Management Framework

↑ Contents

69. Design a service that lets several teams query a shared dataset without exposing the underlying raw records.

System design · AI platform and operating model · Checklist

The central decision is what counts as a permissible result, because no raw records isn't a policy until it says which outputs are allowed. Per team and purpose I'd define fields, aggregations and a minimum group size below which a result is suppressed. Teams never touch the store: they query a service that authenticates the caller, applies that purpose's policy, rewrites the query to enforce row and column restrictions, and checks the result before release. The non-obvious leak is through aggregates: two allowed queries whose difference isolates one person, or the same small group sliced repeatedly. So I'd add small-cell suppression, limits on overlapping queries, and for the most sensitive data differential privacy, calibrated noise under a privacy budget, which costs accuracy most on small segments. Every query is logged with caller and purpose, and I'd test by trying to re-identify individuals through the allowed interface. If a team genuinely needs record-level analysis, a de-identified extract or a controlled workspace where code runs next to the data is more honest than stretching the aggregate API.

Read: Guidelines for Evaluating Differential Privacy Guarantees (SP 800-226)

↑ Contents

70. An enterprise customer says Claude hallucinates in their retrieval-based knowledge assistant. How would you investigate the first failures?

Applied scenario · Applied and customer scenarios · Checklist

Hallucination is a symptom, so I'd begin by getting concrete cases, a few dozen if they exist, each with its full trace: the question, what retrieval returned, the assembled prompt and the answer. Without traces the conversation is about impressions. Then I sort every case into one of three buckets, because each has a different fix. Retrieval failure: the right passage existed but wasn't returned, or a stale or contradictory document was. Unsupported generation: the right passage was in context, yet the answer added to it or contradicted it. Expectation mismatch: the corpus doesn't contain the answer, or the user wanted judgement rather than lookup; even then, filling the gap instead of saying the sources don't cover it is part of the defect. I'd show the customer the split with counts, fix the dominant bucket first, and keep the cases as a regression set. The limit is that escalated examples overrepresent vivid failures, so before claiming improvement I'd also measure a random sample of ordinary traffic.

Read: Your AI Product Needs Evals

↑ Contents

71. You own the next quarter’s AI investment portfolio. A 30% budget cut affects a revenue pilot, a mandatory risk-control project and shared evaluation infrastructure. How would you reallocate money and people and secure executive agreement?

Applied scenario · AI platform and operating model · Checklist

I would first confirm what the 30% cut applies to and which commitments cannot be changed within the quarter. A mandatory control needs a viable minimum scope; its label does not justify every proposed feature. I would compare portfolio options using remaining cost, evidence of value, uncertainty, dependencies and the actual skills available. Shared evaluation may enable both other projects, so cutting each budget equally could destroy more value than stopping one initiative. I would propose a specific allocation with named work to stop or defer, transition costs and owners for affected commitments. The executive discussion should expose the foregone opportunity and residual risk, identify who can accept each tradeoff and produce a recorded decision. I would release discretionary funding in stages against evidence and set a review trigger. Past spending explains the situation but does not, by itself, justify further investment.

Read: The Green Book (2026) · Product Management - Leadership

↑ Contents

72. As the investment owner, decide whether to renew a managed AI service or fund an internal replacement. The vendor’s price doubles in six months and exported data alone cannot reproduce its behaviour. What decision and exit plan would you approve?

Applied scenario · AI platform and operating model · Checklist

I would compare renewal, renegotiation, partial replacement and internal build over the same planning period. The estimate must include hiring or displaced engineering work, evaluation, integration, operations and a period of parallel running. I would ask procurement to establish the real notice and renewal dates, while the technical team tests the hardest portability assumption on representative workflows. Data export is insufficient if prompts, retrieval behaviour, tooling or evaluation assets cannot be recreated. A replacement must meet agreed task-quality and operational thresholds before customers depend on it. I would approve the option with the best supported value under these constraints, which may be a temporary renewal while reducing dependency. The decision includes an owner, reserved migration capacity, spending limits and a continuity fallback if acceptance fails. An exit plan that has neither funded people nor a demonstrated first migration is only an intention.

Read: Managing technical lock-in in the cloud · Cost Optimization Pillar — AWS Well-Architected Framework

↑ Contents

73. You manage three engineering managers delivering a shared AI initiative. Two blame each other for an evaluation gap while the executive sponsor has promised a launch date. How would you restore accountability without taking over their teams?

Applied scenario · Leading engineering teams · Checklist

I would establish which evaluation outcome is missing, what was previously agreed and which decisions each manager could actually make. The immediate objective is to close the risk, but taking over every task would hide a broken management interface. I would name an accountable owner for the shared outcome, agree contributions and escalation conditions with the managers, and leave team planning and execution with them. In parallel, I would take the sponsor a small set of evidenced options: narrow the launch, move the date or accept only risks that the authorised owner can legitimately accept. The promise is a constraint to renegotiate, not proof of readiness. I would coach each manager privately on their part in the conflict and change incentives or boundaries that encouraged it. Success includes the next cross-team decision being resolved through the agreed structure without my intervention.

Read: Director, Engineering · Communication · Team Topologies

↑ Contents

74. An AI product has rising usage but no credible improvement in customer outcomes; review costs keep increasing. The sponsor wants another quarter of investment. What evidence would make you expand, narrow, pause or stop it, and how would you handle existing users?

Product case · AI product strategy and metrics · Checklist

I would separate usage from incremental customer value and calculate the cost of delivering a successful outcome, including human review, support and rework. The comparison needs a credible existing workflow or non-AI baseline and enough attention to selection effects to avoid crediting the product for already successful users. I would look for a segment where the benefit is real before treating the average as a universal failure. If one remaining uncertainty could change the decision, I would propose a bounded experiment with a spending cap, deadline and thresholds agreed with the decision owner. Otherwise, another quarter simply postpones the choice. Stopping requires a plan for current users: notice, an alternative workflow, support ownership and appropriate data handling. I would record what evidence would justify restarting. The objective is to preserve customer value and learning while ending unsupported expenditure, not to defend the original launch.

Read: Patterns of Trustworthy Experimentation: Pre-Experiment Stage · Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications · User Needs + Defining Success

↑ Contents

75. During an AI rollout, a provider changes model behaviour and harmful actions reach customers. The previous model cannot be restored immediately. As the programme lead, how do you coordinate containment, customer response and a defensible restart?

Applied scenario · Programmes and delivery · Checklist

I would activate the incident process and ensure there is one incident commander with clear operational authority. My programme role is to coordinate dependencies and decisions, not to improvise commands outside that structure. If the old model is unavailable, containment may mean disabling actions, limiting traffic or moving customers to a verified manual workflow. I would preserve model and configuration versions, action traces and the affected population while customer owners establish an accurate communication cadence. Provider escalation should seek a bounded remedy without making recovery promises we cannot support. Containment does not establish recovery: the proposed configuration needs representative regression and safety checks, capacity for the degraded path and an authorised restart decision. I would restore scope gradually with stop thresholds and named monitoring owners. Afterwards, the review should address provider-change detection, action reversibility and continuity planning, with follow-up work owned and scheduled.

Read: Site Reliability Engineering · Postmortem Culture: Learning from Failure · AI Risk Management Framework

↑ Contents

76. Candidates now use AI assistants during technical interviews. How would you redesign your loop so that it still produces a usable hiring signal?

Applied scenario · Leading engineering teams · Checklist

I would start with the work the hire must perform: framing a problem, checking an unfamiliar change, diagnosing failure and explaining a decision. Then I would identify which existing tasks now mostly measure access to an assistant. Each stage needs an explicit policy: tools may be permitted for realistic work, restricted for a particular independent skill, or supplied in a controlled environment. Candidates should know the policy beforehand and have comparable access. I would score observable reasoning, verification, corrections and ownership, using a follow-up change that exposes whether the candidate understands the result. Before rollout, current engineers and trained interviewers should try the tasks and reconcile scoring disagreements. I would monitor candidate experience and later work outcomes, while acknowledging that small, selectively hired samples cannot establish predictive validity. The goal is a defensible hiring decision, not winning an arms race with assistants.

Read: Designing AI resistant technical evaluations · AI Interview Success: An Interviewer's Inside Guide

↑ Contents

77. Several product teams share one internal AI gateway. How would you keep one team's traffic, data and caches from affecting another's?

System design · AI platform and operating model · Checklist

I would decide isolation per shared resource rather than once for the gateway. The resources are provider quota, the gateway's own capacity, prompt and embedding caches, request logs and evaluation data. Each tenant gets its own credentials, quota and rate limit, so a retry storm in one team degrades only that team; without per-tenant limits one incident consumes the shared quota. Caches are keyed by tenant by default. A shared prefix cache does not hand one tenant another's text, but a cache hit is faster than a miss, so timing reveals whether a prefix has been seen before — worth closing where prompts are themselves sensitive. Logs carry prompts and completions, so I would scope read access per tenant and set retention, while aggregate cost and latency telemetry stays central. I would test isolation by loading one tenant and watching the others. Strict isolation costs cache hit rate and money; where every tenant is one legal entity under one data policy, I would say so and relax it deliberately.

Read: Architect multitenant solutions on Azure · Prompt caching · Scaling your API with rate limiters

↑ Contents

78. A provider announces that a model version your products depend on retires in three months. How would you run the migration across the teams that use it?

Applied scenario · AI platform and operating model · Checklist

The deadline is fixed and the risk is schedule, not technology, so I would start with an inventory: which services, prompts, evaluation baselines and tuned models name that version, and what each depends on. A model version is a dependency, and treating it as a configuration value is how a quiet regression ships. Each owning team re-runs its evaluation against the replacement before anything moves, because a new version changes behaviour: prompts, output formats and thresholds usually need work, and that work is part of the migration rather than a follow-up. I would move services one at a time behind a flag while the old version still answers, so rollback is real rather than theoretical, and I would keep the riskiest and the highest-traffic cases apart. What I would leave behind matters as much: pinned versions, an inventory that stays current and an owner per integration. This is harder where a tuned model has no equivalent on the new version; then retraining, not migration, is the real plan.

Read: Managing technical lock-in in the cloud · Rules of Machine Learning

↑ Contents

79. Inference spend on a shared AI platform doubles in a quarter and no team recognises the bill as theirs. How would you make the cost attributable and controllable?

Applied scenario · AI platform and operating model · Checklist

Nobody owns a number they cannot see, so attribution comes before control. I would tag every call with team, feature and environment at the gateway, then report cost per request and cost per successful task alongside the total; the second unit is the honest one, because a cheaper model that fails more often can raise the bill. Then I would find what moved. Doubling has different causes — more traffic, longer prompts or retrieved context, a switch to a more capable model, retries after timeouts, or evaluation runs charged to production — and each has its own remedy: caching a stable prefix, trimming context, routing easy work to a smaller model, or fixing the retry policy. Each team then gets a budget and an alert it owns, with a documented way to ask for more. I would avoid a hard cap with no exception path: it does not remove the demand, it moves it to accounts nobody is watching. Showback is usually enough; chargeback is worth it only where teams genuinely control their own volume.

Read: Cost Optimization Pillar — AWS Well-Architected Framework · Prompt caching

↑ Contents

80. Your AI platform's latency and availability depend on an external model provider. What would you promise internal teams, and how would you keep that promise?

System design · AI platform and operating model · Checklist

I would promise an end-to-end objective the platform can actually hold, stated in what a caller feels: latency at a high percentile and the share of requests that finish usefully, measured at our edge rather than taken from the provider's uptime page. Availability I do not control is not a promise I can make, so the objective rests on the controls I own — a timeout that matches the caller's deadline, a bounded retry budget, a smaller fallback model, a degraded but honest answer, and a queue for work that can wait. Retries deserve care: naive retries during provider overload add load to a system already failing. The fallback changes answer quality, so it needs its own evaluation and the caller needs to know which path served them. I would publish an error budget, say what happens when it is spent, and name the failure modes that stay outside my control. Where a team needs a stronger guarantee than that, the answer is a second provider or capacity we reserve, and both cost money.

Read: Site Reliability Engineering · Scaling your API with rate limiters

↑ Contents

81. Before launching an assistant that can use tools, how would you organise adversarial testing, and what would a clean result let you claim?

System design · Safety, security and governance · Checklist

I would start from the harms rather than the techniques, and from two different attackers: a user trying to make the product do something it should not, and untrusted content the model reads — a web page, a document, a tool result — that carries instructions. The second matters more once the assistant has tools, because the damage is an action, not a sentence. Coverage comes from mixing automated generation of many variants with people who did not build the system, and from logging every attempt rather than the successes alone, so the suite can be re-run against the next version and coverage can be argued. Findings go to named owners with fixes, and I would distinguish blocking a class of attack from one phrasing that stopped working. The honest claim at the end is narrow: the attacks we tried, at this version, mostly failed. That is not a safety proof — prompt injection has no complete fix today — so it has to sit on top of least privilege and confirmation for consequential actions.

Read: OWASP Top 10 for LLM Applications · AI Risk Management Framework · Prompt injection (series)

↑ Contents

82. Your organisation adopted AI coding tools and leadership wants to know whether they worked. How would you answer that credibly?

Applied scenario · Leading engineering teams · Checklist

I would first make the question answerable, because "did they work" hides four different claims: faster delivery, more change shipped, better quality, or less drudgery. They need different evidence and can move in opposite directions — more code merged with a higher change failure rate is not a win. I would report outcomes a customer feels, such as delivery lead time and change failure rate, rather than accepted suggestions or lines generated, which measure adoption and rise whenever people use the tool. Any number needs a comparison: a staged rollout gives the cleanest one, otherwise comparable teams, or before and after with the confounders named — hiring, reorganisation, seasonality, a big migration. I would also measure where the effort moved, since generated code shifts load onto review, and a gain in authoring that reappears as review backlog is not a gain. Developer surveys are weak evidence of time saved but good evidence of where friction is. I would state plainly what this design cannot settle rather than imply causation.

Read: Patterns of Trustworthy Experimentation: Pre-Experiment Stage · Rules of Machine Learning

↑ Contents