Skip to content

AI platform and operating model

English · Русский · AI Interview Atlas › Themes

Internal AI platforms, adoption across an organisation, enablement, cost governance and centres of excellence.

On this page: Both tracks (6) · AI Leadership (4)

Both tracks

  • How would you support multiple model providers when some deployment environments permit only a subset of models?
    • System design · Senior · Asked at: Palantir † · ✍ Answer · Suggest a fix
    • Tests: Whether a platform preserves deployment constraints and makes capability gaps visible.
    • A strong answer covers:
      • Make environment policy and data restrictions explicit in model selection.
      • Define a portable interface without pretending all models have identical capabilities.
      • Evaluate allowed fallbacks and fail closed when no compliant route exists.
    • Read: Architect multitenant solutions on Azure (Microsoft Learn) · AI Risk Management Framework (NIST)
  • How would you evaluate an enterprise AI assistant when your team cannot inspect customer data?
    • System design · Senior · ✍ Answer · Suggest a fix
    • Tests: Whether you can obtain useful quality evidence within data-access restrictions.
    • A strong answer covers:
      • Move measurement into the customer's environment and return aggregates, not their content.
      • Have the customer's own reviewers label a sample against a rubric you supply and audit.
      • Build a public or synthetic proxy set shaped like their tasks, and state what it cannot show.
    • Read: Demystifying evals for AI agents (Anthropic)
  • Design a service that lets several teams query a shared dataset without exposing the underlying raw records.
    • System design · Senior · Asked at: Palantir † · ✍ Answer · Suggest a fix
    • Tests: Whether you define permissible results and enforce access across a shared platform.
    • A strong answer covers:
      • Define permissible results per team and purpose: fields, aggregations and a minimum group size.
      • Enforce policy in the service: caller identity, query rewriting, row and column controls, output checks.
      • Defend against differencing and repeated slicing, log every query, and test with re-identification attempts.
    • Read: Guidelines for Evaluating Differential Privacy Guarantees (SP 800-226) (NIST)
  • Several product teams share one internal AI gateway. How would you keep one team's traffic, data and caches from affecting another's?
    • System design · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether isolation is decided per shared resource instead of being assumed from the fact that the platform is managed centrally.
    • A strong answer covers:
      • Name the shared resources — provider quota, gateway capacity, prompt and embedding caches, logs and evaluation data — because each needs its own isolation decision.
      • Give each tenant its own credentials, quota and rate limit, and key caches by tenant unless the cached content is demonstrably public.
      • Decide what may cross the boundary: aggregate cost and latency telemetry may, prompts and completions may not; scope who can read traces, and for how long.
    • Read: Architect multitenant solutions on Azure (Microsoft Learn) · Prompt caching (Claude Platform Docs) · Scaling your API with rate limiters (Stripe)
  • A provider announces that a model version your products depend on retires in three months. How would you run the migration across the teams that use it?
    • Applied scenario · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether a model version is treated as a dependency with a lifecycle rather than a setting that can be swapped.
    • A strong answer covers:
      • Find the real usage first: which services, prompts, evaluations and tuned models are pinned to that version, and which behaviour each one relies on.
      • Re-run each team's evaluation on the replacement before anything moves; a model change is a behaviour change, so prompt and threshold work belongs inside the migration.
      • Sequence the move behind a flag while the old version still answers, so rollback is real, and leave behind the pinning and inventory the next deprecation will need.
    • Read: Managing technical lock-in in the cloud (Government Digital Service and Central Digital and Data Office) · Rules of Machine Learning (Google for Developers)
  • Your AI platform's latency and availability depend on an external model provider. What would you promise internal teams, and how would you keep that promise?
    • System design · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether a service level comes from measured behaviour and controls you own, rather than from a provider's published figure.
    • A strong answer covers:
      • Measure what a user feels — end-to-end latency at a high percentile and the share of requests that finish usefully — not the provider's own uptime number.
      • Promise only what your own controls can hold: timeouts, a retry budget, a smaller fallback model, a degraded answer, and a queue for work that can wait.
      • Publish the error budget and what happens when it is spent, and say plainly which failure modes remain outside your control.
    • Read: Site Reliability Engineering (Google) · Scaling your API with rate limiters (Stripe)

AI Leadership

  • Define a strategy for an internal AI platform, including who owns its success metrics.
  • You own the next quarter’s AI investment portfolio. A 30% budget cut affects a revenue pilot, a mandatory risk-control project and shared evaluation infrastructure. How would you reallocate money and people and secure executive agreement?
    • Applied scenario · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether you exercise portfolio authority, distinguish constraints from preferences and make the cost of displaced work explicit.
    • A strong answer covers:
      • Establish the budget boundary, non-negotiable obligations, committed spend and decision authority.
      • Compare complete options with ranges for value, remaining cost, dependencies and people capacity; exclude sunk cost from the forward choice.
      • Fund, narrow, defer or stop named work; record executive tradeoffs, transition costs and conditions for revisiting the allocation.
    • Read: The Green Book (2026) (HM Treasury) · Product Management - Leadership (GitLab Handbook)
  • As the investment owner, decide whether to renew a managed AI service or fund an internal replacement. The vendor’s price doubles in six months and exported data alone cannot reproduce its behaviour. What decision and exit plan would you approve?
    • Applied scenario · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether build-versus-buy includes organisational capacity, contract timing, validated replacement quality and a funded exit rather than just model cost.
    • A strong answer covers:
      • Compare renewal, renegotiation, partial replacement and internal build over a common period, including staff and transition costs.
      • Inventory portable data, evaluations, integrations and undocumented behaviour; demonstrate the hardest replacement dependency.
      • Approve an accountable option, reserve exit capacity and define contract dates, acceptance gates and a continuity fallback.
    • Read: Managing technical lock-in in the cloud (Government Digital Service and Central Digital and Data Office) · Cost Optimization Pillar — AWS Well-Architected Framework (Amazon Web Services)
  • Inference spend on a shared AI platform doubles in a quarter and no team recognises the bill as theirs. How would you make the cost attributable and controllable?
    • Applied scenario · 🧪 generated from job-posting themes · ✍ Answer · Suggest a fix
    • Tests: Whether spend is connected to a unit someone owns, and whether control changes behaviour without stopping useful work.
    • A strong answer covers:
      • Attribute before you limit: tag every call with team, feature and environment, and report cost per request and per successful task, not only the monthly total.
      • Find what actually moved — traffic, longer context, a model change, retries or evaluation runs — because each has a different remedy.
      • Give each team a budget and an alert it owns, and keep a route for deliberate overspend; a hard cap with no exception pushes work onto unmanaged accounts.
    • Read: Cost Optimization Pillar — AWS Well-Architected Framework (Amazon Web Services) · Prompt caching (Claude Platform Docs)

← AI product strategy and metrics · Leading engineering teams →