AI system design

English · Русский · AI Interview Atlas › Themes

End-to-end design of AI products and platforms, explained aloud with requirements, trade-offs and failure modes.

On this page: Both tracks (5) · AI Engineering (11)

Both tracks

• Design a knowledge assistant over ten million enterprise documents with per-user permissions and a continuously changing corpus.
  • System design · Asked at: Amazon †, Databricks †, Microsoft †, OpenAI † · ✍ Answer · Suggest a fix
  • Tests: Whether ingestion, retrieval, access control and serving form an operable system.
  • A strong answer covers:
    • Clarify query volume, freshness and permission semantics before sizing.
    • Separate ingestion and index versions from query-time authorisation and retrieval.
    • Evaluate grounded answers and test revocation, stale data and partial outages.
  • Read: Document-level access control (Microsoft Learn) · Contextual Retrieval in AI Systems (Anthropic)
• Design a customer-support agent that can execute service actions and transfer the case to a human when needed.
  • System design · ✍ Answer · Suggest a fix
  • Tests: Whether execution authority and human escalation are part of the architecture.
  • A strong answer covers:
    • Map supported intents to explicit tools and permission boundaries.
    • Use observed operation outcomes and persistent case state to prevent duplicate actions.
    • Define escalation triggers and transfer enough context for a human to continue.
  • Read: Building Effective AI Agents (Anthropic) · Writing effective tools for AI agents—using AI agents (Anthropic)
• Design a moderation system that combines specialist classifiers and LLMs while controlling delay and review workload.
  • System design · Asked at: Meta † · ✍ Answer · Suggest a fix
  • Tests: Whether risk classes, escalation and false decisions shape system design.
  • A strong answer covers:
    • Define risk classes by harm, with separate tolerance for wrongful removal and missed violations per class.
    • Cascade: fast classifiers on everything, an LLM on the uncertain band, people on severe ambiguous cases.
    • Set thresholds per class from measured precision and recall, tracking reviewer queue, appeals and overturns.
  • Read: A Holistic Approach to Undesired Content Detection in the Real World (Markov et al., arXiv) · Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (Inan et al., arXiv)
• Design an LLM gateway with provider routing, failover, caching, rate limits and enforceable spending budgets.
  • System design · Asked at: Palantir †, Perplexity † · ✍ Answer · Suggest a fix
  • Tests: Whether provider differences and failure handling preserve the service contract.
  • A strong answer covers:
    • Define provider capabilities and normalise only compatible request semantics.
    • Enforce tenant quotas and budgets before dispatch, including retries.
    • Specify failover boundaries for partially streamed responses and side-effecting tool calls.
  • Read: Scaling your API with rate limiters (Stripe) · Designing robust and predictable APIs with idempotency (Stripe)
• Design a secure developer API for model access, including authentication, quotas, streaming responses and backwards compatibility.
  • System design · Asked at: Anthropic † · ✍ Answer · Suggest a fix
  • Tests: Whether a model endpoint is treated as a durable platform contract.
  • A strong answer covers:
    • Authenticate per key with scopes, and meter tokens and concurrency rather than requests alone.
    • Stream incrementally with request ids, and define client behaviour on mid-stream failure.
    • Evolve the contract additively, with versioning, deprecation windows and published limits.
  • Read: Scaling your API with rate limiters (Stripe)

AI Engineering

• Design a repository-aware coding assistant, from indexing and context assembly to applying edits and checking results.
  • System design · Suggest a fix
  • Tests: Whether code context, safe edits and task evaluation are connected.
  • A strong answer covers:
    • Decide how context is found: a symbol graph and search on demand, or a precomputed embedding index.
    • Apply edits as diffs against the file version that was read, in an isolated workspace, with user review.
    • Close the loop with the repository's build, linters and tests; evaluate on historical tasks with known fixes.
  • Read: Effective context engineering for AI agents (Anthropic) · Building Effective AI Agents (Anthropic)
• Design semantic search for a large product catalogue, including relevance measurement and rapidly changing inventory.
  • System design · Suggest a fix
  • Tests: Whether semantic relevance is combined with valid catalogue constraints.
  • A strong answer covers:
    • Combine lexical matching for brands, sizes and SKUs with dense retrieval for descriptive queries.
    • Keep stock and price out of embeddings; apply them as query-time hard filters from a fresh store.
    • Measure relevance on judged queries by segment, and online track zero-result rate and out-of-stock items shown.
  • Read: Contextual Retrieval in AI Systems (Anthropic)
• Design a pipeline that extracts structured fields from ten million scanned documents and makes uncertain results reviewable.
  • System design · Asked at: Palantir † · ✍ Answer · Suggest a fix
  • Tests: Whether extraction quality, provenance and scalable processing are planned together.
  • A strong answer covers:
    • Process documents as idempotent jobs, storing each field with page, region, model version and confidence.
    • Route fields with low calibrated confidence or failed validation rules to review, sized to reviewer capacity.
    • Measure per-field accuracy on a stratified audit sample, and feed reviewer corrections back into evaluation.
  • Read: Interpret and improve model accuracy and confidence scores (Microsoft Learn) · Structured outputs (Anthropic)
• Design natural-language querying over a warehouse with thousands of tables, from schema selection to safe query execution.
  • System design · Asked at: Databricks †, Palantir † · ✍ Answer · Suggest a fix
  • Tests: Whether semantic correctness and query permissions constrain generation.
  • A strong answer covers:
    • Retrieve a small candidate schema from catalog metadata, governed metric definitions and verified example queries.
    • Run generated SQL read-only under the requesting user's own warehouse permissions, with cost, row and time limits.
    • Compare result sets with verified queries on real questions, and measure schema-retrieval recall separately.
  • Read: OWASP Top 10 for LLM Applications (OWASP Gen AI Security Project) · Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (Lei et al., arXiv)
• Design a meeting assistant that records discussion, distinguishes speakers and publishes summaries and action items to connected tools.
  • System design · Asked at: Microsoft † · Suggest a fix
  • Tests: Whether audio processing, attribution and integration reliability fit together.
  • A strong answer covers:
    • Separate transcription from diarisation; map speakers to people using per-participant audio channels where available.
    • Link each action item to its transcript span, owner and confidence before anything is published.
    • Publish with an idempotency key per item so retries never duplicate tasks, and let users confirm first.
  • Read: Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., arXiv) · Designing robust and predictable APIs with idempotency (Stripe)
• Design a streaming chat service for hundreds of millions of users, including capacity, conversation storage and graceful overload behaviour.
  • System design · Asked at: Anthropic †, Google and Google DeepMind †, Meta †, OpenAI † · ✍ Answer · Suggest a fix
  • Tests: Whether scale assumptions lead to explicit capacity and reliability decisions.
  • A strong answer covers:
    • Estimate concurrent streams, token demand and storage from declared traffic assumptions.
    • Separate admission control, inference scheduling and durable conversation state.
    • Plan capacity headroom, cancellation, overload rejection and recovery from partial failure.
  • Read: Site Reliability Engineering (Google) · Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., arXiv)
• How would you support multiple questions within one conversation while keeping turn ordering and shared context consistent?
  • System design · Asked at: Anthropic † · Suggest a fix
  • Tests: Whether concurrent conversation updates have clear consistency semantics.
  • A strong answer covers:
    • Decide the semantics first: serialise turns per conversation, or fork branches that each see a snapshot.
    • Append turns with a per-conversation sequence number and an idempotency key, rejecting stale writes.
    • Test interleaved requests, retries of the same turn and cancellations for duplicated or reordered history.
  • Read: Designing robust and predictable APIs with idempotency (Stripe)
• Design distributed search for a billion documents at a million queries per second; state your partitioning and capacity assumptions.
  • System design · Asked at: Anthropic † · Suggest a fix
  • Tests: Whether indexing, fan-out and tail latency are grounded in estimates.
  • A strong answer covers:
    • Estimate index bytes per document and queries per shard to derive shard and replica counts.
    • Choose document versus term partitioning, knowing that fan-out to every shard amplifies tail latency.
    • Contain tails with hedged requests and partial results, and test p99 under shard failure and reindexing.
  • Read: The Tail at Scale (Dean and Barroso, Google Research)
• Design reliable delivery of events to customer webhook endpoints despite timeouts, duplicate attempts and unavailable receivers.
  • System design · Asked at: OpenAI † · Suggest a fix
  • Tests: Whether delivery guarantees and consumer responsibilities are stated precisely.
  • A strong answer covers:
    • State the guarantee as at-least-once and unordered, and require consumers to deduplicate by a stable event id.
    • Write events to a durable outbox in the same transaction, then retry with capped backoff and jitter.
    • Isolate failing endpoints in per-endpoint queues, disable after a retry window, and support replay.
  • Read: Designing robust and predictable APIs with idempotency (Stripe) · Standard Webhooks specification (Standard Webhooks)
• Critique the weaknesses of an existing production service architecture, then design a new product feature on top of it.
  • System design · Asked at: LangChain † · Suggest a fix
  • Tests: Whether you reason about a real, imperfect system rather than a blank page.
  • A strong answer covers:
    • Read the system as it runs: request path, data ownership, failure modes and who operates it.
    • Rank weaknesses by the risk they pose to the new feature, not by how ugly they look.
    • Design the feature within existing constraints, naming the minimal changes it forces and a safe migration.
  • Read: Working Inside Existing Systems (Palantir Careers)
• Design an ML system for a product feature end to end: frame the task and requirements, decompose it, plan data collection, choose model architectures, then deploy and test it.
  • System design · Asked at: T-Bank ✅ · Suggest a fix
  • Tests: Whether you structure an ML system from problem framing to deployment instead of jumping to a model.
  • A strong answer covers:
    • Frame the product decision, offline and online metrics, and latency and cost limits before any model.
    • Decompose into subtasks, plan where labels come from, and beat a heuristic baseline before adding models.
    • Plan deployment through shadow mode or an A/B test, with drift monitoring and a rollback.
  • Read: Rules of Machine Learning (Google for Developers)

← Multimodal and voice · Practical coding →