RAG and retrieval

English · Русский · AI Interview Atlas › Themes

Grounding answers in documents: chunking, hybrid search, reranking, permissions, freshness and citation.

On this page: Both tracks (2) · AI Engineering (12)

Both tracks

• When should a retrieval application plan multiple searches instead of making one fixed retrieval call?
  • Knowledge · Suggest a fix
  • Tests: Whether iterative retrieval earns its extra cost and failure modes.
  • A strong answer covers:
    • Iterate when later queries depend on earlier findings: multi-hop questions, cross-entity comparisons or underspecified requests.
    • Weigh added latency, tokens and compounding errors such as query drift and loops against the measured recall gain.
    • Compare single-shot and iterative retrieval per question type on accuracy, evidence recall, calls per answer and cost.
  • Read: ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., arXiv) · How we built our multi-agent research system (Anthropic)
• Design retrieval that enforces the source system's access rights, including permission changes and shared caches.
  • System design · Asked at: Databricks †, Microsoft †, Palantir † · ✍ Answer · Suggest a fix
  • Tests: Whether access checks cover the full path to generated output.
  • A strong answer covers:
    • Propagate identity and enforce authorisation before content reaches the model.
    • Keep permission updates, deletions and cache keys consistent with the policy.
    • Test denied access and revocation paths; citations and logs need the same boundary.
  • Read: Document-level access control (Microsoft Learn)

AI Engineering

• Choose a chunking approach for technical documentation, including how you preserve meaning across section boundaries.
  • Applied scenario · ✍ Answer · Suggest a fix
  • Tests: Whether chunk size follows document structure and retrieval evaluation.
  • A strong answer covers:
    • Preserve headings, source locations and the context needed to interpret each chunk.
    • Vary chunk size and overlap on representative questions.
    • Measure retrieval misses and downstream answer quality, including table and boundary cases.
  • Read: Contextual Retrieval in AI Systems (Anthropic)
• When would you combine lexical and dense retrieval, and how would you merge their rankings?
  • Knowledge · Asked at: Microsoft †, Perplexity † · ✍ Answer · Suggest a fix
  • Tests: Whether exact terms and semantic similarity receive appropriate treatment.
  • A strong answer covers:
    • Explain lexical strength on identifiers and dense strength on paraphrases.
    • Fuse rankings without assuming raw scores are comparable.
    • Ablate each retriever and reranking on a fixed relevance set.
  • Read: Contextual Retrieval in AI Systems (Anthropic)
• Where should a cross-encoder reranker sit in a retrieval pipeline, and when does its quality gain justify latency?
  • Knowledge · Asked at: Microsoft †, Perplexity † · ✍ Answer · Suggest a fix
  • Tests: Whether candidate count, recall and reranking cost are balanced.
  • A strong answer covers:
    • Retrieve a broad candidate set first, then score query-document pairs with the reranker.
    • Measure initial recall separately: a reranker cannot recover a document missing from its input.
    • Sweep candidate and final-context counts against held-out answer quality and end-to-end latency.
  • Read: Contextual Retrieval in AI Systems (Anthropic)
• How would you separately evaluate document retrieval and answer generation in a RAG application?
  • Applied scenario · ✍ Answer · Suggest a fix
  • Tests: Whether failures can be attributed to the correct stage.
  • A strong answer covers:
    • Build questions with relevant passages and acceptable answer criteria.
    • Measure retrieval recall separately from answer support and correctness.
    • Use oracle passages to distinguish retrieval failures from generation failures.
  • Read: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., arXiv) · Demystifying evals for AI agents (Anthropic)
• Explain hypothetical-document retrieval and how you would establish whether it improves your query distribution.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether generated query expansion is treated as a testable hypothesis.
  • A strong answer covers:
    • Generate a hypothetical document, embed it and retrieve real corpus documents near that embedding.
    • Treat the generated document as a search aid, never as a factual source for the answer.
    • Compare with direct-query retrieval on representative queries, measuring relevance, answer quality and added latency.
  • Read: Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao et al., arXiv)
• Search quality declines as the corpus changes. How would you detect embedding drift and separate it from indexing defects?
  • Applied scenario · Suggest a fix
  • Tests: Whether diagnosis uses stable relevance sets and production slices.
  • A strong answer covers:
    • Hold a frozen relevance set and corpus snapshot; stable scores there clear the model and point to new content or indexing.
    • Check indexing directly: document counts against the source, embedding model version per vector, parse failures, ANN recall against exact search.
    • Slice production queries by document age and topic; new-term misses that lexical search catches signal genuine drift.
  • Read: Rules of Machine Learning (Google for Developers) · Faiss indexes (Faiss maintainers)
• Compare exact search, HNSW and IVF-PQ for an embedding index under memory, recall and latency constraints.
  • Knowledge · ✍ Answer · Suggest a fix
  • Tests: Whether approximation is evaluated against a measurable exact baseline.
  • A strong answer covers:
    • Use exact search on a sample as the recall baseline.
    • Compare graph traversal with coarse partitions and product quantisation.
    • Measure memory, update cost and filtered-search behaviour at target latency.
  • Read: Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (Malkov and Yashunin, arXiv) · Faiss indexes (Faiss maintainers)
• How would you index tables, diagrams and multi-column pages without losing their relationships to surrounding text?
  • System design · Suggest a fix
  • Tests: Whether parsing errors and layout structure are considered before embedding.
  • A strong answer covers:
    • Recover layout before embedding: reading order across columns, table structure with headers, figure captions with page coordinates.
    • Index tables and figures as linked units with header context and generated descriptions, tied to the paragraphs that cite them.
    • Audit parsing on sampled pages against the rendered originals, then evaluate retrieval on questions answerable only from tables or figures.
  • Read: Contextual Retrieval in AI Systems (Anthropic)
• Design continuous index updates, including changed documents, deletions and queries during a rebuild.
  • System design · Asked at: Perplexity † · ✍ Answer · Suggest a fix
  • Tests: Whether freshness has consistency guarantees and an observable lag.
  • A strong answer covers:
    • Apply ordered changes idempotently by document ID and version; define when a complete new chunk set becomes visible.
    • Enforce deletions and revoked access at query time; rebuild from a consistent snapshot, catch up changes and validate before switching the read alias.
    • Measure source-to-search lag and reconcile counts, representative queries and permissions; rollback must not resurrect deleted or forbidden content.
  • Read: Aliases (Elastic) · Document-level access control (Microsoft Learn)
• How would you connect each factual claim in an answer to the retrieved passage that supports it?
  • System design · Asked at: Perplexity † · ✍ Answer · Suggest a fix
  • Tests: Whether a citation is checked for support rather than merely attached.
  • A strong answer covers:
    • Bind each claim to a passage identifier during generation instead of appending links at the end.
    • Verify that the cited passage entails the claim before the answer is shown.
    • Define behaviour when nothing supports a claim: drop it, mark it unsupported or abstain.
  • Read: Patterns for Building LLM-based Systems & Products (Eugene Yan)
• A user asks for the count of open orders blocked by supplier issues. Why might document retrieval fail, and what should answer the query?
  • Applied scenario · Asked at: Palantir † · Suggest a fix
  • Tests: Whether structured aggregation is separated from semantic retrieval.
  • A strong answer covers:
    • Explain why top-k retrieval fails: it returns similar passages, not a complete filtered set, so any count is a sample.
    • Route to the system of record: a governed query over order status and block reason, with the model only translating intent.
    • Pin down what counts as blocked by a supplier, show the query and data timestamp, and test against known counts.
  • Read: Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (Lei et al., arXiv) · Table Expressions (PostgreSQL Global Development Group)
• How can a patient's existing medication list help transcribe drug names without causing the recogniser to insert drugs that were never spoken?
  • Applied scenario · Suggest a fix
  • Tests: Whether contextual retrieval is balanced against confirmation bias.
  • A strong answer covers:
    • Use the list as a bounded decoding bias toward acoustically plausible matches, never as text to insert.
    • Guard against confirmation bias: compare biased and unbiased hypotheses, and flag listed drugs with weak acoustic evidence for review.
    • Test listed, new unlisted and unmentioned drugs; tune bias strength on recall against false insertions and substitutions.
  • Read: Improve recognition accuracy with phrase list (Microsoft Learn) · Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models (World Health Organization)

← Inference, serving and cost · Agents, tools and protocols →