Skip to content

AI product strategy and metrics

English · Русский · AI Interview Atlas › Themes

When AI is worth it, how to design AI features, measure them and build the business case.

AI Leadership

  • How would you improve ChatGPT for enterprise customers?
  • A model offers much greater capability but costs ten times as much. How would you decide what product to build with it?
  • How do you decide whether an LLM is useful for a proposed product problem?
    • Product case · Senior · Asked at: Google and Google DeepMind † · ✍ Answer · Suggest a fix
    • Tests: Whether you start with the task and evidence rather than the technology.
    • A strong answer covers:
      • Describe the job, baseline solution and cost of mistakes.
      • Check data access, evaluability, latency and the need for predictable behaviour.
      • Run a bounded comparison and name a reason to reject the LLM approach.
    • Read: Rules of Machine Learning (Google for Developers) · Building Effective AI Agents (Anthropic)
  • Design a Gemini-based assistant for university students.
  • Users say Gemini gives incorrect answers with excessive confidence. What would you change?
    • Product case · Senior · Asked at: Google and Google DeepMind † · ✍ Answer · Suggest a fix
    • Tests: Whether you diagnose failure types and improve calibrated user trust.
    • A strong answer covers:
      • Collect representative failures and separate knowledge, retrieval and presentation issues.
      • Compare grounding, abstention and interaction changes on the same tasks.
      • Track unsupported claims, useful task completion and segment-level regressions.
    • Read: AI Risk Management Framework (NIST) · Demystifying evals for AI agents (Anthropic)
  • How would you prioritise features for an AI writing assistant in Microsoft Word?
  • How would you measure success for an AI feature in a Microsoft product?
  • What North Star metric would you choose for an AI search product, and what would constrain it?
  • Notification engagement rises while total time spent remains flat. How would you interpret that result?
  • Choose an AI agent or product you use and propose its most valuable improvement.
  • You have invented a memory machine that generates video, images, smells and sounds. How would you bring it to market?
    • Product case · Senior · Asked at: OpenAI † · Suggest a fix
    • Tests: Whether you turn an unfamiliar capability into a focused market entry.
    • A strong answer covers:
      • Choose one beachhead user and occasion, and only the senses that occasion actually needs.
      • Measure repeat paid use, with guardrails on user distress and the consent of people depicted.
      • Test willingness to pay with a hand-run concierge pilot before building scent hardware.
    • Read: User Needs + Defining Success (Google PAIR) · How the alpha phase works (Government Digital Service)
  • Which business use cases suit AI agents, and how would you compare them?
    • Product case · Senior · Asked at: Microsoft † · ✍ Answer · Suggest a fix
    • Tests: Whether you distinguish the value of autonomous action from its operational cost.
    • A strong answer covers:
      • Score candidates on volume, a verifiable outcome, tolerance for error and reversibility.
      • Separate the value of the action taken from the cost of supervising and correcting it.
      • Compare against the cheaper option: a prompt, a fixed workflow or ordinary automation.
    • Read: Building Effective AI Agents (Anthropic)
  • Explain the product trade-offs in choosing a model.
    • Knowledge · Senior · Asked at: Microsoft † · ✍ Answer · Suggest a fix
    • Tests: Whether model selection considers task quality, cost, latency and constraints together.
    • A strong answer covers:
      • Fix the task and a quality bar first, then measure candidate models on your own data.
      • Compare cost per successful outcome and latency as the user experiences it.
      • Add the constraints that disqualify a model: deployment region, data handling, change over time.
    • Read: Reasoning models (OpenAI API documentation)
  • Propose a new feature for Perplexity and explain whom it serves.
    • Product case · Senior · Asked at: Perplexity † · Suggest a fix
    • Tests: Whether you can make a differentiated, evidence-seeking product proposal.
    • A strong answer covers:
      • Pick one segment and a recurring research job that users still finish outside the product.
      • Explain why rivals cannot simply copy it, and measure completed tasks with a citation-accuracy guardrail.
      • Test the riskiest assumption cheaply with a fake-door test or a manual prototype for target users.
    • Read: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications (Rodden, Hutchinson and Fu, Google Research)
  • How do you prioritise a product roadmap when several valuable opportunities compete?
  • Design a product connecting local tradespeople with customers.
    • Product case · Senior · Asked at: Meta 🗣 · Suggest a fix
    • Tests: Whether you reason about both sides of a marketplace and initial liquidity.
    • A strong answer covers:
      • Name both sides, the decision each makes, and what each must trust: vetting, price, payment.
      • Measure requests ending in a completed job and time to match, guarded by disputes and repeat bookings.
      • Launch in one area and one trade, recruiting tradespeople by hand before automating the match.
    • Read: How to match supply and demand in your marketplace (Sharetribe)
  • A new service grows while Facebook monthly active users decline. How would you assess whether the service is succeeding?
  • Design a volunteer-matching product and explain how it gets through cold start.
    • Product case · Senior · Asked at: Meta 🗣 · Suggest a fix
    • Tests: Whether you make the first useful matches possible before scale arrives.
    • A strong answer covers:
      • Pick which side is scarce, then narrow to one city and cause where matches can happen.
      • Measure posted needs filled by a completed shift, with no-shows and organiser churn as guardrails.
      • Match the first organisations and volunteers by hand to test whether matches turn into shifts.
    • Read: How to match supply and demand in your marketplace (Sharetribe)
  • Where does an AI answer engine outperform traditional search, and where does it fall short? What would you improve first?
    • Product case · Senior · Asked at: Perplexity † · ✍ Answer · Suggest a fix
    • Tests: Whether you compare products through user tasks and evidence quality.
    • A strong answer covers:
      • Split queries by task: navigation, a single fact, comparison across sources, open exploration.
      • Judge an answer by whether its evidence is checkable, current and correctly attributed.
      • Name the first improvement and the metric that would show it worked.
    • Read: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., arXiv)
  • Walk through the product you built that had the greatest impact, including the business case and evidence of results.
  • How would you price an enterprise AI product, which costs must you count, and how does customising it for each client affect its margin?
    • Product case · ✍ Answer · Suggest a fix
    • Tests: Whether you reason about unit economics and margin, not only product value.
    • A strong answer covers:
      • Name the cost stack per account: inference, retrieval and storage, evaluation, human support.
      • Choose a price metric the customer can predict and you can defend as volume grows.
      • Treat per-client customisation as recurring cost, and price or productise it deliberately.
    • Read: What We Learned from a Year of Building with LLMs (Part I) (O'Reilly Radar)
  • In discovery for an enterprise AI product, who is the persona, where do you find B2B respondents, and how will you tell that a problem is worth solving?
    • Product case · ✍ Answer · Suggest a fix
    • Tests: Whether you adapt customer research to enterprise buyers, users and decision-makers.
    • A strong answer covers:
      • Split the persona into user, budget-owning sponsor and blockers such as security, legal and IT.
      • Recruit through existing customers, account teams and partners, interviewing several roles in the same account.
      • Judge the problem by current spend on workarounds and a sponsor's willingness to commit scarce resources.
    • Read: How to recruit UX Research participants (GitLab Handbook)
  • What do you do when an MVP of an enterprise AI product is too expensive or too slow to build?
    • Product case · ✍ Answer · Suggest a fix
    • Tests: Whether you test the riskiest assumptions with cheaper prototypes before committing to a build.
    • A strong answer covers:
      • Treat the MVP as an experiment: list its assumptions and rank them by uncertainty and cost of error.
      • Test the riskiest cheaply: the model offline on customer data, a concierge pilot, or a paid pilot commitment.
      • Agree in advance which evidence justifies the build, and name what a cheap test cannot prove.
    • Read: Rules of Machine Learning (Google for Developers)
  • An AI product has rising usage but no credible improvement in customer outcomes; review costs keep increasing. The sponsor wants another quarter of investment. What evidence would make you expand, narrow, pause or stop it, and how would you handle existing users?

← Practical coding · AI platform and operating model →