PRE-KICKOFF ALIGNMENT · AI FUNDAMENTALSKey concepts for LLMs and AIagents: the shared vocabularyyour first project needsA one-hour session for mixed project teams before writing the first prompt or line of codeStrategy & Engineering · October 2026
Welcome the room and set expectations: this session covers ten concepts that will appear in every AI project conversation. The goal is not to make everyone an AI engineer but to ensure managers and engineers use the same words to mean the same things. Each slide introduces one concept and ties it to a concrete project decision. Expect questions; the vocabulary is new but the underlying logic is familiar.
2FRAMINGAgree on ten concepts before kickoff, orre-litigate them in change-requests laterSITUATIONOur team is preparing its first AI project. Managers own scope, risk and budget; engineers own the build. Bothsides have read vendor material and feel ready.COMPLICATIONThe same words mean different things across the table. 'Fine-tuning' is rarely the right first move; 'token' is a billingunit to one person and a model internal to another. These gaps become change-requests once the build starts.RESOLUTIONSpend one hour now aligning on ten concepts — each tied to a project decision — so scope, costand acceptance criteria rest on shared ground before a single line of code is written.Source: McKinsey Global Survey on AI 2024 (2024)
Open with this slide to earn the room's attention. The complication is the real hook: misaligned vocabulary is not a soft problem — it shows up as change-requests and cost overruns. The ten concepts we will cover are: tokens and context window, training vs inference, embeddings and RAG, fine-tuning vs prompting, hallucinations, function calling, agents, evaluation, guardrails, and cost and latency. Each one maps to a project decision the team will face. [Sources] - McKinsey Global Survey on AI 2024 — McKinsey — 2024 — https://www.mckinsey.com/capabilities/quantumblack/our-insights/state-of-ai
3FRAMINGOne concept per slide, each tied to the project decision it drives1How an LLM works: tokens and training shape whatthe model holds and when costs landSlides 4–5 cover the context window and training versus inference— the two facts that govern prompt design and per-requestbudgeting.2How you shape its behaviour: retrieval, promptingand hallucination controls are your leversSlides 6–8 cover RAG, fine-tuning versus prompting, andhallucination mitigation — the decisions that determine data pipelinedesign and quality controls.3How you extend it into an agent: function calls andloops turn answers into actionsSlides 9–10 cover function calling and the agent plan-act-observeloop — the decisions that open your internal APIs and setautomation boundaries.4How you ship it responsibly: evaluation, guardrailsand economics decide what goes liveSlides 11–12 cover test-set evaluation, guardrails, cost and latency— the gate criteria every AI feature must clear before launch.Source: The Minto Pyramid Principle: Logic in Writing, Thinking and Problem Solving (1987)
Use this slide as the deck map. The four branches form a logical build: you must understand what the model is before you shape it, shape it before you extend it, and extend it before you ship it. Point to each branch and name the slides it covers so the audience can follow along. This also sets the expectation that every concept will end with a 'decision this drives' statement. [Sources] - The Minto Pyramid Principle: Logic in Writing, Thinking and Problem Solving — Barbara Minto — 1987
4HOW AN LLM WORKSThe context window caps what the model reads atonce, so it caps your document and prompt design~4 charsA token is roughly fourcharacters of English text8K–1MContext windows span 8K to1M tokens across vendorsOverflow forces a choice:chunk, summarise or retrieveModels read tokens, not words. Onetoken is about four characters inEnglish. A 1,000-word document isroughly 1,300 tokens. This determineshow much text fits in a single request.GPT-4o: 128K; Claude: 200K; Gemini:1M tokens. The window holds yoursystem prompt, retrieved docs andquestion. Larger windows enablebigger documents and longer histories.When content exceeds the window,choose: compress via summarisation,split into chunks, or retrieve onlyrelevant passages. This choice is thefirst architectural decision in your datapipeline.Source: OpenAI platform documentation — models and pricing (2024); Anthropic Claude model documentation (2024); Google AI — Gemini and long context documentation (2024)
Anchor this with a physical analogy: the context window is a whiteboard in the room — the model can only reason about what is written on it right now, not what is in a filing cabinet. Everything the model will use in this request must fit on that whiteboard. The practical implication: long contracts, large codebases or big datasets cannot be handed to the model wholesale. The data pipeline must be designed around the window size. Source: OpenAI platform docs, Anthropic docs for vendor-specific window sizes. [Sources] - OpenAI platform documentation — models and pricing — OpenAI — 2024 — https://platform.openai.com/docs/models - Anthropic Claude model documentation — Anthropic — 2024 — https://docs.anthropic.com/claude - Google AI — Gemini and long context documentation — Google DeepMind — 2024 — https://ai.google.dev/models
5HOW AN LLM WORKSTraining is a sunk cost you rarely pay;inference is the per-request bill you always payOne-offTraining is a one-off costalready paid by the vendor$/tokenInference bills on every APIcall you makeBudget for inference, nottrainingFrontier models are trained on trillionsof tokens at a cost of tens of millions ofdollars (Stanford HAI AI Index 2024).That cost is built into vendor pricing —your team does not re-run training on atypical project.You pay per token — input and outputpriced separately — on every userrequest. GPT-4o and Claude bothoperate this way (OpenAI, Anthropicpricing pages; vendor-dependent).Volume drives cost.Retrieving your data at request time(RAG) is faster, cheaper and easier toupdate than fine-tuning. Model yourinference cost: tokens per requesttimes volume times per-token price.Source: OpenAI platform documentation — models and pricing (2024); Anthropic Claude model documentation (2024); Stanford HAI AI Index Report 2024 (2024)
This slide neutralises the most common misconception: that the team will need to train a model on company data. Almost no first project requires this. The decision this drives is budget: model inference costs — tokens per request multiplied by volume — and do not confuse that with a training investment. Source: Stanford HAI AI Index 2024 for training scale; OpenAI and Anthropic pricing pages for inference billing model. [Sources] - OpenAI platform documentation — models and pricing — OpenAI — 2024 — https://platform.openai.com/docs/models - Anthropic Claude model documentation — Anthropic — 2024 — https://docs.anthropic.com/claude - Stanford HAI AI Index Report 2024 — Stanford HAI — 2024 — https://aiindex.stanford.edu/report/
6HOW YOU SHAPE ITRetrieval, not re-training, is the defaultway to put your data in front of the model01Embed yourdocumentsConvert each documentchunk into a vectorcapturing its meaning.An embedding model(e.g. OpenAI text-embedding-3-small)processes your corpusonce.Data pipeline02Embed the user'squestionAt request time, convertthe user's question intoa vector using the samemodel. Semanticsimilarity is nowmeasurable across allchunks.Application03Retrieve the closestchunksFind chunks whosevectors are closest tothe question vector. Avector database returnsthe top-k most relevantpassages inmilliseconds.Application04Add chunks to thepromptPut retrieved passagesand the question intothe context window. Themodel reads only therelevant content andanswers from your data— no retraining.Application05Return and cite theanswerThe model respondsgrounded in retrievedtext. Ask it to citesources, giving users apath to the original. Re-embed whendocuments change.ApplicationSource: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020); OpenAI platform documentation — models and pricing (2024); Hugging Face fine-tuning and embeddings documentation (2024)
RAG is the pattern that solves 'how do we use our internal documents without re-training?' Highlight step 4 as the insight: the model never stores your data — it reads it on each request from the context window. This means you can update documents without touching the model, and you can apply access controls at the retrieval layer. The practical decision this drives: do you have a document corpus, and what format is it in? RAG requires a chunking strategy, a refresh cadence and access control at retrieval. Source: Lewis et al. 2020 is the foundational RAG paper. [Sources] - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., arXiv — 2020 — https://arxiv.org/abs/2005.11401 - OpenAI platform documentation — models and pricing — OpenAI — 2024 — https://platform.openai.com/docs/models - Hugging Face fine-tuning and embeddings documentation — Hugging Face — 2024 — https://huggingface.co/docs
7HOW YOU SHAPE ITTry prompting first; fine-tune only when prompting plateaus on a stable taskVerdict: Start with prompting; add RAG when you need live data; fine-tune only on a stable proven taskRECOMMENDEDPromptingInstructions in text onlyRAGRetrieval adds live data to each promptFine-tuningModel weights updated on yourexamplesWhat it changesInstructions and examplesin textAdds relevant docs at requesttimeModel weights retrained onlabelled pairsSetup timeHours to daysDays to weeks (pipeline build)Weeks (labelling + trainingrun)Update cost when data changesEdit a text file; instantRe-embed changeddocumentsRe-run training; high costWhen to useAlways try firstData changes; facts must staycurrentPrompting plateaued; stablehigh-volume taskSource: OpenAI platform documentation — models and pricing (2024); Hugging Face fine-tuning and embeddings documentation (2024)
The key message is sequencing, not a permanent hierarchy. The audience often arrives expecting to fine-tune from day one because vendor marketing emphasises it. This slide reframes fine-tuning as a last resort after prompting and retrieval are exhausted. The practical gate for fine-tuning: you need hundreds of labelled examples, a stable task definition, and evidence that prompting has truly plateaued. Source: OpenAI and Anthropic official guides both recommend this sequence explicitly. [Sources] - OpenAI platform documentation — models and pricing — OpenAI — 2024 — https://platform.openai.com/docs/models - Hugging Face fine-tuning and embeddings documentation — Hugging Face — 2024 — https://huggingface.co/docs
8HOW YOU SHAPE ITTreat confident wrong answers as a product riskand mitigate with retrieval, structure and checksGround answers inretrieved sourcesConstrain the outputwith schemasCheck with a secondpass or externallookupSurface uncertaintyhonestly in the UXRAG constrains the modelto facts in the retrievedtext. Ask it to citepassages. If no relevantsource is found, thesystem should say sorather than improvise.Grounding is the mosteffective single mitigation.Structured outputs (JSONschema, enum choices)remove space for inventedvalues. Supply allowedlabels for classification;supply field names andtypes for extraction. Themodel cannot hallucinatewhat it cannot choose.For high-stakes outputs, asecond model call asks: 'Isthis answer supported bythe provided sources?'Validate extracted valuesagainst a database. Addscost and latency but isright for regulateddecisions.When confidence is low,show it. Labels like 'notfound in our documents' or'consult a specialist' shiftaccountabilityappropriately. Hidinguncertainty is a productdesign choice thatamplifies risk.Source: OpenAI platform documentation — models and pricing (2024); Anthropic Claude model documentation (2024)
Hallucination is the concept that most worries managers and most surprises engineers the first time they see it. The frame here is product risk, not model failure: every product team ships with known defect rates; the question is how you measure and bound hallucination rates for this specific task. The mitigation list is ordered by effectiveness and cost: retrieval-grounding is free if you already have RAG; structured output is a small prompt change; the second-pass check is a deliberate cost decision. Source: OpenAI and Anthropic system cards acknowledge hallucination risk in their own documentation. [Sources] - OpenAI platform documentation — models and pricing — OpenAI — 2024 — https://platform.openai.com/docs/models - Anthropic Claude model documentation — Anthropic — 2024 — https://docs.anthropic.com/claude
9HOW YOU EXTEND ITFunction calling turns the model from a writer into an actor on your systemsUser sends a requestA user asks somethingrequiring live data or anaction — e.g. 'What is thestatus of order #4821?'The model cannot answerfrom memory alone.UserModel selects a toolGiven available functionswith descriptions, themodel decides which tocall and what argumentsto pass. It outputsstructured JSON, notprose.ModelApplicationexecutes the callYour application codereceives the call,validates it, and runs thereal function — adatabase query, API call,or file read. The modelwaits.ApplicationResult is returned tothe modelThe function result entersthe conversation context.The model now has real,current data — a liveorder status, accountbalance or calendar slot.ApplicationModel writes thefinal answerWith the result in context,the model composes anatural-languageresponse grounded inreal data. The user seesan answer, not rawJSON.ModelSource: OpenAI platform documentation — models and pricing (2024); Anthropic Claude model documentation (2024)
Highlight step 2 as the architectural shift: the model is making a decision about which tool to call and what arguments to pass. This is not the model executing code directly — your application retains control of execution. The practical decision this drives: which internal APIs or data sources will you expose as tools? Each tool needs authentication, authorisation, rate limiting and an audit log. Start with read-only tools; add write tools only after the loop is validated. Source: OpenAI introduced function calling June 2023; Anthropic and Google followed with equivalent tool-use APIs. [Sources] - OpenAI platform documentation — models and pricing — OpenAI — 2024 — https://platform.openai.com/docs/models - Anthropic Claude model documentation — Anthropic — 2024 — https://docs.anthropic.com/claude
10HOW YOU EXTEND ITAn agent is a loop that plans, acts and checksitself; memory decides how far it can go01Receive goalThe agent receives amulti-step goal, e.g.,'Research vendors anddraft a comparison.'Held in working context.02Plan next actionThe model reasonsabout the next actiongiven the goal and priorobservations. It selectsa tool or judges it hasenough to produceoutput.03Act with a toolThe agent calls afunction — web search,database query, filewrite — and waits forthe result. Multipleactions may occurbefore the goal isreached.04Observe and updateThe result enters thecontext. The modelreads it and decides:goal met or anotheraction needed? Theloop runs until done or astep cap is reached.05Return result orescalateWhen the goal iscomplete, the agentdelivers the result. If ithits a step cap or needshuman judgement, itescalates to a human-in-the-loop gate.Source: ReAct: Synergizing Reasoning and Acting in Language Models (2022); Anthropic Claude model documentation (2024)
The ReAct pattern — reason, act, observe — is the standard framework for agent design and is cited in virtually all agent tutorials and research. Highlight step 4 as the design crux: the context window is the agent's short-term memory. When the context fills, the agent loses track of earlier steps unless you implement external memory (a database of past observations). The decisions this drives are: maximum steps before a human reviews, what memory store persists between sessions, and where human approval is required before an action executes. Source: Yao et al. 2022 coined ReAct. [Sources] - ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al., arXiv — 2022 — https://arxiv.org/abs/2210.03629 - Anthropic Claude model documentation — Anthropic — 2024 — https://docs.anthropic.com/claude
11HOW YOU SHIP ITIf you cannot measure quality on a fixed testset, you cannot ship or improve the systemBuild a frozen test set before writingany promptsFixed Q&A pairs created before build are your only honest measure. Tuning on your testexamples measures memorisation, not generalisation. Target 50–200 examplesdepending on task complexity.Mix automated metrics with humanreviewAutomated checks — exact match, semantic similarity, factual consistency — run fast inCI. Human review catches errors automation misses: tone, coherence, unsafe outputs.Both are needed; neither is sufficient.Run the test set before everychange, not just at launchPrompt changes, model upgrades and new data can degrade quality silently. Treatingthe test set as a regression gate — like unit tests in software — catches regressionsbefore they reach users.Monitor production quality afterlaunchA static test set cannot cover the full distribution of real queries. Sample productionoutputs and route a fraction to human review. Track metrics over time; plan for modeldeprecations that force re-evaluation.Source: Stanford HAI AI Index Report 2024 (2024); OpenAI platform documentation — models and pricing (2024)
This slide often surprises engineering teams who assume that because the model 'feels right' during development it is ready to ship. The frozen test set is the acceptance criterion: the team agrees in advance on the pass threshold, and the system does not go live until it meets it. For managers, frame this as the equivalent of a UAT sign-off. The number of examples depends on the task — a narrow extraction task may need only 50; an open-ended Q&A system may need hundreds. Source: Stanford AI Index 2024 cites evaluation gaps as a top barrier. [Sources] - Stanford HAI AI Index Report 2024 — Stanford HAI — 2024 — https://aiindex.stanford.edu/report/ - OpenAI platform documentation — models and pricing — OpenAI — 2024 — https://platform.openai.com/docs/models
12HOW YOU SHIP ITGuardrails, cost and latency are the threegates every AI feature passes before launchDECISION REQUESTEDConfirm alignment onthese ten concepts,assign owners forguardrails, cost andlatency, and schedulethe project kickoff1Define guardrails: prompt injection tests, PII masking, content policythresholdsProduct lead · Before kickoff2Set inference cost budget per request; configure usage alerts (44% citecost unpredictability)Engineering lead · Before kickoff3Agree latency SLA and identify caching or fallback strategy when themodel is slowEngineering lead · Before kickoff4Schedule pre-launch eval review against frozen test set and the threegate criteriaProject sponsor · Before launch5Assign on-call owner for post-launch monitoring of quality, cost andguardrail alertsTo be named · At launchSource: OpenAI platform documentation — models and pricing (2024); Anthropic Claude model documentation (2024); McKinsey Global Survey on AI 2024 (2024); OWASP LLM Top 10 2025 (2025)
Close with the three gates as a checklist, not a warning. Guardrails cover the safety and policy surface: prompt injection (OWASP LLM Top 10 #1, 2025), PII in outputs, content that violates your policy. Cost covers per-request economics and the cumulative monthly bill. Latency covers the user experience: what is an acceptable wait time, and what happens when the model is slow or unavailable. If the team can answer all three before launch, the feature is ready. [Sources] - OpenAI platform documentation — models and pricing — OpenAI — 2024 — https://platform.openai.com/docs/models - Anthropic Claude model documentation — Anthropic — 2024 — https://docs.anthropic.com/claude - McKinsey Global Survey on AI 2024 — McKinsey — 2024 — https://www.mckinsey.com/capabilities/quantumblack/our-insights/state-of-ai - OWASP LLM Top 10 2025 — OWASP Foundation — 2025 — https://owasp.org/www-project-top-10-for-large-language-model-applications/