How LLMs Actually Work

This is a practitioner's course on AI — not a prompting workshop. You may already know how to talk to AI tools. This course teaches you how to think with AI — to understand what's happening under the hood, and to reason from first principles about how to build with it.

Before getting into mechanics, it helps to clear the slate. Everyone arrives with a head full of AI assumptions shaped by science fiction, news headlines, and marketing copy. If these aren't named and addressed directly, they quietly distort everything else you'll learn. Read each one and check your own assumptions against the reality.

Before reading further — pause and ask yourself: when you send a message to ChatGPT or Claude, what do you think is actually happening inside the model? Hold that intuition in mind, then read what follows.

Text → Tokens. AI doesn't read words the way humans do. It reads chunks called tokens. A single word might be one token, or it might be split across several. The model never sees raw text — only a sequence of numbers.

You can see this yourself by visiting tiktokenizer.vercel.app — paste any text and watch how it gets split into tokens. Try a company name, a technical term, or a sentence from a document you work with, and notice how it tokenizes. Then paste a long paragraph and see the token count. This builds direct intuition for why token limits exist and why phrasing matters at scale.

Tokens → Numbers (Embeddings). Every token gets converted into a point in a very high-dimensional space — imagine a map with thousands of axes. Words that mean similar things cluster close together. The classic example: King − Man + Woman ≈ Queen. That's a real result from this vector space. The model has learned meaning through geometry.

The Prediction Machine. At its core, every LLM is doing one thing: predicting the next most likely token, over and over. Every word in a response is the output of this prediction, chained together. This is the single most important fact to internalise — because it explains both the power and the failure modes of AI.

Everything we've described so far — tokens, embeddings, prediction — applies to text. But modern enterprise models are not text-only. They can receive and process multiple types of input. This is called multimodality, and it's what makes AI genuinely useful across functions that don't primarily work in text.

For someone in finance reviewing invoices, HR processing resumes, or compliance checking contracts — their work is documents, tables, and images. The moment they realise the model can read those directly, the session becomes relevant to them.

This is the "secret sauce" of the Transformer architecture. Keep it conceptual — no matrix math needed. The goal is just the mental model.

Key idea: not all words in a sentence matter equally to each other.

Attention is why models can handle long, complex sentences without losing track of what refers to what. And it's why longer, richer prompts often perform better — you're giving the model more context to "attend to" when forming its response.

This is also the computational reason why longer contexts are more expensive. Every token is attending to every other token. Quadratic cost. More tokens = exponentially more computation.

Most people conflate these two phases. Separating them is one of the most clarifying things you can do early in this course.

This sets up Session 2 perfectly — because the next question is: how do companies build AI systems that actually do remember things? That's where RAG and vector databases come in. Plant that question tonight and let it sit.

Here's a question the session so far hasn't answered: if the base model is just predicting the next token from internet text, why does it behave helpfully instead of generating random noise, hate speech, or conspiracy theories — all of which exist in its training data?

The answer is Reinforcement Learning from Human Feedback (RLHF). It's the step that turns a raw text predictor into a useful assistant — and it's why Claude, ChatGPT, and Gemini feel like products and not just autocomplete engines.

When you send a message to an AI, you're seeing one small piece of what the model actually receives. In production systems — the kind you'll build at work — a "prompt" is three distinct things, and conflating them is one of the most common engineering mistakes.

There's a question every non-technical person in an organisation will eventually face: "should we use Claude.ai, buy a tool built on AI, or build something ourselves?" These are three completely different decisions — commercially, technically, and in terms of control — but they look identical from the outside. Here's the actual picture.

Using AI as an individual and deploying AI inside an organisation are fundamentally different problems. Most AI content online is written for the individual user — the person experimenting alone, optimising their own workflow. Enterprise AI is a different discipline entirely. The technical challenge is often the easier half.

Here is how the enterprise context changes every question you'll face when building or adopting AI systems:

You've heard "high temperature = creative, low temperature = consistent." That's true, but it's not the real explanation — and as engineers, you deserve the actual picture.

Every time the model predicts the next token, it doesn't just pick the single most likely one. It generates a probability distribution over every possible next token in its vocabulary. "The" might have a 40% probability, "A" might have 20%, "It" might have 15%, and thousands of other tokens share the remaining 25%.

Temperature controls how you sample from that distribution.

Every LLM has a knowledge cutoff date — the point at which its training data stopped being collected. Anything that happened after that date is invisible to the model. It doesn't know about last week's news, a law that changed last month, or your company's Q2 results.

What was it actually trained on? Mostly: publicly available web text (Common Crawl), books, Wikipedia, code repositories, scientific papers, and curated datasets. This means the model is extraordinarily good at general knowledge, reasoning, writing, and code — but it has zero awareness of your proprietary business data, internal documents, or anything non-public.

The context window is the single most important constraint in practical AI engineering. Everything you need the model to "know" for a given request must fit inside it — and what doesn't fit, doesn't exist as far as the model is concerned. Understanding how it fills up, what happens when it overflows, and how to manage it deliberately separates working prototypes from production systems.

Four patterns for managing context in production:

You'll make model selection decisions when building AI products. "GPT-4 is better than GPT-3.5" — but what does that mean? What are you actually choosing between?

The most important dimension is parameter count. Parameters are the numerical weights that get updated during training — the millions or billions of numbers that collectively encode the model's knowledge and behavior. A 70B model has 70 billion of these weights. A 7B model has 7 billion.

The practical decision framework: Use the smallest model that reliably handles your task. Start with a mid-tier model, measure quality, then step up only if needed. In high-volume enterprise systems, the cost difference between a small and large model can be 10–50x. That's not a detail — it's the business case.

Tokens are the atomic unit of both capability and cost. Every API call is billed on two dimensions: how many tokens went in (input) and how many came out (output). Understanding this changes every design decision you make — from how you write system prompts to how long you let conversations run.

The deeper issue is that token costs are not linear. They compound. A conversation that starts cheap becomes expensive fast — and most teams don't notice until the invoice arrives.

This is the context tax. Every time you send a new message, you pay for the entire conversation history again — not just what you typed. By turn 10, the majority of your token spend is on context you've already paid for in previous turns. This compounds across thousands of daily users in a production system.

The four levers of context economy:

The fastest way to internalise these concepts is to experience them directly. Each exercise below takes 3–5 minutes and demonstrates something you cannot fully grasp from reading alone. Open a browser tab and work through them.

You may already know how to prompt — but likely by feel and intuition. Now that you understand the mechanics, you can reason from first principles about why the techniques that work, work. That shift — from intuition to reasoning — is what makes prompting a disciplined skill rather than guesswork.

AI bias is not a design choice or a political statement — it's a mathematical consequence of training. A model learns from data. If the data reflects historical inequities, the model encodes them. If the data over-represents certain demographics, languages, or viewpoints, the model performs better for those groups and worse for everyone else. This is not fixable by prompting. It's a property of the training process.

What this means for enterprise deployment: Before deploying any AI system that makes or influences decisions about people — hiring, credit, healthcare, access to services — you need to audit the model's performance across your specific demographic and language groups, not just its average benchmark performance. Average performance can look excellent while hiding severe underperformance for specific subgroups.

This is the single biggest mindset shift for engineers and product managers coming from traditional software. It changes how you test, how you debug, how you measure quality, and how you design systems. Until you make this shift, AI systems will feel unreliable and unmanageable. After it, they become tractable.

There are tasks where AI can produce output independently and tasks where a human must review that output before anything happens. The design decision about where this boundary sits is one of the most important — and most frequently skipped — decisions in enterprise AI deployment.

Getting it wrong in one direction means under-using AI (humans reviewing everything, defeating the efficiency gain). Getting it wrong in the other direction means AI acting autonomously in situations where the cost of error — legal, financial, reputational, or human — is too high.

The three review patterns in practice:

Most people prompt the way they'd text a colleague — casually, partially, assuming shared context. That works for quick personal tasks. It fails consistently when you're building something that needs to work reliably across hundreds of different inputs from different users. You need a structure.

The PCTFE framework is a five-element scaffold for writing prompts that are explicit, testable, and maintainable. Think of it the same way you'd think about writing a function: inputs, behaviour, outputs — specified completely.

Additional techniques that compound on the framework:

These are the eight patterns that consistently produce bad output — not because the model is broken, but because the prompt is underspecified. Each one has a simple fix once you know what to look for.

A prompt that runs in production thousands of times a day is not a casual instruction — it's a critical piece of software. It should be treated with the same discipline as code: versioned, tested, documented, and reviewed before it ships.

Every prompt you send to an external API crosses a network boundary and is processed on someone else's infrastructure. Most enterprise teams don't think about this until after an incident. Know these rules before you build anything that touches real user data.

Beyond hallucination (which the session has already covered), there are structural limitations that every person in this room should know — because overestimating AI capability in any of these directions leads to failed projects, broken trust, and real business risk.

Work through these questions yourself, or bring them to the group session. They're designed to bridge what you've just read with how it applies to your actual work.

This exercise primes everything in Session 2 — your real example will be the raw material for understanding RAG, memory, and context injection. Generic examples are forgettable. Your actual workflow is not.