What Actually Happens When You Ask AI a Question
I'm back doing my Master's this term, taking two modules: Generative AI, and AI for Business. In the first week, one slide caught my attention — the architecture diagram for something called a "Transformer." It looks intimidating, all boxes and arrows, but it's actually the blueprint behind ChatGPT, Claude, and nearly every generative AI tool in use today.
What struck me is how few people who use these tools every day have any real sense of what's happening underneath. So I wanted to walk through it — not for the exam, but because I genuinely think it's one of the more fascinating pieces of technology most of us interact with and never think twice about.
Let's trace what happens between you typing a question and the model replying.
1. Breaking your question into tokens
The model doesn't read your question the way you do. It splits it into "tokens" — small chunks of text, sometimes whole words, sometimes just parts of words. "Transformers" might become "Trans" + "formers," for example. This is the model's raw input.
2. Turning tokens into numbers
Each token is converted into a long list of numbers — a vector — that represents its meaning in a mathematical space. Words with related meanings end up numerically closer to each other. This is how a model that has never "understood" anything in the human sense can still work with meaning.
3. Adding a sense of order
Here's something non-obvious: the model processes your entire question at once, not word by word like we read. That's actually one of its biggest strengths — it can weigh the whole sentence for context. But it means the model has no built-in sense of order, so a step called positional encoding is added, essentially tagging each token with where it sits in the sentence.
4. Self-attention — the core idea
This is the heart of the whole architecture. For every token in your question, the model asks: which other tokens in this sentence matter most to understanding this one? Take the sentence "The trophy didn't fit in the suitcase because it was too big." Which does "it" refer to — the trophy or the suitcase? Self-attention is the mechanism that lets the model work that out, by having every token essentially compare notes with every other token.
This process repeats across many layers, each one refining the model's grasp of the relationships in your question a bit further.
5. Processing and stabilising
Between rounds of attention, the model runs the information through a feed-forward network — effectively a moment to "digest" what attention just revealed — along with normalisation steps that keep the numbers stable as they pass through dozens of layers. There's also a shortcut (a residual connection) that lets information skip ahead, which helps very deep models like this train and reason effectively.
6. Generating the answer, one word at a time
Once the model has built a rich representation of your question, it starts generating a response — but not all at once. It predicts one token, adds it to what's been generated so far, then predicts the next one, using both your original question and everything it has written so far. This repeats until the answer is complete. That's why longer responses sometimes take visibly longer to appear — the model is doing this whole cycle again for every single word.
Why this matters if you're not technical
A few things fall out of this that matter for anyone using AI in a business context:
The model isn't "looking things up" — it's predicting the most statistically plausible next word based on patterns learned from enormous amounts of text. That's why it can be fluent and confident while still being wrong — a phenomenon we call hallucination. It's not lying; it genuinely has no concept of true or false, only plausible or implausible.
It also explains why the length and clarity of your question matters so much. The self-attention mechanism has more to work with when your input is well-structured — vague or ambiguous questions genuinely produce vaguer, less useful attention patterns, and therefore vaguer answers.
And it explains why these tools are so good at language tasks and comparatively weaker at things like arithmetic or strict logic — the entire system is built around predicting plausible text, not performing calculations or applying formal rules.
I'll be sharing more as I go deeper into both modules this term — there's something genuinely worthwhile in understanding the mechanics behind tools we now use every day, whether or not you ever plan to build one yourself.