ONE FORWARD PASS · 124M PARAMETERS · 0.93 GFLOPS · ONE WORD
Five words go in; a sixth comes out. This tutorial rides “The cat sat on the” through a real model — GPT-2 small, every number public — from token IDs to embeddings, through attention and twelve layers, to a 50,257-way vote, then answers the classic question: what actually separates an encoder from a decoder? A Σ FLOPs meter runs the whole way. The scenes play themselves — watch, or just scroll. are tappable. You are the prompt.
A machine can only multiply matrices, and “The cat sat on the” is not a matrix. What follows is the clerical step nobody stays for and everybody depends on: turning five words into five integers a model can look up. The Σ FLOPs meter — the running count of floating-point operations spent choosing the sixth word — starts at zero, and this whole chapter will not move it.
The prompt is “The cat sat on the” — five words, 18 characters, and the whole of this tutorial. The machine is GPT-2 small: 124 million arranged as 12 layers, 12 attention heads, and a working width of 768 — small enough that every number ahead is public and printable, built from exactly the same blueprint as the frontier models. Before any of those 124M numbers get involved, a tokenizer splits the text into 5 and swaps each for an integer ID. The Σ meter reads zero and will stay there all chapter: splitting text is table lookups, not arithmetic.
Where did those 50,257 entries come from? No linguist chose them. builds the vocabulary by brute statistics: start from the 256 possible bytes, find the most frequent adjacent pair in the training text, merge it into a new symbol, and repeat 50,000 times. Frequent words fuse into single tokens (t + h → th → the); rare words stay shattered into pieces. The algorithm predates the transformer by 23 years — it was invented in 1994 to compress files, and a tokenizer is exactly that: a compressor for text.
Our prompt contains “the” twice — and the model receives two different integers. The at the start is token 464; the with its leading space is token 262; a bare lowercase the would be 1169. Capitalization and whitespace are baked into the ID, and the letters inside a token are invisible from then on — which is why models famously fumble “how many r’s in strawberry”: they have never seen an r, only chunks. Nothing about the text is lost, but its atoms are chunks, not characters.
Both obvious alternatives lose. Feed the model letters and the vocabulary is a tidy 256, but every sentence gets long — our prompt becomes 18 steps instead of 5, and attention (next chapter) charges by the square of the length. Feed it whole words and sequences are short, but the vocabulary is unbounded and any typo or new name becomes an unknown word the model simply cannot read. Subwords split the difference — and because the merge ladder bottoms out at raw bytes, nothing is ever out of vocabulary: worst case, a weird string just costs one token per byte.
The merges were learned from WebText. GPT-2’s 50,000 merges were extracted from ~40 GB of pages linked from Reddit with ≥3 karma — so the vocabulary encodes 2019 internet frequency. That is why “ the” (with its leading space) is the 262nd entry: spaces glue to word-starts because that pairing is overwhelmingly frequent in English text.
The 50,257th token. 256 bytes + 50,000 merges leaves one special entry: <|endoftext|>, ID 50256 — the document separator GPT-2 saw between every training text. It doubles as this tutorial’s sign-off.
Tokenizers are compressors. Our prompt: 18 characters ÷ 5 tokens = 3.6 characters per token, right on the English average of ~4. A vocabulary is judged exactly like a compression codec — by how few symbols it needs for typical text.
The lineage. GPT-2’s exact vocabulary (r50k) was reused by GPT-3; GPT-4’s cl100k grew to ~100K entries and o200k to ~200K. Bigger vocabularies buy shorter sequences at the price of a fatter embedding table and final softmax — the tradeoff this chapter’s last step priced out.
Five integers are still not a matrix. This chapter turns each ID into the thing the model actually thinks with — a 768-number vector where meaning is geometry — then stamps on word order and opens the highway those vectors will ride for the rest of the pass. By the end of it the meter has finally twitched: 3,840 FLOPs, the cheapest arithmetic this prompt will ever buy.
At the model’s front door, each ID is swapped for its — row #3797 of a 50,257 × 768 table of learned numbers. Token cat becomes a vector of 768 floats; so do the other four. That table alone holds 38.6 million parameters — 31% of the entire model is, quite literally, a dictionary of meanings. And the meter still reads zero: a lookup is a copy, not a computation. The prompt is now a 5 × 768 grid of numbers, ready for arithmetic.
Why 768 numbers? Because in a high-dimensional space, meaning behaves like geometry. Words that appear in similar contexts get pulled toward similar vectors during training — cat, kitten, and dog end up neighbors while on and the drift elsewhere — and directions encode relations. The 2013 party trick that announced this era: take the vector for king, subtract man, add woman, and the nearest word is queen. Nobody programmed that; it fell out of predicting text. GPT-2’s embeddings inherit the same property, 768 dimensions deep.
There is a bug to fix before any thinking happens: attention (next chapter) is order-blind. Fed only the five embeddings, the model literally could not tell “dog bites man” from “man bites dog” — a bag of words. The fix is almost embarrassingly simple: add a learned to each token vector. GPT-2 keeps 1,024 of them, one per slot it can see; the 2017 paper used frozen sine waves instead. Five positions × 768 additions = 3,840 FLOPs — the first arithmetic of the entire run. The meter moves at last.
Each token’s 768-dim vector is now on what interpretability researchers call the — a running workspace that flows unbroken from here to the final verdict. Every layer ahead will read this vector, compute an edit, and add the edit back: x + f(x), never f(x) alone. Layers annotate; they never overwrite. This one design choice (the 2017 paper borrowed it from computer vision) is what lets gradients survive 12 — or 96 — layers of depth, and it is the spine every scene in this tutorial hangs off.
Sinusoids: the 2017 original. “Attention Is All You Need” added no learned positions at all — each slot got a fixed pattern of sines and cosines at geometrically spaced wavelengths, so any two positions’ distance is readable from their encodings and the scheme extends past any trained length in principle.
Learned positions: GPT-2’s choice. Simpler and slightly stronger in practice: just train a 1,024 × 768 table like a second embedding matrix. The cost is a hard ceiling — position 1,025 simply has no vector, which is one reason GPT-2’s context ends exactly there.
RoPE: what modern models use. Llama, Qwen, and most current LLMs rotate the query and key vectors by a position-dependent angle instead of adding anything — relative positions fall out of the geometry, and clever rescaling stretches trained models to million-token contexts.
Why add rather than concatenate? Adding keeps the width at 768 (concatenation would balloon every matrix downstream), and in high-dimensional space nearly-orthogonal subspaces are abundant — the network learns to keep “what” and “where” separable inside one vector.
This is the chapter the 2017 paper is named after. Attention is how a word looks at other words — how “the” discovers it belongs to “cat sat on…” — and it is nothing more exotic than dot products, a division, and a normalizer, run in parallel. We watch it through one attention head in layer 1, five tokens wide, and the Σ meter takes its first real jump: from 3,840 FLOPs to 24 million.
Attention starts by projecting each token’s vector three ways, through three learned matrices, into a : the query is what this token is looking for (“I’m a determiner — where’s my noun phrase?”), the key is what it advertises to others (“I’m a subject noun”), and the value is the payload it will hand over if anyone picks it. Three projections × 768 numbers = 2,304 derived numbers per token before any looking has happened. The metaphor is a library: queries are search requests, keys are the spines on the shelf, values are what’s inside the books.
Now every query meets every key: score(i, j) = qᵢ · kⱼ ÷ √64. A dot product is agreement made numeric — vectors pointing the same way score high — so the 25 pairings of our 5 tokens fill a 5 × 5 grid of . The odd little ÷ √64 (each head works in 64 dimensions, √64 = 8) is load-bearing: raw dot products grow with dimension, and unscaled they would shove softmax into a regime where one token takes literally all the attention and gradients die. One divide per score keeps the contest fair.
Before those scores are used, GPT-2 overwrites half its own grid: every cell where a token would look at a later token is set to −∞. That triangle is the , and it exists because of how the model trains — predicting every position’s next token simultaneously, which is only honest if position 3 cannot see position 4. After softmax, e^−∞ = 0: not approximately zero, exactly zero. Hold on to this triangle. In chapter 05 it turns out to be the entire answer to “encoder vs decoder.”
turns each row of surviving scores into probabilities that sum to exactly 1.0 — so attention is literally a budget: every token gets one unit of focus to spend across the tokens it is allowed to see. Our final the might spend 34¢ on “cat”, 24¢ on “sat”, keep 20¢ on itself, and scatter the change — a determiner hunting for the noun phrase it belongs to. Exponentiate, normalize, done: the same two-step trick will referee the 50,257-way vote in chapter 04.
Payout time: each token walks away with the weighted blend of everyone’s value vectors — output = Σ weightⱼ · vⱼ — so “ the” now literally carries a piece of “cat” inside it. One more learned matrix (the output projection) reshapes that mix, and the result is added back onto the residual highway: x + attention(x). The bill for layer 1’s attention, all heads, all 5 tokens: ~24M FLOPs — and here is the twist: 99% of it went on the q/k/v/output projections. The famous score grid itself cost a rounding error.
The full shapes. Our 5 × 768 grid multiplies a 768 × 2,304 matrix to mint Q, K, V (5 × 768 each). Per 64-dim head: scores are Q·Kᵀ → 5 × 5, softmax per row, times V → 5 × 64. Twelve heads’ outputs concatenate back to 5 × 768 and meet the 768 × 768 output projection. One formula holds the whole chapter: Attention(Q, K, V) = softmax(QKᵀ ÷ √dₖ) · V.
Why “scaled dot-product attention”. The paper’s Figure 2 is exactly our five steps in one box: MatMul → Scale → Mask → SoftMax → MatMul. Dot-product attention was chosen over the additive variant for speed — it compiles to the matrix multiplies GPUs are built for.
The FLOPs receipt, itemized. For 5 tokens: q/k/v projections 2 × 5 × 768 × 2,304 ≈ 17.7M; output projection ≈ 5.9M; the score grid and value mixing ≈ 0.15M. The bottleneck of attention at short context is not attention — it is the linear algebra wrapped around it. (At 1,024 tokens the n² term stops being a rounding error; chapter 06 sends the bill.)
Softmax, precisely. Each score row subtracts its max (numerical stability), exponentiates, and divides by the sum. The −∞ cells become e^−∞ = 0 before normalization — which is why masked positions get exactly nothing, and why the remaining weights still sum to exactly 1.
One attention pass does not make a mind. This chapter widens the picture twice: sideways, to the 12 heads that run in parallel per layer, then upward, through the MLP blocks and the full 12-layer climb. It is the expensive chapter — the Σ meter multiplies 35-fold, from 24M to 851M FLOPs — and by the top of the stack, “ the” has become a 768-dim summary of everything five tokens could tell each other.
Chapter 02 showed one ; the layer actually runs twelve at once. The 768 dimensions split into 12 parallel circuits of 64, each with its own q/k/v matrices, each free to look for something different, and their outputs concatenate back to 768. And they do specialize — when interpretability researchers opened up GPT-2 they found previous-token heads, punctuation heads, and the famous induction heads, which spot a repeated pattern (“A B … A”) and bet on B coming next. Nobody programmed any of that. It condensed out of gradient descent.
After attention, each token visits the layer’s second machine: the , which widens 768 → 3,072, applies a GELU nonlinearity, and narrows back — 768 → 3,072 → 768. No token looks at any other here; it is pure per-token processing, and it is where most of the model’s knowledge appears to live: attention fetches the right context onto each token, the MLP looks things up about it. At 4.7M weights per layer versus attention’s 2.4M, the “thinking” blocks outweigh the famous part two to one.
One layer = attention + MLP, both adding their edits to the . Now repeat: 12 identical blueprints with 12 different sets of learned weights, and our five tokens ride the full stack. What changes floor by floor is the level of abstraction — probing studies show early layers doing spelling-and-syntax work while later layers track meaning, with the model’s guess about the next word sharpening layer over layer. Depth is the point: each floor reasons over the previous floor’s annotations, not over raw text.
The stack’s bill: ~71M per layer for our five tokens, ×12 = 851 million — every one of them a multiply or an add, no magic anywhere in the building. For scale: a 1976 Cray-1 supercomputer would have ground on this for five seconds; the chip in your phone clears it in about a millisecond. That asymmetry — the blueprint is cheap, the era makes it fast — is the quiet reason this 2017 architecture ate the world.
The logit lens. Decode the residual stream at every floor — as if the model had to answer early — and the prediction visibly sharpens: gibberish low, plausible parts of speech by the middle, near-final answers in the last layers. The stack is not 12 different algorithms; it is one guess being iteratively refined.
Head atlases. Mapping all 144 heads of GPT-2 small became a small research industry: previous-token heads, duplicate-token heads, heads that track syntactic dependencies, and induction-head pairs (a “pointer” head in an early layer feeding a “copier” downstream). Induction heads emerge abruptly during training, right when in-context learning ability jumps — one of mechanistic interpretability’s cleanest results.
Depth vs width. At equal parameter count, deeper usually beats wider — until it doesn’t: past a few dozen layers the returns flatten and training stability, not capacity, becomes the fight. Modern frontier models sit near GPT-3’s ~100-layer mark while growing enormously in width and head count instead.
The bookkeeping we skipped. Each block is wrapped in a LayerNorm (GPT-2 moved it before the block; the 2017 paper had it after), which rescales the stream for stability at a negligible FLOPs cost. Modern models swap it for the even cheaper RMSNorm — refinements, not redesigns.
The stack has done its thinking; now 768 numbers must become one word. One row survives, one giant matrix turns it into 50,257 scores, softmax turns scores into odds, and a die roll turns odds into text. Every probability below is measured from the real GPT-2 small running on our prompt — which is how we know the cat is about to sit somewhere unexpected.
Twelve layers produced five refined vectors — and now four of them are thrown away. Only the final position, “ the”, predicts the next token, because only it has attended over the whole prompt; positions 1–4 could never see past themselves. They were not wasted: their keys and values were read at every one of the 144 head-visits on the way up, and chapter 06 will cache them. But the verdict belongs to one row of 768 numbers.
That one vector now multiplies the unembedding matrix — and GPT-2 pulls a thrifty trick: it reuses the 50,257 × 768 embedding table from chapter 01, transposed. Reading the dictionary backwards turns 768 numbers into 50,257 , one raw score per vocabulary entry, at a cost of 2 × 768 × 50,257 ≈ 77.2M FLOPs — the single most expensive matrix multiply of the pass, spent entirely on the scoreboard. Every token GPT-2 has ever known, from “ the” to “ Archdemon”, just received a number.
returns for its second job — the same exponentiate-and-normalize that split attention budgets now referees all 50,257 candidates at once. We ran the real GPT-2 small on our prompt, and its verdict is a genuine surprise: the nursery-rhyme “ mat” is nowhere near the top. “ floor” leads at 7.6%, then “ bed” (6.5%), “ couch” (5.4%), “ ground” (5.2%) — “ mat” limps in 31st at 0.4%. Also worth staring at: the winner got 7.6%, not 90. On five words of context, the model is honestly, numerically unsure.
A distribution is not yet a word — someone must pick. Greedy decoding takes the argmax every time: this prompt would yield “ floor” forever, and long greedy runs loop into repetition. Real systems reshape the odds with (divide the logits by T before softmax: low T sharpens toward the favorite, high T flattens toward chaos), trim the long tail with , then roll a die on what survives. That die roll is the only randomness in the entire building — everything before it was deterministic arithmetic. The sixth word is chosen: “ floor”.
The greedy repetition trap. Always taking the argmax sounds sensible and reads terribly — GPT-2 era models famously spiraled into “I don’t know. I don’t know. I don’t know.” Loops are self-reinforcing: repeated text raises the probability of repeating it. A little randomness is not a luxury; it is an exit.
Beam search, and why chat models skip it. Beam search tracks the k most probable whole continuations — great for translation, where faithfulness beats flair, but in open-ended text the most probable continuation is reliably the dullest. Chat models sample instead.
Top-k vs top-p. Top-k keeps a fixed shortlist (say, 40 tokens); top-p keeps the smallest set whose probabilities sum to p (say, 0.9), so the shortlist grows when the model is unsure and shrinks when it is confident. Both exist to kill the tail: without them, a 1-in-50,000 nonsense token eventually gets rolled.
Where temperature acts. Dividing logits by T before softmax: T → 0 makes the argmax certain; T = 1 reads the model’s honest odds; T > 1 flattens them. The model’s knowledge never changes — only how boldly the die is thrown against it.
The chapter this tutorial is subtitled for. You have now watched a decoder end to end — so what is an encoder, why did the original 2017 machine have both, and why does nearly every model you talk to today have only one? The answer is smaller than the words suggest: it is the triangle from chapter 02. The Σ meter holds at 0.93B all chapter — no new arithmetic, just the same arithmetic pointed in different directions.
June 2017, eight Google researchers post “Attention Is All You Need” — a machine-translation model, because that was the hard problem of the day. The famous diagram is two towers: a 6-layer that reads the whole source sentence at once, and a 6-layer that writes the translation word by word. The base model was 65M parameters and trained in 12 hours on 8 GPUs; the big one (213M) took 3.5 days and set records: 28.4 BLEU English→German, 41.8 English→French. Everything you have watched in this tutorial so far is the right-hand tower, unplugged from the left.
Take our chapter-02 score grid and simply refuse to draw the triangle: every token attends to every token, backward and forward. That is the entire recipe for an encoder, and it is what BERT is — an encoder-only stack trained in 2018 to fill in masked blanks, using future context on purpose. It cannot write (it has no notion of “next”), but for understanding — search queries, classification, extraction — bidirectional context is a superpower. Within a year of publication, BERT was reranking 1 in 10 Google searches. Same arithmetic as our pass, same bill: reading in both directions costs no extra FLOPs.
Keep the triangle instead, throw away the other tower, and you have the GPT recipe: a decoder-only stack whose only training signal is “predict the next token” — which, it turns out, teaches grammar, facts, and reasoning as side effects. The lineage is steep: GPT-1 (June 2018, 117M parameters) → GPT-2 (Feb 2019, 1.5B — our specimen is its smallest sibling) → GPT-3 (May 2020, 175B). Parameters ×1,500 in 23 months while the architecture diagram barely moved a box.
How did the 2017 machine’s towers talk? A third flavor of attention: , where the queries come from the decoder but the keys and values come from the encoder’s output. The decoder asks “what should I write next?” and the answers live in the other tower — that is the moment a translation crosses languages. The pattern outlived translation: Whisper transcribes speech with the 2017 blueprint nearly intact, an audio encoder feeding a text decoder across exactly this bridge.
The scorecard, then. Encoder-only (BERT): sees everything, understands, cannot generate. Decoder-only (GPT, Llama, Claude, nearly every modern chat model): sees only the past, generates one token at a time. Encoder-decoder (T5, Whisper, translation systems): reads with one tower, writes with the other, transforms. Strip away the family names and the difference is one triangular matrix of −∞ — the mask you met in chapter 02. The triangle won the decade: it turned out that “predict the next token,” scaled hard enough, does everything else too.
Where the eight authors went. Every author of “Attention Is All You Need” has left Google. Aidan Gomez co-founded Cohere; Noam Shazeer built Character.AI (and was famously re-acquired by Google for billions); Llion Jones co-founded Sakana AI; Ashish Vaswani and Niki Parmar founded Adept, then Essential AI; Illia Polosukhin co-founded NEAR Protocol; Jakob Uszkoreit founded Inceptive; Łukasz Kaiser joined OpenAI. The paper itself passed ~170,000 citations — one of the most-cited papers of the century, in any field.
T5’s reframing. Google’s 2019 encoder-decoder treated every task — translation, summarization, even classification — as text-to-text (“translate English to German: …” in, German out), showing one seq2seq architecture could be the universal interface. Its heirs still run production translation and captioning.
Why decoder-only won chat. One tower means one stack to scale, a single self-supervised objective with no encoder/decoder plumbing, and a prompt that can hold instructions, context, and conversation in one stream — the “source” text just lives in the same context window the answer will. Bidirectional understanding turned out to be learnable implicitly at sufficient scale.
Encoders never died. Embedding models that power semantic search and RAG retrieval are BERT’s descendants — when the job is “represent this text,” reading in both directions still beats reading left to right.
One word cost 0.93 billion FLOPs — and one word is never the job. This final chapter closes the circuit. The predicted token feeds back in, a cache spares the machine from re-deriving its own past, and an n² tax sets the horizon. The same blueprint scales from our 124M-parameter specimen to every frontier model you have ever talked to.
“ floor” does not get remembered — it gets appended. The prompt is now “The cat sat on the floor”, six tokens, and the entire machine runs again from chapter 00 to produce word seven. That is , and it is the punchline of the whole architecture: a model “writing a paragraph” is the same forward pass rerun once per word, with no memory between passes except the text itself. Every chatbot conversation you have ever had is this loop, spinning.
Rerunning everything would be criminal waste: the keys and values of “The cat sat on the” cannot change — the causal mask guarantees old tokens never see new ones. So inference engines keep a : every token’s keys and values, all 12 layers, filed on first computation. Each new word then computes only its own q/k/v and reads the pad. Per token in GPT-2 that pad holds 2 × 12 × 768 = 18,432 numbers (~36 KB in fp16) — and this bookkeeping trick is the entire reason chat is affordable: without it, token n would cost n times too much.
The cache kills redundant compute, but one bill cannot be dodged: attention compares every token with every earlier token — n² pairs. Our 5 tokens made 25; GPT-2’s full 1,024-token makes ~525,000; a million-token context makes ~500 billion. Same formula, six orders of magnitude apart. That square is why context windows are finite, why long contexts cost real money, and why 2024’s million-token headlines (and the attention-taming tricks behind them) were headlines at all.
Now scale the blueprint and change nothing else. GPT-3 is 96 layers × 96 heads × 12,288 dims = 175 billion parameters — the same diagram as our specimen, printed taller and wider — and by the 2-FLOPs-per-weight rule it pays ~350 GFLOPs per token to GPT-2 small’s ~0.25: the same walk, ×1,400 the bill. Finish our story properly: a 100-token reply from GPT-2 costs ~25 billion FLOPs, one 0.93B-FLOP verdict at a time. The loop is the product. Everything else was one turn of it.
The scaling laws. Kaplan et al. (2020) found loss falls as a smooth power law in parameters, data, and compute — the discovery that made “just build it bigger” a rational strategy. Chinchilla (2022) fixed the exchange rate: compute-optimal training wants ~20 tokens per parameter, which meant most giant models of the era were undertrained for their size.
What changed since 2017 — and what didn’t. The living pieces have been swapped: LayerNorm moved before the blocks then became RMSNorm, learned positions gave way to RoPE, multi-head attention slimmed to grouped-query attention, and mixture-of-experts made most weights sleep through most tokens. The skeleton — embed, attend, MLP, residual stream, unembed, loop — is untouched. A 2017 reader can still follow a 2026 model’s diagram.
Taming the square. FlashAttention reorganized the n² computation to respect GPU memory (exact, just faster); sliding-window and sparse variants trade global sight for linear cost; state-space models (Mamba) drop attention entirely. So far, for frontier quality, the square keeps being paid.
Prefill vs decode. Serving splits the loop into two regimes: the prompt is processed in one parallel pass (compute-bound), then tokens stream out one by one (memory-bound, the KV cache doing the heavy lifting). It is why your answer starts after a pause and then arrives at typing speed.
The whole path on one strip: five words became five integers, the integers became geometry, attention let them read each other, twelve floors of arithmetic refined them, and one row of 768 numbers bought a 50,257-way vote. That is 928 million FLOPs for one word — “ floor”, 7.6%, and the nursery rhyme overruled. Then the loop arrow turns a word machine into everything else.