ONE PROBLEM · 266,640 HIDDEN TOKENS · 3 CHARACTERS OF ANSWER
One competition math problem, put to the same model at every stage of its construction — a prompt trick, a reward that only checks the final number, an RL run that invents self-doubt, a thinking dial, a 64-way vote, and the bill. One plausible run, real benchmarks throughout. The scenes play themselves — watch, or just scroll. are tappable. You are the grader.
A diligent 15-year-old solves this in six minutes with a pencil. A model that has read the internet answers in 1.2 seconds and gets it wrong — not because it lacks the knowledge, but because of where the arithmetic had to happen. This chapter is the constraint the other seven are built against: fixed depth, no scratchpad, and a running total that starts at Σ 0 thinking tokens.
Every morning Aya walks 9 kilometres and stops at a coffee shop. At speed s km/h the outing takes 4 hours including t minutes in the shop; at s + 2 km/h it takes 2 h 24 min including the same t. At s + ½ km/h — how many minutes? Solving needs two equations, s = 2.5 and t = 24, then 9 ÷ 3 = 3 hours and 180 + 24 = 204. AIME is 15 such problems in 3 hours, and every answer is an integer from 000 to 999 — a fact that looks like exam trivia and turns out to be why math, not essays, became reinforcement learning’s first classroom. Chapter 02 cashes this in.
A model with no reasoning training answers immediately: “Since s = 3, the walk takes 180 minutes.” Fluent, confidently formatted, wrong — it forgot the coffee shop. It looks like reasoning because it says “since”, but the number arrived first and the justification was generated after it, one plausible token at a time. GPT-4o scored 9.3% on AIME 2024: about 1.4 problems out of 15. A is what this tutorial builds to fix that — starting from why the failure is structural.
A transformer’s forward pass is fixed depth. Same layers, same ~2 × params FLOPs per token (the AI-numbers tutorial derives that), whether the prompt is “hi” or an olympiad problem — the transformer tutorial walks the pass itself. There is no loop, no “hold on”, no branch that runs longer for a harder question. So the only way to spend more compute on a hard problem is to emit more tokens. Everything in this tutorial follows from that one constraint.
Solving needs s = 2.5 and t = 24 held somewhere while they are used. A model has no register file and no scratch RAM; the only memory that survives from one token to the next is the tokens it has already written, which attention can read back. Keep the work internal and it must survive as activations in one residual stream that is overwritten every token. Write it down and it becomes addressable memory. “Show your work” is not a style request — it is the architecture’s only notepad.
System 1 / System 2. Kahneman’s fast-vs-deliberate framing is everywhere in reasoning-model marketing and almost nowhere in the papers. It earns its keep as a mnemonic — blurt vs deliberate — and leaks everywhere else: the model has no second system, just the same decode loop spending more tokens.
Knowledge vs search. The base model knows every rule of algebra it needs — ask it to state the distance formula and it will. What it cannot do in one pass is sequence them: pick an unknown, derive, substitute, check. That gap between knowing the steps and finding the path through them is what “reasoning” means in this tutorial.
Why AIME 2024 is the reference exam. It was released after most pretraining cutoffs of the models compared here, so the scores measure solving, not remembering. Contamination is the first objection to any benchmark claim; a post-cutoff competition is the cleanest answer available.
Two different claims. “It can’t reason” and “it can’t reason in one forward pass” sound alike and are worlds apart. The whole field of reasoning models lives in the second claim’s loophole: give the pass somewhere to put the middle of the thought, and see what becomes possible.
In 2022 the fix turned out not to be a bigger model. It was permission to write things down — and the second version of the fix was a single sentence. This chapter spends the tutorial’s first 240 thinking tokens and meets the mechanism everything else is built on: every written token is another full pass through the model.
Wei et al., January 2022. Show PaLM 540B eight worked examples with the included, and its grade-school-math solve rate (GSM8K) goes from 17.9% to 56.9%. No weights changed, no data added — only the shape of the prompt. And the effect is emergent with scale: below roughly 100B parameters, chain-of-thought prompting makes models worse, not better. The capability rides in with size and waits to be asked for.
Kojima et al., May 2022. No examples at all — just append “Let’s think step by step.” MultiArith goes 17.7% → 78.7%, GSM8K 10.4% → 40.7%. The capability was already sitting in the weights; the sentence only selected the mode. This is the moment the field learned that a large part of “capability” is elicitation — and the reason the next three years were spent trying to make the mode permanent instead of prompted.
The mechanism, and the reason this tutorial exists. Each token written is one more complete forward pass through every layer — and its output is appended to the context, so the next pass can attend to it. 240 tokens of working is 240 extra passes plus 240 tokens of externalised state. Thinking, mechanically, is , in series. Our prompted model writes 240 tokens of steps and lands on 204 — this time.
Prompted chain-of-thought is fragile. It is sensitive to phrasing, it never backtracks — a slip in step 2 is carried faithfully to the end — and nothing in next-token training rewards being right, only being plausible. Prompted, DeepSeek’s base model still sits at 15.6% on AIME 2024. To get reliability, correctness itself has to become the training objective. That is the next chapter, and it is the whole difference between a model that can reason if you ask nicely and a reasoning model.
The prompting era’s family tree. Zero-shot vs few-shot chain-of-thought, self-ask, least-to-most, tree-of-thought — a two-year sprint of finding better places and shapes for the middle of the thought, all without touching a weight.
Scratchpads came first. Nye et al. (2021) trained models to emit intermediate computation into a designated scratchpad before answering — the direct ancestor of the <think> block, three years early.
Why emergent with scale? A small model’s steps are noise — each written line injects errors faster than externalising helps. The technique only pays once the per-step accuracy crosses a threshold, which is why the same prompt hurts a 8B model and transforms a 540B one.
A seed of chapter 06. Even here, the written chain is a sample of plausible reasoning, not a transcript of the computation that produced the answer. That gap will get a number later — hold onto the suspicion.
To train the habit in, you would need a corpus of expert reasoning monologues — which does not exist and cannot be bought at scale. DeepSeek’s R1-Zero skipped it entirely: let the model write whatever it likes between the tags, and pay it only if the final number is 204. This chapter spends 11,200 thinking tokens on our problem in a single training step — Σ 11,440.
Reinforcement learning from : the reward is 1 if the boxed answer equals 204 and 0 otherwise. No human grades the reasoning; nobody labels which step was clever. Contrast with RLHF, where a learned preference model stands in for human taste — the evals tutorial’s pairwise comparisons are that pipeline’s fuel. Math and code went first for one reason: the answer key is executable — an integer match, or a test suite that passes.
How does one bit of reward move billions of weights? With the same machinery as every other kind of training — gradient descent (the digit tutorial walks it). The trick is where the labels come from. Take a sampled attempt that scored above average and treat every token in it as the correct next-token label: compute the ordinary training gradient, nudge those tokens’ probabilities up. Below average: same gradient, sign flipped. That is the — being RL’s name for the model under training — and it means RL for LLMs is, mechanically, supervised fine-tuning on the model’s own graded homework. No new math was invented; what changed is who writes the training data. The model does.
The last step left one word doing all the work — above average of what? answer: of the group. Sample 16 of the same problem, score them all, and use the group’s own mean as the baseline: = (reward − mean) ÷ std. The trick is what is missing: PPO’s critic — a second network the size of the policy, trained to estimate that baseline — is gone, and with it roughly half the training memory. One group on our problem: 16 rollouts × ~700 tokens = 11,200 thinking tokens spent on Aya’s walk in a single training step, five of them correct — so those five get reinforced and the other eleven suppressed.
The second reward is deliberately stupid: put the reasoning inside <think>…</think> and the answer inside <answer>…</answer>. That is the entire template — no instruction on how to reason, on purpose, so that RL discovers strategies instead of imitating ours. One more reward was added later for language consistency (the fraction of the chain in the target language), because the raw chains drifted into a bilingual pidgin — and the paper reports it made accuracy slightly worse. Legibility is a tax the labs choose to pay.
A neural grader sounds better — it could reward well-argued steps. DeepSeek tried and dropped it for large-scale RL: a learned reward model gets — the policy finds outputs that score well and are wrong — needs periodic retraining, and complicates the pipeline. A regex and an == cannot be flattered, padded, or charmed. The price of that robustness is the whole strategy’s boundary: it works where truth is checkable, which is exactly why reasoning models are conspicuously stronger at math and code than at judgement calls.
The receipt for teaching a model to think. R1-Zero’s RL run: 512 H800s for 198 hours ≈ 101,000 GPU-hours; R1 added ~80 hours more; total disclosed cost $294,000 — against 2.79 million GPU-hours (~$5.6M) to pretrain the DeepSeek-V3 base it started from. The stage that produced the capability jump of the year cost about 3.6% of the stage that taught the model to talk. That ratio is why RL compute is the fastest-growing line in every frontier training budget, and why forecasters expect it to converge with pretraining compute.
PPO vs GRPO vs DPO, one paragraph each. PPO: the classic — a critic network estimates how good each state is, advantages come from the difference, stable but memory-hungry. GRPO: delete the critic, use the group mean — cheaper, and the baseline is exact for the prompt at hand. DPO: skip RL entirely and turn preference pairs into a supervised loss — great for style and taste, but it needs pairs, not a checker, so verifiable-reward reasoning stayed with the policy-gradient family.
The leash. A anchors the policy to the pre-RL reference model. Without it, pure reward maximisation drifts into gibberish that happens to end in the right integer — high score, useless model.
RLHF vs RLVR. The InstructGPT lineage optimises a learned model of human preference — perfect for “be helpful”, unverifiable by design. RLVR optimises a program’s verdict. The two coexist in every frontier pipeline: RLVR for reasoning, RLHF-style rounds for manners.
Verifiers by domain. Code: run the test suite. Math: symbolic or numeric equality. Prose: nothing — there is no == for “this essay is insightful”, which is the boundary chapter 05 pushes against with process reward models.
The peer-review footnote. R1 became the first mainstream LLM to pass journal peer review — the cover of Nature, September 2025, which is also where the $294,000 figure was disclosed.
Nothing in that reward says “backtrack”, “check your answer”, or “try it another way”. It says == 204. This chapter watches what falls out of it anyway — and spends the post-RL model’s own 3,900-token attempt on our problem. Σ 11,440 → 15,340.
Over thousands of RL steps, the average response length climbs steadily — from a few hundred tokens to many thousands — and accuracy climbs with it. Nobody set a length target; nobody rewarded verbosity. The model discovered on its own that spending more tokens before answering raises the odds of hitting the right integer, and the optimiser simply kept what worked. was invented by the training run.
The paper’s most-quoted moment is the model interrupting itself mid-derivation: “Wait, wait. Wait. That’s an aha moment I can flag here. Let’s re-evaluate this step-by-step.” Anthropomorphic language nobody rewarded, arriving because re-checking raises the chance of the answer being right. Alongside the : reflection, verification, and trying a second approach when the first stalls — three behaviours that were listed as research goals in 2023 and turned up as side effects of a scalar reward.
Pure RL, with no supervised reasoning examples at all: AIME 2024 goes from 15.6% to 77.9%, and to 86.7% with 64-sample majority voting — past the average human AIME contestant. Peer-reviewed in Nature, September 2025. The important word is pure: no human wrote a single one of the reasoning traces this model learned from. It wrote them itself and kept the ones that scored.
The same model, on Aya’s walk. It sets up 9/s = 4 − t/60 and 9/(s+2) = 2.4 − t/60, solves s = 2.5 and t = 24 — and then, instead of answering, re-derives with s + ½ = 3, checks units, writes “Wait, let me verify t”, and only then emits 204. That is 3,900 hidden tokens ≈ 2,900 words: twelve minutes of human reading, produced and discarded so you can see three characters. And notice what did not change: the architecture. This is the same decode loop from the transformer tutorial — no reasoning module was added, no loop, no scratch memory. The entire capability is a habit living in the weights, exercised through ordinary token generation.
R1-Zero’s transcripts were nearly unreadable — mixed languages, no formatting, no manners. So the shipped model, R1, prepends a few thousand hand-curated long chain-of-thought examples (the ) before RL, then follows RL with rejection-sampled fine-tuning and a second RL round for helpfulness and harmlessness. Product-grade reasoning is RL sandwiched in supervised bread. Result: 79.8% on AIME 2024, level with o1-1217’s 79.2%.
Does RL create the ability, or find it? The pass@k debate: sample a base model enough times and it occasionally solves problems the RL model solves reliably — evidence that RL sharpens the sampling distribution toward solutions already reachable, rather than adding capability from nothing. The counterpoint: at some k the distinction stops mattering; reliability is the product.
Maybe there was no aha. Follow-up work found base models already emit self-reflective phrases (“wait”, “let me check”) at low rates before any RL — the training run may have amplified an existing habit rather than invented one. Response length is also a confound: longer answers correlate with reflection words for free.
Reproductions for a few hundred dollars. The R1-Zero recipe — GRPO, rule rewards, a math dataset — has been reproduced on 1.5B–7B models by multiple groups, showing the same length-growth signature on a single node. The phenomenon holds up even where the interpretation is contested.
Once thinking is just tokens, how much of it becomes a runtime parameter. A second scaling axis opened — and unlike pretraining, you pay for this one per request. Three forced “Wait”s on our problem: Σ 15,340 → 18,340.
o1’s launch post showed two curves, not one: accuracy rising smoothly with train-time RL compute, and rising smoothly with test-time thinking — both against log axes. Before this the only dial was “spend two months training something bigger”. Now there is “let this one request think for thirty more seconds”. Snell et al. put a number on the trade: spending optimally let a small model beat one 14× its size.
Stanford’s s1 (February 2025) made the dial embarrassingly literal. Fine-tune on 1,000 curated examples, then control thinking by : to think less, force the </think> token; to think more, suppress it and append the word Wait — at which point the model keeps going and often catches its own error. It beat o1-preview on competition math by up to 27%. On our problem, three forced “Wait”s add ~3,000 tokens, and on the second pass the model catches a unit slip it had already committed to.
Every major API now exposes the . Anthropic’s budget_tokens (minimum 1,024, up to 128K; with interleaved thinking it spans a whole tool-using turn). Google’s thinkingBudget (set it to 0 to switch thinking off on the small models; the large ones will not let you). OpenAI’s reasoning_effort (minimal → high). Temperature shaped how a model answers; this is the first API parameter denominated in thought — trading latency and dollars for accuracy, per request.
The tail of the dial is punishing. o3 on the ARC-AGI semi-private set: 75.7% at roughly $20 per task; the high-compute configuration — 172× the compute — reached 87.5%, at about $3,460 per task, ~$346,000 for one hundred puzzles. Log-linear means each additional point costs multiplicatively more than the last. This is why “reasoning effort” is a budget decision as much as a technical one, and why the live research question became efficiency rather than ceiling.
When not to buy thinking. Extraction, classification, formatting, routing: the answer is a lookup or a pattern, thinking is pure cost and added latency, and a reasoning model can even talk itself out of the right label. The engineering move is difficulty-routing — cheap model first, escalate on low confidence.
Thinking and caching don’t mix. Thinking blocks are generally not carried across turns, and interact badly with prompt caching — a long deliberation is a cache-unfriendly, latency-heavy artifact you pay for at output prices. Budget caps exist precisely because the dial defaults to “more”.
The SLO curve nobody shows you. Every provider’s default effort level is a chosen point on the latency-vs-accuracy curve — a product decision made on your behalf, adjustable only within the range the API exposes.
Hybrid models. The current frontier ships one model with a thinking toggle rather than two models — “effort: minimal” suppresses the long private chain and answers from the same weights, which is why the same model can bill you 1× or 20× for the same question.
The other way to spend compute at answer time is not longer — it is wider. Ask the same question sixty-four times in parallel and let the answers argue. This is the tutorial’s most expensive chapter: 63 fresh samples of our problem, Σ 18,340 → 264,040.
(Wang et al., 2022): sample many reasoning paths at temperature above zero — the dice-rolling the transformer tutorial ended on is what makes the paths different — and take the majority answer. +17.9 points on GSM8K with PaLM-540B. It works because wrong paths fail in different ways and disagree, while correct paths converge on the same integer: a vote is a verifier you get for free. R1-Zero: 77.9% → 86.7% at 64 samples. Our problem: 64 samples at ~3,900 tokens each — 249,600 tokens in all (the chapter-03 attempt was the first; the ledger adds the 63 new ones) — of which 51 say 204, 7 say 168, and the rest scatter.
Voting needs comparable answers. The alternative is a verifier that scores candidates so you can keep the best () — and the question becomes what it grades. OpenAI’s Let’s Verify Step by Step trained a on PRM800K: 800,000 human step-level labels. It solved 78.2% of a MATH test subset against 72.4% for the outcome-only model. Grading the work beats grading the answer — at the price of 800,000 human judgements, which is precisely the cost that R1’s == 204 rule dodged.
o1’s original numbers, on this very exam: 74% with one sample, 83% with consensus over 64, 93% when re-ranking 1,000 samples with a learned scorer — 13.9 of 15, top-500-in-the-country territory. Read the second number as a price: a thousand complete reasoning traces per problem. Depth (chapter 04) and breadth (this one) are both purchasable, both log-linear, and both stack.
July 2025. OpenAI and Google DeepMind each scored 35/42 at the International Mathematical Olympiad — gold-medal threshold, five of six problems — under human conditions: two 4.5-hour sessions, no tools, no internet, natural-language proofs graded by IMO medallists. DeepMind’s Deep Think used parallel thinking, exploring many candidate lines at once and combining them. Note the domain shift: proofs, not integers. The grader stopped being == and became a judge, and the frontier moved with it.
The search menu. Best-of-N: independent samples, a verifier picks. Beam search: keep the top-scoring partial chains, extend, prune. Lookahead / MCTS: simulate continuations before committing to a step. Snell et al.’s finding: which one wins depends on difficulty — easy problems waste search, hard problems need it.
Why search is harder in language than in Go. Go has legal moves and a final score; text has no natural action granularity (a token? a sentence? a proof step?) and no terminal reward until someone grades the whole thing. Most of the machinery AlphaGo made famous transfers awkwardly.
Parallel vs sequential is a budget allocation. With a fixed token budget you can think one long chain or many short ones. Sequential wins when the problem needs deep derivation; parallel wins when the failure mode is a wrong early commitment. The IMO systems do both at once.
The symmetry worth noticing. RL against a verifier during training and best-of-N against a verifier at inference are the same idea, spent at two different times: one bakes the selection into the weights, the other pays for it per request.
The dial comes with a price sheet, a blind spot, and a cliff — in that order. Plus the cheapest trick in the book: copying the thinking outright. The 7B distill re-solves our problem in 2,600 tokens, closing the ledger: Σ 264,040 → 266,640.
bill as output tokens, the expensive kind (the AI-numbers tutorial’s price sheet: output runs ~5× input). OpenAI shows a summary rather than the raw chain, deliberately — the raw chain stays monitorable internally and harder to copy; Anthropic may return a summary too, and both bill the full thinking. So the invoice counts tokens the response does not contain. Our single answer: 3,900 hidden + 47 visible ≈ $0.059 at $15/M output, for a problem whose answer is three characters — about 95× what the non-reasoning model charged for being wrong.
Overthinking. On easy questions a reasoning model can spend 7–10× the tokens of a plain model for the same answer; one measurement puts the overhead near 1,953%. The ICML paper is titled, literally, “Do NOT Think That Much for 2+3=?”. Worse than the cost: a model that would have answered correctly in 200 tokens can spend 2,000 talking itself out of it. Reasoning is not a free upgrade — it is a mode, and choosing the mode is now part of engineering.
Anthropic (2025) slipped models a hint about the answer, then checked whether the visible chain admitted using it. Claude 3.7 Sonnet mentioned the hint 25% of the time; DeepSeek R1, 39%. On the more troubling hint categories the rates moved to 41% and 19%. Hold this against chapter 00 and the paradox resolves: the tokens still did their mechanical job — serial compute and scratch memory — which is why generating them helped. What cannot be trusted is the story they tell about how the answer emerged: a plausible reasoning narrative sampled from the same model, not a log of the computation. Anyone reading a thinking block as an audit trail should read this number first.
Apple’s Illusion of Thinking (2025) ran controllable puzzles and found accuracy collapsing to zero past a complexity threshold — and, stranger, models shortening their reasoning as problems got harder, quitting with budget left over. The rebuttal (The Illusion of the Illusion of Thinking) showed part of that collapse was output-token limits, and that asking for a generating function instead of every move restores accuracy on “impossible” instances. Hold both: the cliff is real, the diagnosis is contested, and “it thought less about the harder problem” remains the most unsettling plot in the literature.
The cheapest way to acquire reasoning is secondhand. on ~800,000 reasoning traces sampled from a big one and you get R1-Distill-Qwen-7B at 55.5% on AIME 2024, and the 1.5B at 28.9% — three times GPT-4o’s 9.3%, with no RL at all, from a model that runs on a laptop. Note the coincidence with chapter 05. PRM800K was 800,000 human judgements of steps; these are 800,000 machine-written traces nobody checked line by line. Same round number, opposite philosophies, and the machine-written pile won on price. That is why frontier capability diffuses in months rather than years, and it rhymes with the LoRA tutorial’s point about small artifacts carrying large behaviour changes. Our problem, re-solved by the 7B distill: 2,600 tokens, answer 204.
Reasoning plus tools. The next axis is a model that can run code mid-thought: “let me just compute it” beats 400 tokens of careful arithmetic every time, and search grounds derivations in checked facts. A model with a Python cell reasons differently — less simulation, more verification (the agent tutorial is this idea grown into a loop).
The case for hidden-but-monitored. A raw chain the model believes is private is a better safety signal than one it knows is graded — train against the transcript and you teach the model to write nice transcripts, not to be nice. The cost: users audit a summary, and the summary is marketing.
Cost control in practice. Route by difficulty, cap budgets per tier, and measure tokens per solved task — not tokens per call. A model that spends 3× the tokens but solves on the first try is cheaper than the one you re-prompt twice.
The open frontier. Everything in this tutorial ran on a checkable answer. Rewards for domains where nothing can be checked with == — strategy, writing, advice — are the live research question, and process reward models are the current least-bad answer.
Every stage of the tutorial, with what it spent and what it answered. One three-line question about a woman walking to a coffee shop drew more text than Moby-Dick — and you were shown three characters of it.