ONE PROMPT · 10¹⁵ FLOPS · ¾ OF A CENT · 24 ORDERS OF MAGNITUDE
Seven rungs and a printed card: the tokenizer, 405 GB of weights, the memory wall, prefill, decode, the bill, and the training debt — one 11-word question ridden end to end through a frontier model, every number on the way down earned and checked. The scenes play themselves — watch, or just scroll. are tappable. You are the request.
At 09:00:00.000 you press Enter on an 11-word question bound for a frontier-class model. In 9.3 seconds you’ll have a 300-token answer, and this tutorial will have a running meter: Σ FLOPs, the floating-point operations your request consumes. First comes the strange rung at the bottom — everything that happens to your words before any mathematics touches them, which is why the meter will not move once.
You press Enter on “Why is the sky red at sunset but blue at noon?” — 11 words, 46 characters. A splits it into 12 and replaces each with an integer ID from a fixed dictionary. That integer list is all the model will ever see — no letters, no words, no sounds. The rules of thumb every engineer eventually memorizes: 1 token ≈ ¾ of an English word ≈ 4 characters; 100 tokens ≈ 75 words. Check: 46 ÷ 12 = 3.8 characters per token. And the meter reads zero — splitting text is lookups, not math.
The dictionary — Llama 3’s has exactly 128,256 entries; OpenAI’s o200k has ~200,000 — was written by no linguist. builds it by statistics: start from raw bytes, repeatedly merge the most frequent adjacent pair, stop at the target size. Frequency decides everything: “ the” (leading space included) is one token; rare words shatter into pieces. That’s why models famously fumbled counting the r’s in strawberry — the model receives st·raw·berry-style chunks, and the letters inside a chunk are invisible to it.
The dictionary was learned mostly from English-heavy data, so the same meaning costs different token counts in different languages: an identical sentence can cost 3–10× more tokens in Telugu, Burmese, or Khmer than in English — and since every API bills per token and every is measured in tokens, that’s a real tax on both price and how much fits. Code pays it too: deep indentation and rare identifiers fragment badly. Token count, not word count, is the unit your bill, your latency, and your context budget all share.
Your question does not travel alone. Ahead of it the app stacks a system prompt — identity, rules, tone, formatting — plus instructions and retrieved context: 988 tokens of boilerplate wrapping your 12. Total input: 1,000 tokens, and the model will weigh all of them equally; it has no idea which part “is” the user. This ratio is the norm, not the exception — production prompts are overwhelmingly scaffolding (the agent tutorial’s 4,200-token “constitution” is the same fact again). Everything downstream — prefill time, the bill — scales with the 1,000, not your 12.
At the model’s front door each ID is swapped for its — a learned vector of 16,384 numbers (405B’s hidden dimension). That’s 32 KB of meaning per token; your 1,000-token prompt becomes a 1,000 × 16,384 grid of numbers ready for arithmetic — 16.4 million values, and still the meter reads zero, because a lookup is a copy, not a computation. Everything so far was clerical. The next chapter meets the thing the grid is about to hit: 405 billion learned numbers, and the machines that hold them.
Byte-level fallback. Modern vocabularies bottom out at raw bytes, so there is no such thing as an unknown token — any string, in any script, tokenizes to something. The cost of weirdness is length, not failure.
Why vocab sizes cluster at 32K–256K. A bigger vocabulary means shorter sequences (cheaper attention) but a bigger embedding table and softmax. The sweet spot has drifted upward with model size: Llama 2 used 32,000, Llama 3 jumped to 128,256, o200k sits near 200,000.
Images are tokens too. Multimodal models slice images into patch tokens — a photo costs ~85 tokens at low detail to ~1,600+ at high detail in OpenAI’s scheme. The context window doesn’t care what a token means, only that it counts.
Glitch tokens. Strings like SolidGoldMagikarp made it into GPT-2/3’s vocabulary from scraped data (a Reddit username) but barely appeared in training — prompting older models with them produced babble, evasion, or insults. A vocabulary entry with no learned meaning is a tiny haunted house.
The ugliest surviving part of the stack. Karpathy’s judgment stands: most LLM “weirdness” — arithmetic failures, spelling blindness, language inequity — traces back to tokenization. Byte-level and tokenizer-free architectures keep being proposed; the lookup table keeps winning on cost.
Your 1,000 tokens arrive at a server rack. What is physically waiting there? Not a program in any usual sense — a single enormous array of learned numbers, sharded across eight GPUs, resident in memory day and night. This chapter weighs the thing itself: how many numbers, how many bytes, why it fits on no single chip, and why it never, ever gets unloaded. The meter stays at zero one more chapter — numbers sitting still compute nothing.
A model is its : 405 billion learned numbers arranged into a few hundred giant matrices. Nothing else — no database, no rules, no code that “understands.” The count is the industry’s headline stat for a reason: GPT-2 (2019) had 1.5B, GPT-3 (2020) 175B, and by 2024 you could download 405B of open weights. For scale: print one parameter per millimeter and the line runs 405,000 km — past the Moon (384,400 km away). Every one of those dials was set by training; your request will disturb none of them — inference only reads.
Each parameter is a 16-bit number — 2 bytes — so the model is, physically, an 810 GB pile of floats. That pile is the knowledge: that Paris is in France, how Python indents, why sunsets redden — all of it is somehow in the matrices, and nobody can point to where. For contrast, the text of English Wikipedia compresses to ~24 GB. The weights are ~34 Wikipedias of bytes holding a lossy digest of far more. There is no file inside labeled facts.db; interpretability research is the ongoing attempt to find out where “Paris” lives.
The fastest memory money can buy — stacked directly on the GPU — tops out at 80 GB on an H100. The model is 810 GB. No single accelerator on Earth holds it, so serving begins with surgery: slices every matrix into 8 shards across 8 GPUs, which then must confer over NVLink at 900 GB/s on every layer of every token. Every frontier answer you have ever received was a group project — computed by a committee of chips voting in nanoseconds.
810 GB across eight 80-GB cards leaves no room for anything else — so production serving puts the model on a diet. stores each weight in fewer bits: FP8 halves every number to 1 byte — 810 → 405 GB — with accuracy loss measured in fractions of a benchmark point; 4-bit halves it again (203 GB) and is how a 405B-class model squeezes onto hobbyist rigs. Our node serves FP8: 405 GB of weights, 235 GB left over for the working memory the next chapters will spend. Precision, it turns out, is mostly padding.
Could the server load the model when your question arrives? 405 GB over a very good 10 GB/s disk array is 40+ seconds — over a typical network, minutes. Nobody waits that long, so weights are loaded once and stay resident in HBM around the clock, request or no request. That standby has a price: an 8×H100 node rents for roughly $24 an hour whether it’s answering questions or listening to silence — which is why idle GPUs are the industry’s quietest money leak, and why strangers together (chapter 04) is a moral imperative of the economics.
Total vs active: the MoE asterisk. Mixture-of-experts models route each token through a fraction of their weights: DeepSeek-V3 is 671B total but 37B active per token; Llama 4 Maverick 400B/17B. Param counts stopped being comparable across architectures — always ask which number you’re being told.
The rumor economy. Closed labs stopped publishing counts after GPT-3; GPT-4’s widely-reported ~1.8T MoE figure is a leak, not a paper. The open models are where the checkable numbers live — which is exactly why this tutorial rides one.
The VRAM napkin. Will it fit? Weights (params × bytes) + KV cache (chapter 03’s ½ MB/token × context × batch) + activations (small at inference). An 8×80 GB node at FP8 = 640 GB budget: 405 weights + 235 for everything else. Every deployment conversation is this arithmetic.
Why not just always 4-bit? Quality loss is real but small and task-dependent (long-tail knowledge and math degrade first). The industry default drifted FP16 → FP8 → FP4 with each hardware generation shipping native support — Blackwell does FP4 in silicon.
Before the big numbers, the smallest ones. This chapter drops to the floor of the ladder — the single operation, measured in femtoseconds and picojoules — and meets the physical law that governs everything above it: arithmetic is nearly free, and moving data to the arithmetic is not. The meter ticks exactly once, from 0 to 2. Everything else here is calibration you’ll spend in the next four chapters.
The atom of all of AI is the multiply-accumulate — : a × b + c, two floating-point operations. Every embedding, every attention score, every one of the 405 billion weights participates in the answer only through this one primitive, repeated ~10¹⁵ times. At an H100’s pace one MAC amortizes to ~2 femtoseconds — 2×10⁻¹⁵ s, an interval in which light itself travels 0.6 micrometers, less than the width of a bacterium. After two chapters at zero, the meter finally moves: Σ 2 FLOPs.
One H100 sustains up to 989 trillion of those operations per second in BF16 — 1,979 trillion in FP8 — from a 700-watt card you can lift with one hand. Calibration: the fastest computer on Earth in June 2000, ASCI White, peaked at 12.3 TFLOPS, weighed 106 tons, and cost $110M. At AI precision, one H100 out-computes it ~80-fold (an honest asterisk: ASCI White ran 64-bit science; the comparison is about scale, not substitution). Our node has eight of them — ~15.8 FP8 petaFLOPS on tap — and, as the next step shows, they will still spend most of their time waiting.
Here is the number that runs the industry, and almost nobody quotes it. Measured at silicon level (Horowitz, ISSCC 2014): a 32-bit floating-point multiply costs ~3.7 picojoules; fetching those same 32 bits from DRAM costs ~640 pJ — moving a number is ~170× more expensive than multiplying it. And compute has been improving faster than memory for decades, so the gap grows. This is the , and it inverts every intuition: the question is never “can the GPU do the math?” — it’s “can memory feed the math?” Remember 170:1; chapters 03–05 are consequences of it.
The counter-weapon is : DRAM stacked on top of the processor. One H100 reads its 80 GB at 3.35 TB/s — an entire Blu-ray collection per second. Now the division that will explain the next two chapters: our model is 405 GB of FP8 weights; reading it once through one GPU’s 3.35 TB/s takes 121 ms — spread across all 8 GPUs, ~15 ms. Fifteen milliseconds just to look at every weight once, before any arithmetic is even attempted. Hold that number; in chapter 04 it becomes the speed limit of every word the model says.
Whether hardware runs at full speed comes down to one ratio — : FLOPs performed per byte fetched. An H100 needs ~590 FLOPs per byte (1,979 TF ÷ 3.35 TB/s) to stay busy. Big matrix-×-matrix work — a full prompt at once, many users batched — clears that bar easily. But generating one token for one user is matrix-×-vector: each weight byte is used for ~2 FLOPs, then discarded — 0.3% of the bar. The most expensive chip in the building does arithmetic well under 1% of the time at batch size 1. It isn’t a math machine; it’s a memory pump with a math habit.
Above the H100. B200/GB200 (Blackwell): HBM3e, native FP4, and NVL72 racks that make 72 GPUs behave like one — the industry’s answer to the memory wall being “buy more wall.” TPUs take the systolic-array route: silicon shaped exactly like a matrix multiply.
The SRAM heretics. Groq and Cerebras skip HBM entirely, keeping weights in on-chip SRAM (orders of magnitude faster, and tiny). Result: 1,000+ tokens/s on models that fit — proof that decode speed is purely a memory story.
The interconnect ladder. Every request descends it: HBM 3.35 TB/s → NVLink 900 GB/s → InfiniBand ~50 GB/s → datacenter Ethernet. Each hop is roughly an order of magnitude slower — which is why sharding within a node is routine and sharding across nodes is a last resort.
MFU, the honesty metric. Datasheet FLOPS assume a perfect kernel with perfect data flow. Real workloads sustain a fraction: Meta’s tuned 405B training run held 38–43% MFU — and that’s considered excellent. When someone quotes peak TFLOPS, divide by two and a half.
The 1,000 embedded tokens meet the 405 billion weights. What happens in the 200 silent milliseconds before the first word of the answer appears? The meter finally earns its fourteen zeros — 810 trillion operations just to read the question — and yet this turns out to be the cheap, fast, hardware-friendly half of inference. This chapter is that gulp: the FLOPs, the parallelism, the user-facing silence it becomes, and the ½-megabyte notes it leaves behind per token.
The forward pass is not a lookup — it is a full tour. Each token’s vector is multiplied through essentially all 405 billion : ~2 FLOPs per parameter per token, or 810 GFLOPs per token. There is no shortcut, no index, no skipping to the relevant parts; the relevance is computed by the multiplication. Your 1,000-token prompt therefore costs 1,000 × 810 GFLOPs = 810,000,000,000,000 FLOPs just to be read. The meter, stuck at 2 for a whole chapter, jumps fourteen orders of magnitude in one step.
810 trillion operations sounds fatal — it takes ~130 ms. The trick: processes all 1,000 tokens simultaneously, as one giant matrix-×-matrix multiplication — exactly the high-arithmetic-intensity shape from the roofline step, thousands of FLOPs per byte, tensor cores saturated. 8.1×10¹⁴ FLOPs ÷ ~6.3 effective PFLOPS (the node at ~40% ) ≈ 129 ms. The model reads your entire prompt in one parallel gulp. This is the half of inference hardware loves — the other half, one chapter away, is the one it hates.
Prefill time surfaces in the product as — the “thinking…” silence between Enter and the first word. Our request: ~130 ms of prefill + queueing and network ≈ 200 ms, comfortably under the ~1-second threshold where a user’s flow of thought breaks (Nielsen’s 1993 numbers still hold). But TTFT scales with prompt length: paste a 100,000-token codebase and prefill alone is 2 × 405×10⁹ × 10⁵ = 8.1×10¹⁶ FLOPs ≈ 13 seconds of dead air. Every “why is it just sitting there?” bug report is this arithmetic, rediscovered.
Prefill leaves something behind. Attention requires every later token to consult every earlier one, and recomputing that would be quadratic suicide — so the model caches each token’s keys and values: the . For 405B: 2 × 126 layers × 8 KV-heads × 128 dims × 2 bytes = 516,096 bytes ≈ ½ MB per token. Your 1,000-token prompt just parked 0.5 GB of notes in HBM — and a full 128K context parks 66 GB, nearly a whole H100 of the eight. When long-context pricing looks steep, this is the real estate it’s paying for.
The window those notes live in has exploded: GPT-2 held 1,024 tokens (2019); GPT-3, 2,048; GPT-4 launched at 8K/32K (2023); Claude hit 100K then 200K; Gemini 1.5 reached 2M; Llama 4 Scout claims 10 million — a ~10,000× climb in six years. But capacity outran attention: Chroma’s 2025 “” study measured 18 frontier models degrading as input grows even on trivial tasks, long before the window fills. A window is an invitation, not a promise: the model holds 10M tokens the way you hold a phone book — technically, all of it; usefully, the page you’re on.
Where 2N bends. The 2-FLOPs-per-param rule ignores attention’s O(n²) term — negligible at 1,000 tokens, dominant past ~100K. Long-context prefill is quadratically worse than the napkin says, which is another reason long prompts bill extra.
FlashAttention. The memory wall strikes inside attention too: the naive n×n score matrix wouldn’t fit in fast memory. FlashAttention computes attention in tiles that never leave on-chip SRAM — an IO-aware algorithm; same FLOPs, far fewer bytes moved. The 170:1 lesson, applied.
GQA, already priced in. Full multi-head attention would store keys/values for all 128 query heads — 8 MB per token. Grouped-query attention shares 8 KV heads across them: 16× smaller. Our ½ MB figure is the discount already applied.
Prefix caching. If the first 988 tokens are the same for every user (they are — that’s the system prompt), prefill them once and keep the KV. This is exactly what providers sell as prompt caching at ~90% off — chapter 05 shows the price tag. Moral: keep your system prompt byte-stable.
Chunked prefill. A 100K-token prefill would monopolize the node for seconds, starving everyone mid-decode. Servers slice big prefills into chunks and interleave them with decode steps — one more scheduler trick between you and the datasheet.
The first token appears at t+200 ms. The remaining 299 take nine more seconds. You now have the tools to answer why: why is writing 40× slower than reading, when it’s the same weights and the same arithmetic? (Chapter 02’s bookmark — 15 ms to read the model once — is about to become the punchline.) Along the way: why streaming exists, why batching is the entire economics of inference, and the draft-and-grade trick that speeds decode without changing a single output token.
Generation is : the model predicts one token, appends it to the context, and runs again — because token №2 depends on token №1, there is nothing to parallelize inside your own answer. The 1,000-token prompt was read in one gulp; the 300-token answer will be 300 separate full passes through the model. This is the serial heart of the whole product experience: the typing effect in every chat window isn’t a UI flourish, it’s the physics of the loop, streamed to you at its native speed.
Here the bookmark from chapter 02 pays off. Each decode step multiplies one vector through all the weights — which means all 405 GB stream from through the cores for every single token. Chapter 02’s floor: ~15 ms per read at 26.8 TB/s. That is the speed limit — not FLOPs (a token’s 810 GFLOPs would take ~0.1 ms of pure compute), bandwidth. Real serving lands at 30–60 tok/s; our request runs at ~33 tok/s, 30 ms per token: 299 more tokens ≈ 9.1 seconds. Generation speed is a memory-bandwidth number, not a compute one.
Is 33 tokens/second slow? Humans read English at ~238 words per minute — about 5 tokens per second. The model writes 6× faster than you can read, which is why streaming works as UX: the text outruns your eyes, so generation feels instant even though it takes 9 seconds. The budgets flip for machine consumers: an agent chaining 7 calls (see the agent tutorial) waits on every one serially, and “reasoning” models burn thousands of decode tokens thinking before the first visible word — decode you pay for, at output prices, that no human ever reads.
The 405-GB-per-token read is decode’s shame — and its business model. The read serves one token for one user… but the same read can serve 64 tokens for 64 users at once: stack their vectors, multiply through the weights together, and per-user cost collapses ~64× until compute becomes the binding wall again. — admitting new requests mid-flight as others finish — is the single most important serving optimization in existence. It is why your quadrillion-FLOP answer will cost cents: you never rode alone.
One more trick bends the serial law. : a small, fast draft model guesses the next 4–8 tokens; the big model then verifies the whole guess in one parallel pass — checking is a prefill-shaped problem, so it’s cheap. Accepted tokens ship; the first wrong one is discarded and decoding resumes — provably identical output, 2–3× faster. At t+9.3 s our model emits <eos> — end of sequence, token 300. The meter rests: Σ 1,053,000,000,000,000 FLOPs. A quadrillion operations to explain a sunset. Now: the receipt.
Continuous batching (Orca, 2022). Static batching waited for the slowest request in the batch; continuous batching admits and retires requests every step. Throughput gains of 10–20× over naive serving came from scheduling alone — no model change.
PagedAttention / vLLM. KV caches used to be allocated as contiguous max-length blocks — mostly-empty reservations that wasted 60–80% of memory. vLLM manages KV like virtual-memory pages, cutting waste to <4% and lifting real-world throughput 2–4×. Operating-systems ideas keep winning in inference.
Disaggregated serving. Prefill wants maximum compute; decode wants maximum bandwidth per byte of KV. Big providers now run them on separate GPU pools and ship the KV cache between them — the two halves of your request may never touch the same silicon.
Why your local 405B is slower. Benchmarks quote batched throughput; alone at batch 1 you get the bandwidth floor and nothing amortized. Same math explains Groq/Cerebras headlines in reverse: SRAM bandwidth turns the same model into a 1,000+ tok/s sprinter.
The TTFT ↔ throughput dial. Bigger batches = more tokens/s served, worse individual latency. Every provider’s “speed” is a chosen point on that curve, and the SLO (time to first token vs tokens/s) is set by the business, not by the hardware.
The answer is on your screen. Somebody now has to pay for a quadrillion operations and three hundred re-reads of a 405 GB file — so why is the invoice less than a cent? This chapter reads the receipt line by line and finds the previous four chapters hiding in it: the memory wall priced as the output premium, the KV cache resold as a caching discount, batching as the reason any of it costs cents. The Σ meter stays frozen throughout: money is arithmetic about FLOPs, not FLOPs.
At representative frontier-class pricing — $3 per million input tokens, $15 per million output — the damage is: 1,000 in = $0.003; 300 out = $0.0045; total $0.0075. A quadrillion FLOPs, eight GPUs conferring for 9.3 seconds, 405 GB read three hundred times: three-quarters of one cent, retail. The meter freezes this chapter — money is arithmetic about FLOPs, not FLOPs — but stare at the ratio: ~1.4×10¹⁷ FLOPs per dollar. The deflation curve behind that number is the story of the decade, and it’s step 4.
Look closer at the menu and the whole tutorial reappears as pricing. Output costs 5× input ($15 vs $3): decode holds the machine serially (chapter 04) while prefill batches well (chapter 03) — the asymmetry is the roofline, invoiced. costs $0.30/M — 90% off: a re-sent system prompt re-uses stored KV (chapter 03’s notes) instead of re-prefilling. Long context bills extra past thresholds: that’s the 66-GB KV real estate. Every line item is a hardware fact wearing a dollar sign.
Zoom out from one model to the whole mid-2026 menu and the spread is comic: the cheapest usable API token (small open models) runs ~$0.10 per million output; the priciest frontier tier lists at $180 per million — a ~1,800× range on the same unit. No other commodity is priced like this. The engineering discipline it created is routing (the agent tutorial’s first move): classify the task, spend Opus-class tokens only where Haiku-class fails. Model choice is a bigger cost lever than any prompt optimization ever written.
Now the axis nobody believes until they see it dated. GPT-3-quality output (MMLU ≈ 42) cost $60 per million tokens in November 2021. By late 2024 the cheapest model matching that score cost $0.06 per million — a 1,000× collapse in 3 years, ~10× per year, faster than Moore’s law ever moved and faster than dotcom bandwidth deflated (a16z named it “LLMflation”). The twist: the frontier price barely moves — o1 launched at the same $60/M output that GPT-3 launched at. Capability at fixed price explodes; price at fixed capability collapses; the invoice line stays the same and only the model behind it changes.
The other bill. In August 2025 Google published the first serious production measurement: the median Gemini text prompt = 0.24 Wh of energy, 0.03 g CO₂e, and 0.26 mL of water — about five drops. 0.24 Wh is ~9 seconds of television. And the same report’s buried headline: that footprint fell 33× in one year — , , better silicon: chapters 01–04, again. Honest ledger: per-prompt is small; the aggregate — billions of prompts, and chapter 06’s training runs — is the number utilities plan around.
Cache writes cost extra. Before reads get 90% off, writing the cache bills a ~1.25× premium — caching is a bet that the prefix will be reused. Stable system prompts win the bet; timestamps in your system prompt lose it byte by byte.
Batch APIs: 50% off for patience. Submit jobs to run within 24 h and pay half — the provider schedules you into idle capacity. The discount is literally the shape of their load curve.
Reasoning tokens bill as output. The thousands of “thinking” tokens a reasoning model burns are decode at full output price, whether or not you ever display them. A hard question can silently cost 10–50× its visible answer.
$/token is the wrong unit for agents. A 7-call agent loop re-reads its whole history each call (see the agent tutorial’s 43,230-token ticket) — the honest unit is $/task, and it multiplies fast. Rate limits, not prices, are usually what actually gates production: tokens-per-minute quotas are the real scarcity.
The margin debate. Analyst estimates put inference pricing at roughly 5–10× the raw compute cost at scale — the profit pool that 10×/yr deflation keeps eating. When a price drops 80% overnight, it’s the curve arriving, not charity.
Your request is closed: answered, billed, forgotten by the server in milliseconds. But the 405 GB it flowed through — someone set every one of those dials, once, at colossal expense. This final numbered chapter opens the other ledger: the training run. One napkin formula, one staggering division, a finite library, a cluster where failure is weather, and the growth curve that dates every number in this tutorial. The Σ meter can’t contain it; it gets a second line.
Training cost has a napkin formula as clean as inference’s 2N: FLOPs ≈ 6 × N × D — six operations per parameter per training token (2 for the forward pass, 4 for the backward). Plug in Llama 3.1 405B: 6 × 405×10⁹ × 15.6×10¹² tokens = 3.79×10²⁵ — Meta’s published figure is 3.8×10²⁵, a miss of under 1%. One multiplication predicts a nine-figure industrial project. Every training-compute number in every AI-policy debate (the EU Act’s 10²⁵ threshold, the US 10²⁶ reporting line) is this napkin, regulated.
Divide the two meters this tutorial has been keeping: training 3.8×10²⁵ ÷ your request’s 1.05×10¹⁵ = ~36,000,000,000. The run that set the dials equals 36 billion of your sunset questions — roughly one for every five humans alive, asked 25 times each. This is the deepest economic fact in AI: capability is bought in one colossal prepayment, then amortized over inference forever after. It is why labs overtrain (next step), why weights are guarded like reactors, and why your ¾¢ receipt is, in accounting truth, a 36-billionth share of a datacenter’s season.
The D in the formula: 15.6 trillion training tokens — call it 12 million novels’ worth of text. The 2022 Chinchilla result said compute-optimal training wants ~20 tokens per parameter (405B → 8.1T); Meta fed it nearly double, on purpose — over-training wastes training compute to buy a smaller, cheaper-to-serve model, because inference is forever ( are advice, not law). The ceiling is in sight: Epoch AI estimates the total stock of quality public human text at ~300 trillion tokens, on track to be fully used by frontier training between 2026 and 2032 (median ~2028). The library is finite, and the industry is a fast reader.
The run itself, from Meta’s paper: 16,384 H100s for 54 days — 30.84 million GPU-hours, ~11.5 MW of GPUs before cooling, ~22 GWh, ~2,000 US-home-years of electricity for one model. The detail every infrastructure engineer frames on the wall: the cluster suffered 419 unexpected interruptions — one every three hours, for eight straight weeks — 78% of them hardware, GPUs and their the top culprits. And it still finished, at >90% effective training time, because all but three failures were healed by automation. At 16,384 GPUs, hardware failure isn’t an incident; it’s weather.
Last rung — the derivative. Frontier training compute has grown 4–5× per year for over a decade: AlexNet (2012) used ~5×10¹⁷ FLOPs; GPT-3 (2020) 3.1×10²³; our 405B (2024) 3.8×10²⁵; the largest known run by 2026 (Grok 4 class) ≈ 5×10²⁶. That is a billion-fold climb in fourteen years, doubling roughly every six months while transistors took two years. Costs run $10⁸ and rising; Epoch projects gigawatt-scale runs by 2030. Every absolute number on tonight’s card is a snapshot of this curve mid-flight — which is exactly why the card’s final section (Σ) insists on ratios, not constants.
Where 6ND comes from. Forward pass: 2 FLOPs per param per token (chapter 03’s rule). Backward pass: gradients flow through every weight twice — roughly 2× the forward cost. 2 + 4 = 6. What it excludes: data pipelines, failed and ablation runs, evals, and post-training.
Post-training is compute-cheap, talent-expensive. RLHF/DPO/fine-tuning add single-digit percentages to the FLOPs bill; the expensive inputs are human preference data and researcher time. The 10²⁵ is almost all pretraining.
The falling floor. While the frontier climbs, the floor collapses: GPT-2 cost ~$43K to train in 2019; Karpathy’s llm.c reproduced it in 2024 for about $20 of spot H100 time. Yesterday’s frontier is today’s weekend project — the same curve, read from below.
Dodging the data wall. The ~2028 exhaustion date assumes human text only. The hedges: synthetic data (models teaching models), multimodal data (video is vast), and better data efficiency per token. Chinchilla optimality already bends when serving cost enters the objective — expect the 20:1 rule to keep bending.
The next binding constraint: electrons. Epoch projects frontier runs needing gigawatt-scale power by ~2030 — a large nuclear plant’s output for one training job. The bottleneck migrates: data (2028) → power (2030) → whatever survives contact with both.
The payoff the title promised — the heir to 2012’s “latency numbers every programmer should know,” with every row earned somewhere above (the right margin says where). One request descended it: femtoseconds to seasons, two FLOPs to 10²⁵, one question to the whole library.