8 TOKENS · 16,384 LATENT NUMBERS · 40 DENOISER PASSES · 786,432 PIXELS
Eight chapters and a receipt: the seed, the prompt, the first guess, the steering, the loop, the picture, the school that taught it, and what four years changed — one 512×512 image ridden end to end, every number on the way earned and checked. The scenes play themselves — watch, or just scroll. are tappable. You typed the prompt.
A 512×512 photograph is 786,432 numbers, and generating one means choosing every single one of them. Before that can start, only two things exist: a grid of random numbers and a sentence. This tutorial follows one image — six words, seed 3,461,922, twenty steps — from that grid to a PNG. It carries a running meter: Σ denoiser passes, the number of times the network at the centre of all this has been run. For the next two chapters the meter reads zero, because nothing here is a computation.
A 512×512 photograph is 262,144 pixels, and each pixel is three numbers between 0 and 255: 786,432 values, 768 KB before any compression. Generating an image means choosing every one of them. Put another way, every possible 512×512 picture is one point in a 786,432-dimensional space — and the overwhelming majority of those points are static. The whole problem is landing on one of the rare points that looks like something.
No diffusion model works in that space. Before anything else runs, a compresses the image 8× on each side into a 64×64 grid with 4 channels: 16,384 numbers, exactly 48× fewer. Every denoising step in this tutorial happens in that small grid, and pixels come back only at the very end. This is the in latent diffusion (Rombach et al., December 2021), and it is the single change that moved image generation off a research cluster and onto a desktop GPU.
The — here 3461922 — initializes a pseudo-random generator, which draws 16,384 samples from a with mean 0 and standard deviation 1. That grid is the starting picture. Same seed, same prompt, same sampler, same step count gives the same image bit for bit, which is the only reason any of this is debuggable. Change 20 steps to 21 and you get a different picture, not a better one.
The latent has 4 channels, but they are not red, green, blue and alpha. They are whatever the autoencoder found worth keeping, and nobody assigned them meanings. A second surprise sits underneath. The random draw is not portable: PyTorch’s CPU generator and its CUDA generator produce different numbers from the same seed, so the same seed and prompt give different images depending on where the noise was drawn. Reproducing an image needs the backend, not just the integer.
Why 8× and not 16×. The compression factor is a trade: fewer latent numbers means cheaper denoising, but reconstruction error grows faster than the saving does. Factor 8 with 4 channels was where the original paper landed after trying 4, 8, 16 and 32 — and every Stable Diffusion release through SDXL kept it.
The constant 0.18215. Stable Diffusion multiplies the encoder’s output by this number so the latent’s variance sits near 1, which is what the noise schedule assumes. It was measured on the training set, not derived, and it is hard-coded in every implementation.
Pixel-space diffusion still exists. Imagen and DALL·E 2 ran the loop directly on pixels at low resolution and then bolted on a cascade of super-resolution models. It works, and it costs enough compute that neither shipped as a file you could download.
A latent is not an image. Decoding a latent halfway through the loop is a debugging convenience, not a defined operation. The live previews in web interfaces use a cheap approximate decoder — a few thousand parameters instead of tens of millions — which is why the preview and the final image often disagree.
Six words go in; 59,136 numbers come out, and the denoiser will read the same 59,136 numbers forty times without recomputing them once. The conversion is done by a separate network that runs before the loop starts — and it runs twice, for reasons that will not be clear until chapter 03. Σ holds at zero denoiser passes throughout, because none of this touches the denoiser.
A byte-pair tokenizer with a 49,408-entry vocabulary splits a red fox in falling snow into six word , then adds a start marker and an end marker: eight in all. The buffer is not eight long. It is exactly 77 slots, so 69 padding tokens follow. Prompts longer than 77 tokens were truncated with no error for years — the words past the limit never reached the model, and nothing in the interface said so.
’s text encoder — 123 million parameters, the text half of a model trained to match pictures with captions — turns those 77 slots into 77 vectors of 768 numbers each. That is 59,136 numbers, and it is computed exactly once. The denoiser reads the same block on all 40 of its passes. This is why prompt length costs almost nothing at generation time and step count costs almost everything.
CLIP’s text encoder is causal, so each token’s vector carries everything to its left. That is why word order and comma placement change the picture even when the words are identical. The 69 padding slots are not skipped either: they get vectors too, and the denoiser attends to all 77 of them. Past this point nothing downstream ever sees a word again — only numbers, in a fixed-size block.
CLIP now runs a second time, on the empty string, producing another 77 × 768 block. Nobody asked for it. The guidance step in chapter 03 requires it, and the pipeline computes it whether or not you know it is there. Typing a does not add a mechanism: it replaces the empty string in a slot that classifier-free guidance already had to fill, which is why a negative prompt costs nothing extra.
Why CLIP and not a language model. CLIP’s text tower was trained to sit in the same space as images, so its output vectors are already organized around what pictures look like. Later models hedged: SD 2.x swapped to OpenCLIP at 1024 dimensions, SDXL concatenated two encoders (768 + 1280 = 2048), and SD3 added T5-XXL on top for prompts that read like sentences.
Prompt weighting is not in any paper. The (fox:1.3) syntax is client-side scaling of the vectors for those tokens, invented by the AUTOMATIC1111 community and adopted afterwards by everyone else. The model has no idea it happened.
Long prompts. Chunking a 200-token prompt into three 77-token windows and concatenating the resulting blocks is a client convention, not a model capability. The denoiser is happy to attend over 231 vectors; nothing in training ever showed it more than 77.
Why asking for spelled-out text fails. The tokenizer hands CLIP learned chunks, not letters, so a request for a sign reading a particular word arrives as chunks with no letter-level structure. It is the same mechanism that makes language models miscount letters, and it is one of two reasons text in generated images comes out wrong — the other one is in chapter 05.
The denoiser takes 16,384 numbers and returns 16,384 numbers, and the numbers it returns are not a picture. Five steps here: the interface, what the output actually is, the training objective that explains why, the shape of the network, and what one pass already knows. Σ moves for the first time in this tutorial — to 1 denoiser pass.
The is called with three things: the 16,384-number latent, the integer timestep 999, and the 59,136 text numbers from chapter 01. It returns 16,384 numbers — exactly the shape it was handed. That is the entire interface of the , and everything the pipeline does afterwards is arithmetic on those numbers. Σ ticks from 0 to 1.
Those 16,384 output numbers are an estimate of the noise currently sitting in the latent. Not the clean image, not a correction, not a delta — the noise. Across all 40 passes the network is never once asked for a picture. This is , and it is the fact that makes everything downstream legible: if the output were an image, both guidance and sampling would be nonsense.
Training runs the : take a real photo, pick a timestep between 1 and 1,000, draw noise, and mix the two by a fixed rule from the . That rule is closed form, so reaching timestep 900 costs one operation rather than 900. Then ask the network for the noise and score it with mean squared error. The noise the trainer drew is the answer key, and it costs nothing to produce.
The network narrows the 64×64 grid to 32, then 16, then 8, and widens it back — with skip connections carrying fine detail straight across the middle, which is why edges survive the trip through the bottleneck. The convolutions doing the work inside are the same machinery a digit classifier uses. Counted up: 860 million parameters for the U-Net, 123 million for the text encoder, about 83 million for the autoencoder, shipped as one 4.27 GB file.
Subtract the whole predicted noise from the latent in one go and you get : the model’s guess at the finished image after a single pass. At timestep 999 that guess is a soft two-tone smear with the composition roughly in place and no detail anywhere. The layout is already decided. The other 39 passes are about everything except the layout.
Three ways to write the same target. ε-prediction asks for the noise, x₀-prediction asks for the clean image, and v-prediction asks for a mixture of the two that stays well-scaled at every noise level. They are algebraically interconvertible; v-prediction behaves best at high noise, which is why SD 2.1-v and almost every distilled model uses it.
Why mean squared error does not blur this. An averaging loss usually produces blurry images, because the average of all plausible completions is mush. Here the target is not an average: it is one specific noise draw the trainer made, so there is a single right answer and no reason to hedge toward the middle.
The 2015 paper nobody read. Sohl-Dickstein and colleagues described the forward and reverse processes in full five years before DDPM. The framing was right and the samples were poor; what changed in 2020 was the parameterization and the compute, not the idea.
What the many passes bought. A GAN generates in one pass and diffusion needs forty, which looks like a straight loss. The trade was training stability: GAN training collapses in ways that are hard to diagnose, while this objective is a regression that converges. Chapter 07 is the story of buying the single pass back.
Nothing so far explains how the word “fox” reaches a specific corner of a 64×64 grid. The answer is 16 attention layers, a number telling the network which noise level it is looking at, and a subtraction between two predictions. That second prediction means a second call: Σ reaches 2 denoiser passes, and the image is one step into twenty.
Text never enters the denoiser as text. Inside the U-Net sit 16 layers — 6 on the way down, 1 in the middle, 9 on the way up. The queries come from image positions; the keys and values come from the 77 text vectors. Mechanically this is the attention from the transformer tutorial with the two sides drawn from different places, and that is the only difference.
At the top resolution the latent is 64 × 64 = 4,096 positions, each attending over 77 text vectors: 315,392 query–key pairs in one layer alone. The resulting for a single word is not an abstraction. Plot the map for fox and it lights up the region the fox ends up occupying. Editing tools like prompt-to-prompt work by reading those maps and writing new ones back.
The timestep is not metadata. It is turned into a , pushed through a small network, and added into every residual block — so one set of weights behaves like 1,000 different denoisers, one per noise level. Hand the same latent in at t = 999 and at t = 100 and the two outputs have nothing in common, because the model is being asked two different questions.
The same 860-million-parameter network is now called a second time, on the same latent at the same timestep, with the empty-string block from chapter 01 in place of the prompt. Two predictions exist: one made knowing what you asked for, one made knowing nothing. Σ reaches 2 — and this is one step out of twenty. Everything about generation costs double from here.
The two predictions combine by : take the unprompted one, measure how far the prompted one moved away from it, and go 7.5 times that far. It is not a blend between them; the result sits well past the prompted prediction. Set the scale to 1 and the prompt barely registers. Push it to 20 and colours blow out, because you are extrapolating far outside anything the network was trained on.
Guidance used to need a classifier. Dhariwal and Nichol got the same steering in 2021 by training a separate image classifier on noisy images and pushing the sample toward a class label. Ho and Salimans removed it in 2022 by training the diffusion model with its condition randomly dropped about 10% of the time, so one network answers both questions.
Why high guidance oversaturates. Extrapolating past the prompted prediction pushes latent values outside the range the decoder was trained on, and the decoder maps out-of-range values to blown highlights and hard edges. Dynamic thresholding was invented to pull them back in, at the cost of some contrast.
The guidance-rescale fix. The same 2023 paper that named the schedule defect in chapter 04 also showed that high guidance shifts the sample’s overall brightness, and proposed rescaling the guided prediction to match the unguided one’s statistics.
The 16 layers are a control surface. ControlNet, regional prompting and prompt-to-prompt are all interventions on the same cross-attention layers — replacing the maps, masking them by region, or adding a second conditioning signal alongside the text. None of them retrains the denoiser.
The sampler now holds a prediction of all the noise in the latent, and it removes about a twentieth of it. Then it asks again. Five steps here: why the subtraction is partial, why 20 visits stand in for 1,000 noise levels, how early the picture is settled, a defect in the schedule that shipped in every checkpoint, and the arithmetic that takes Σ to 40 denoiser passes.
The takes the predicted noise and removes only the amount the schedule assigns to one step, moving the latent from noise level 999 to 949. Removing all of it in one go is legal, and it gives you the smear from chapter 02. The loop exists because a small move, made twenty times with a fresh prediction each time, lands somewhere a single confident move cannot reach.
Training used 1,000 timesteps. Sampling visits 20 of them — 999, 949, 899, and so on — and skips the other 980 outright. showed in 2020 that the skipping is allowed, by making the reverse process deterministic. DPM-Solver++ went further in 2022, treating the reverse process as an ordinary differential equation and getting good samples in 15 to 20 evaluations where the earlier method wanted 100 to 250.
High noise levels carry only coarse information, so the first few steps decide the layout — where the mass sits, where the ground is — and the last steps add texture on top of decisions already made. Stop the loop at step 12 of 20 and you get the same picture, slightly soft. Stop at step 3 and you get a different picture. This is also why raising the step count changes an image instead of refining it.
Stable Diffusion’s beta schedule runs from 0.00085 to 0.012 over 1,000 steps and never quite reaches . At t = 999 the training data still leaks its average brightness, so the model never learned to start from the pure noise that generation actually starts from. The visible result: SD 1.5 cannot produce a genuinely black or genuinely white image. Everything comes out mid-grey, and nobody published the fix until 2023.
Twenty steps at two passes each is 40 evaluations of an 860-million-parameter network, and that is essentially the entire cost of the image: the text encoder ran twice, the decoder will run once, everything else is arithmetic. On a consumer GPU the loop lands somewhere around a second, but treat that number as hardware rather than architecture — the 40 is the part that transfers. Implementations hide the doubling by stacking the prompted and empty-string inputs into a single batch of two, so a profiler shows 20 calls rather than 40.
Ancestral samplers never settle. Names ending in a — Euler a, DPM++ 2S a — add fresh noise back after each step, so raising the step count keeps producing a different image rather than converging on one. Euler and DPM++ 2M do converge. If your image keeps changing as you add steps, that is the sampler, not the model.
Karras sigmas. Which 20 of the 1,000 levels you visit matters as much as how many. Spacing them by the Karras schedule spends more steps in the middle range, where the image is actually being decided, and fewer at the ends where little changes.
Why more steps stops helping. Past roughly 30 steps the solver error is already below the model’s own prediction error, so the extra passes buy nothing measurable. The number 50 in early documentation came from a slower solver, and it outlived the solver by about two years.
img2img is this loop, entered late. Encode a real photo, noise it to timestep 600 instead of 999, and run the remaining steps. The strength slider is exactly the choice of which timestep to enter at, which is why high strength discards the original and low strength barely moves it.
The last network in the pipeline has never seen your prompt and has no idea what a fox is. Four steps: the single decode, the loss it cannot undo, why small structures come out wrong, and the trip to a file. Σ holds at 40 denoiser passes throughout — the decoder is a different network, and it runs once.
The autoencoder’s decoder — about 50 million parameters — expands the 64×64×4 latent to 512×512×3 in a single forward pass. No iteration, no text input, no timestep. It is the mirror of the encoder that defined the latent back in chapter 00, and both halves were trained before any diffusion happened. Whatever the loop produced, this network is what turns it into something a screen can display.
The compression is lossy, and the loss is measurable without generating anything at all. Encode a real photograph, decode it straight back with no diffusion in between, and it returns slightly wrong: softened text, smeared fine repeating patterns, altered eyes. That round trip is the ceiling on every image the model can make, because no amount of denoising recovers detail the decoder has no way to represent.
Small structures get very few latent cells. A letter on a sign in the scene lands on two or three cells of the 64×64 grid; a hand in the middle distance gets a handful. There is no room there for five distinguishable fingers, or for the difference between an E and an F, so the decoder produces something plausibly shaped instead. SD3 raised the latent from 4 channels to 16 largely for this: more numbers per cell, more small structure survives.
The decoder’s floating-point output is clamped to 0–255, which gives back the 786,432 numbers from chapter 00, and PNG compresses them to a few hundred kilobytes. The latent is discarded and cannot be recovered from the file. One more thing happened on the way out: the reference pipeline ran every sample through an invisible-watermark encoder, so the pixels people published were never quite the pixels the decoder produced.
Why the autoencoder is variational. The extra term in its training objective keeps the latent space smooth, so nearby latents decode to nearby images. Without it the space would be full of holes, and a denoiser walking through it would keep landing on points that decode to nothing.
Swapping the decoder. Community checkpoints like vae-ft-mse-840000 are the same latent space with a decoder fine-tuned for longer. Dropping one in changes faces and fine detail noticeably and changes composition not at all — which is a good demonstration of how little the decoder is responsible for.
Tiled decoding. Decoding a large image at once needs memory proportional to the output, so implementations decode in overlapping tiles. Where the tiles meet, the seams show up as faint grid lines in flat areas like skies.
Why hires-fix runs the loop twice. Generating directly at 1024×1024 with a model trained at 512 produces duplicated subjects, because the composition decisions from chapter 04 get made twice over. The fix is to generate small, upscale the latent, and run the loop again at partial strength.
About $600,000 of rented A100 time separates a network that outputs noise from one that outputs the right noise. Five steps: the corpus, the bill, the number of times chapter 02’s single operation ran during training, what the model kept verbatim, and the second model that ties words to pixels at all. Your image spent 40 denoiser passes; the training run spent 1.22 billion of the identical operation.
The corpus is , 5.85 billion image–text pairs scraped from Common Crawl, of which the English subset holds 2.32 billion. SD 1.5 was fine-tuned on a further slice, filtered by a small model that scored images for how good they looked. The captions are HTML alt text — written by web authors for accessibility and search engines, never for a model.
256 A100 GPUs, roughly 150,000 GPU-hours, about $600,000 at 2022 rental prices. That figure was published at the time as evidence the cost was falling, and that is the point of quoting it: it bought a 4.27 GB file anyone could download and run on their own machine. Set against it, your 40 denoiser passes cost a fraction of a cent.
Training and generation run the identical forward pass. The only difference is that training knows the answer and generation does not. SD 1.5 ran 595,000 optimizer steps at batch size 2,048, which works out to 1,218,560,000 noisy images with one noise guess apiece. Your picture used 40 of the same operation, and the ratio between those two numbers is the whole economics of the field.
Carlini and colleagues took the 350,000 most-duplicated captions in the training data and generated 500 images for each — 175 million images — then looked for near-identical clusters. They recovered 94 near-verbatim training photographs under a strict criterion, and 109 with manual inspection. tracked duplication: the pictures that came back were the ones that appeared many times in the corpus.
No label in the diffusion training data said “fox.” The link between that word and those pixels comes from CLIP, trained earlier and separately on 400 million image–caption pairs to place matching images and captions near each other. Diffusion training only had to learn to denoise while being shown CLIP’s vectors. Two models, two corpora — and the alignment between them is the part everyone credits to the denoiser.
The aesthetic filter is why it has a look. A small linear model on top of CLIP embeddings, trained on a few thousand human ratings, decided which images survived into the final fine-tuning slice. Every stylistic tendency people recognize as “the Stable Diffusion look” traces back to those ratings.
The corpus is no longer downloadable as it was. LAION-5B was taken down in December 2023 after illegal material was found in it, and republished the following year with the flagged content removed. Later models were trained on filtered successors, which is one reason their outputs differ in ways no architecture diagram explains.
SD 1.5 is not a fresh run. It resumed from an earlier checkpoint rather than starting from random weights, so the lineage matters: the 595,000 steps in this chapter sit on top of earlier ones. Reproducing any published number means reproducing the whole chain.
Deduplication is the mitigation. Since memorization tracks duplication, removing near-duplicate images from the corpus removes most of it. It also removes real training data, which is why the fix took a while to be adopted and why published extraction rates fell rather than reaching zero.
Stable Diffusion 1.5 was released in October 2022, and by the end of 2023 the 40 passes you just watched had been compressed into 1. Five steps on which parts of the preceding six chapters are already historical: the backbone, the path, the step count, the doubling, and the four things none of it touched. The meter runs backwards here — Σ 40 denoiser passes down to Σ 1.
The U shape from chapter 02 lasted about two years. SD3 and Flux.1 both replaced it with a , where image patches and text tokens sit in one sequence and attend to each other in both directions — instead of text entering through 16 side layers while the image never talks back. Flux.1-dev’s backbone is 12 billion parameters against SD 1.5’s 860 million.
The route from noise to image under the original formulation is curved, and a curved path needs many small steps to follow accurately. trains the model on straight lines between noise and data instead, and asks it for a velocity along that line rather than the noise in the latent. Fewer steps for the same accuracy, because a straight line is easier to follow. SD3 and Flux both use it.
trains a student network to reproduce in one step what the 40-pass model produces in forty. Latent consistency models got it down to four to eight steps in late 2023. Adversarial diffusion distillation, shipped as SDXL-Turbo weeks later, got it to one, using the original model as a frozen teacher alongside a discriminator. Σ drops from 40 to 1, and the thing that made it possible is the slow model.
Guidance distillation removes the doubling from chapter 03 outright: train the model so its single output already behaves like the extrapolated combination, and take the guidance scale as an ordinary input instead. Flux.1-dev is built this way. It is also why its guidance number behaves differently from SD 1.5’s, and why comparing the two settings directly means nothing.
The network changed, the path changed, the step count changed by a factor of forty. The shape of the loop did not: encode the text once, start from Gaussian noise in a latent, run a denoiser repeatedly, decode once. Video models are the same recipe with a time axis added to the latent, which is why everything in this tutorial still describes them — at higher resolution, and for a great deal longer.
Step counts stopped being the headline. At one denoiser pass the bottleneck moves to the text encoder and the decoder, which used to be rounding errors next to a 40-pass loop. Flux ships a 4.7-billion-parameter text encoder alongside its backbone; encoding the prompt is no longer free.
Diffusion is not about images. The loop only cares about the shape of the data, so the same recipe now generates robot action sequences, protein structures and audio spectrograms. Nothing in chapters 02 to 04 mentions pixels except by way of the latent.
Pixel space is being reconsidered. The ceiling from chapter 05 is a property of the autoencoder, not of diffusion, and as backbones get better that ceiling looks more like the binding constraint. Several 2025 systems went back to running at least part of the loop on pixels.
What to re-check before trusting any of this. Parameter counts, prices and step counts all rot within a year. The ratios do not: the latent’s compression factor, one text encode against N denoiser passes, one decode at the end, and training passes outnumbering generation passes by a factor of tens of millions.
One prompt, one seed, one 512×512 PNG — with every row earned somewhere above, and the right margin saying where. Read the two closing lines together: they are the same operation, counted once for your picture and once for the run that made the picture possible.