ONE FROZEN MODEL · 0.052% TRAINED · 8.4 MB · ZERO ADDED LATENCY
One finished model, one new skill, and not a single base weight touched. Follow one 4096×4096 weight matrix from a 16.7-million-number update down to a rank-8 detour. Then watch the whole adapter — 0.052% of the model — train on a gaming GPU and fold back in for zero-latency serving. The scenes play themselves — watch, or just scroll. are tappable. The base model never changes.
To teach a finished model one new skill, the old way clones all eight billion weights and retrains them. Before LoRA makes its move, weigh the incumbent: what full fine-tuning actually costs — in memory, in dollars, in near-identical copies — and why almost none of that cost is the model itself. The Σ counter stays at zero all chapter: LoRA hasn’t wired in a single trainable number yet.
Llama-3-8B is 8,006,464,512 numbers — 32 layers, hidden width 4096 — frozen after a pretraining run that cost millions. Full unfreezes every one of them to nudge behavior. Our goal is small: make it answer in one company’s support voice. Full fine-tuning treats “learn a small new habit” and “re-open all eight billion dials” as the same operation.
Training memory isn’t just the weights. keeps, per parameter, a 32-bit master copy plus two momentum buffers — the rule of thumb is ~16 bytes per parameter once gradients and mixed-precision states are counted. For 8B that’s ~128 GB, and the weights themselves are only an eighth of it. A full fine-tune of an 8B model doesn’t fit on a single 80 GB datacenter GPU.
Full fine-tuning also produces a full-size artifact — a fresh 16 GB checkpoint — for every task. Ten customers, ten voices: ten separate eight-billion-weight models, 160 GB, ten things to load and serve. The base they share is 99%+ identical; you’ve paid to store the same frozen ocean ten times over to capture ten different teaspoons.
Something is off. It is implausible that teaching “sound like our support team” requires rewriting all eight billion dials. What if the change a fine-tune actually makes is tiny — not in effort, but in shape — so we could train and store just that change and leave the alone? That is the entire bet of Low-Rank Adaptation: freeze the base, learn a small patch.
What full fine-tuning is still for. When you have the GPUs and the data, full fine-tuning remains the frontier move: large domain shifts, new languages, continued pretraining. The catch that stalks it is catastrophic forgetting — moving every weight risks overwriting abilities you never meant to touch, which a frozen base structurally cannot lose. For what the base model itself cost to make, see the AI-numbers tutorial’s training-debt chapter.
The MLOps tax. A separate 16 GB checkpoint per task isn’t just storage — it’s ten models to version, ten to redeploy after every base upgrade, ten resident copies if customers are served concurrently. The economics of one-copy-per-skill are what pushed serving stacks toward a shared frozen base long before the math of LoRA made it cheap.
The family LoRA belongs to. — parameter-efficient fine-tuning — is the umbrella: adapter modules bolted between layers (2019), prefix-tuning that trains fake context tokens, BitFit that trains only biases, (IA)³ that trains per-channel rescales. LoRA is the member that won serving, and Hugging Face’s peft library is where the whole family lives in practice.
Before splitting anything, why believe the update is small in shape? The evidence predates LoRA — and it’s startling: whole models steered through a few hundred directions, and the steering getting easier as models grow. This chapter is the hunch that made the method inevitable. The counter holds at zero — nothing is built yet, only justified.
In 2020 — a year before LoRA — Aghajanyan et al. measured a model’s : the fewest independent directions you must move to fine-tune it. The result that lit the fuse: optimizing just 200 parameters, randomly projected into RoBERTa’s full weight space, recovers 90% of full fine-tuning performance on the MRPC task. Ninety percent of the skill lived in a 200-dimensional room inside a 355-million-parameter house.
The counterintuitive twist from the same work: the larger the pretrained model, the lower its intrinsic dimension. Scale doesn’t just add capacity; it packs skills into ever-smaller adaptable subspaces. So the eight-billion-weight model isn’t harder to steer than a small one — it’s easier, because the direction you need to push already almost exists inside it.
LoRA turns the observation into a design. Its hypothesis: the weight update accumulated during adaptation has a low — it can be written as the product of two much smaller matrices with little loss. Not the weights are low-rank (they’re not); the change to them is. That one sentence is the whole method. Everything after it is bookkeeping.
A full 4096×4096 update matrix has up to 4096 independent directions and 16.7 million free numbers. If the real change only needs, say, 8 directions, you are storing 16.7 million numbers to express something that lives in 8 — a thousand-fold overpayment. Rank is the number of directions; LoRA’s claim is that a handful is enough. The next chapter cashes the claim into two skinny matrices.
Rank, in pictures. A matrix’s rank is how many of its rows (or columns) are genuinely independent — how many directions it can actually move an input in. A rank-8 4096×4096 matrix looks huge but only ever pushes activations around inside an 8-dimensional subspace.
The view. Singular value decomposition sorts a matrix’s directions by importance. Plot the singular values of a fine-tuning update and you see a few tall bars and a long near-zero tail — the picture behind the whole low-rank bet: most of a ΔW’s 4096 possible directions are essentially empty.
Why B·A covers every low-rank matrix. Any matrix of rank ≤ r can be written exactly as a product of a 4096×r and an r×4096 matrix — that’s what rank means. So parameterizing the update as B·A isn’t an approximation scheme; it’s the exact family of all rank-≤r updates, with the full-rank ones excluded on purpose.
The distinction that matters. The pretrained weights themselves are not low-rank — they carry the model’s whole distributed knowledge and use their directions well. It’s the change a fine-tune makes that collapses to a few directions. LoRA compresses ΔW, never W.
Paid off in chapter 05. The LoRA paper later checked the hunch quantitatively: the top singular directions of an r=64 adapter and an r=8 adapter nearly coincide — even r=1–2 captures the dominant directions. The dial chapter turns that measurement into a design rule.
If the update needs only r directions, you never build the full ΔW at all — you build the two skinny matrices whose product is ΔW, and count the numbers you just saved. This chapter zooms to a single protagonist matrix, splits its 16.7-million-number update into a 65,536-number pair, and checks where the 256× discount comes from — and why it grows as models get wider.
Zoom to one weight matrix: the query projection in layer 1, W_q — a 4096×4096 grid, 16,777,216 numbers, one of hundreds of weight matrices in the model. Full fine-tuning would hand this one matrix its own 16.7-million-number update, then do the same for every other matrix in the stack. LoRA is about to express that same update in 65,536.
The : ΔW = B·A, where B is 4096×r (tall and thin) and A is r×4096 (short and wide), meeting at a waist of width r. For r=8, A holds 8×4096 = 32,768 numbers, B holds 4096×8 = 32,768 — together 65,536. Their product is a full 4096×4096 matrix of rank at most 8. You store the two skinny factors; the fat matrix is only ever implied.
Do the division: 16,777,216 ÷ 65,536 = 256. One matrix’s update just got 256 times cheaper to store and train, with no approximation beyond “rank ≤ 8.” The reduction is d / 2r = 4096 / 16 — widen the model (larger d) and the savings grow. The bigger the base, the better LoRA’s deal, which is exactly the direction models keep heading. (For GPT-3 175B, the paper’s r=4 adapter shrank the checkpoint 10,000×, 350 GB → 35 MB.)
Isn’t a rank-8 matrix a crippled version of a rank-4096 one? For an arbitrary matrix, yes. For a fine-tuning update, chapter 01 said no: the change occupies a few directions, so the 4,088 directions you drop were near-empty to begin with. B·A doesn’t approximate the update you wanted — it is the shape the update was always going to take. Now wire it into the network so it can learn.
Best rank-r, guaranteed. The sorts any matrix’s directions by importance, and the Eckart–Young theorem says truncating to the top r of them is the best rank-r approximation you can make. The other face of the same coin: any matrix of rank ≤ r factors exactly as B·A with inner width r — so LoRA’s parameterization loses nothing over “learn a rank-r update” stated abstractly.
The general count. For a d×k matrix the adapter costs r×(d+k) numbers against d×k for the full update — 8×(4096+4096) = 65,536 versus 16,777,216 here. The discount d·k / r(d+k) grows with width, which is why LoRA’s bargain keeps improving as models get larger.
The GQA wrinkle. gives Llama-3-8B 32 query heads but only 8 key/value heads, so K and V projections are 4096×1024 — their adapters would cost 8×(4096+1024) = 40,960, and the tidy 256× becomes a messier 328×. This run adapts the two genuinely square projections — query and output — for clean arithmetic; the mechanism is identical either way.
Spread beats depth. Given a fixed parameter budget, the LoRA paper found that adapting more matrices at low rank beats adapting few matrices at high rank — rank 2 on four projection types outperformed rank 8 concentrated on one. Coverage of the network matters more than capacity per matrix.
The two skinny matrices exist — on paper. Now they must attach to a live, frozen network without wrecking it on the first step: a brand-new, randomly initialized bolt-on that somehow leaves the model’s behavior exactly untouched, then learns while data flows through the old path and the new one at once. The trick is an initialization so careful that the adapter begins life as a mathematical no-op.
The initialization is the clever bit. A gets small random Gaussian values; B is set to exactly zero. So at step zero, B·A = 0 and ΔW = 0 — the brand-new contributes nothing, and the model behaves exactly like the untouched base. You bolt on the detour and the output doesn’t move a hair. Training starts from the model you already trust, not from a random perturbation of it.
The forward pass forks. The input x runs down the frozen highway W₀·x exactly as before, and simultaneously down a bypass: first A squeezes x through the width-r waist, then B expands it back, giving (α/r)·B·A·x. The two rejoin by addition: h = W₀x + (α/r)·BAx. The highway carries the model’s existing knowledge untouched; the ramp carries only the new skill.
Real LoRA doesn’t patch one matrix; it patches a chosen set. Add a second detour on the same layer’s output projection W_o — another 4096×4096, another 65,536-number adapter — and the layer now has two low-rank ramps, 131,072 trainable numbers, its 33 million frozen weights still untouched. Attention has four projection matrices (Q, K, V, O); which of them you adapt is a knob, and adapting a couple is the cheap, common default.
What is that α/r factor? is a fixed constant that scales the detour’s contribution; dividing by r keeps the scale steady as you change rank, so you can retune r without re-tuning everything else. The LoRA authors set α to the first r they tried and never touched it again, noting it behaves almost exactly like a learning-rate scale. It’s the adapter’s volume knob: how loudly the new skill speaks over the base.
Why B = 0 and not A = 0. Zeroing either matrix makes B·A = 0, so the no-op start works both ways — B = 0 is the paper’s convention. With A holding real Gaussian values, B receives a nonzero gradient on the very first step, so the detour starts learning immediately; zero both and every gradient into the adapter vanishes with them.
Dropout on the ramp. Practical LoRA adds a small dropout on the detour’s input (the lora_dropout knob, typically 0.05–0.1). With so few trainable parameters carrying the whole skill, a little noise on the bypass is cheap insurance against memorizing the fine-tuning set.
Parallel is the load-bearing choice. The detour sits beside the frozen matrix and joins by addition — not in sequence after it. That placement is exactly what makes chapter 06 possible: a parallel branch can be folded into W₀ as one matrix, while a sequential module is stuck in the forward path forever, which is the latency tax the 2019 adapters paid.
Beyond attention. Anything that is a linear map can carry a detour: the MLP projections, the token embeddings, the LM head. The LoRA paper stuck to attention for its studies; later practice (QLoRA among others) found that spreading small ramps across more matrices usually beats concentrating rank in a few.
rsLoRA’s correction. The α/r scale over-shrinks the detour as r grows, which is one reason cranking rank often disappoints. rsLoRA rescales to α/√r, keeping the update’s magnitude stable at higher ranks — a one-line change that matters mostly above r ≈ 32.
Run one training step. The loss, if it had its way, would change the whole model — but the gradient is only allowed to land on two skinny matrices per detour. Watch where the memory goes: the optimizer’s 128 GB ledger collapses to 67 MB, the job fits a GPU you can buy retail, and the adapter replicates across all 32 layers to its full 4,194,304 numbers.
computes, for every parameter, how it should change to lower the loss. In LoRA, W₀ is marked frozen: gradients flow through it to reach earlier layers, but none are stored for it, and it is never updated. The only weights that move are A and B — two skinny matrices per detour. The frozen ocean feels the current pass through it and stays exactly where it is.
Here is the memory payoff. Adam’s expensive ~16-bytes-per-parameter bookkeeping is only ever paid for trainable parameters. Full fine-tuning: 8.006B × 16 ≈ 128 GB of states. LoRA — the whole adapter’s 4,194,304 numbers: 4,194,304 × 16 ≈ 67 MB. The frozen base still occupies 16 GB for the forward pass, but it needs no gradients, no momentum, no master copy. The optimizer’s ledger just shrank ~1,900×.
Cash it out in hardware. A full 8B fine-tune (~128 GB) needs two 80 GB datacenter GPUs. The same fine-tune as LoRA fits in ~16–18 GB — a single RTX 4090 you can buy retail. squeezes further: quantize the frozen base to 4-bit (~5 GB) and the whole fine-tune runs under ~10 GB. LoRA is why fine-tuning left the datacenter.
One layer’s detour learns; now instantiate the same pair in all 32 layers. 131,072 per layer × 32 = 4,194,304 trainable numbers — the whole adapter, wired end to end through the frozen stack. That is 0.052% of the model: for every ten thousand weights Llama-3-8B carries, LoRA trains about five. The step runs, the loss falls, and 8,006,464,512 weights haven’t budged.
What “frozen” means mechanically. One flag per tensor — requires_grad=False — and autograd stops storing gradients for it. The deeper consequence: the base is read-only in the backward pass, so it can be represented however you like — including quantized to 4-bit — without breaking the gradient math for A and B. That read-only door is exactly the one QLoRA walks through (chapter 06).
The honest caveat. LoRA shrinks optimizer and gradient memory, not activation memory — the forward pass still stores activations that scale with batch size × sequence length, exactly as in full fine-tuning. Gradient checkpointing still earns its keep on long contexts. For the full map of where GPU memory goes, see AI numbers every AI engineer should know.
Why LoRA tolerates hotter learning rates. Full fine-tuning nudges eight billion coupled weights, so a too-large step compounds into catastrophe; LoRA moves a rank-8 patch against a frozen backdrop, and practitioners routinely run it at roughly an order of magnitude higher learning rate than a full fine-tune of the same model.
LoRA+. The two matrices aren’t symmetric: A starts random and B starts at zero, and giving B a larger learning rate than A (rather than one shared rate) trains faster and slightly better — a 2024 refinement known as LoRA+.
r is the one dial that sets the adapter’s size: parameters scale linearly with it, so every halving of rank halves the file. Turn it down, then — where does quality finally break? The LoRA authors ran that experiment on the biggest model of their day, and the answer embarrassed everyone who assumed bigger updates must know more.
The LoRA paper turned the dial to the floor. On GPT-3 175B, a rank of one — a single direction through the waist — for both the query and value projections matched or nearly matched full fine-tuning: 73.4% on WikiSQL, 91.3% on MNLI, within a point of the full-rank result. One direction, against a 175-billion-parameter model. The intrinsic-dimension hunch from chapter 01 wasn’t hand-waving; the numbers backed it hard.
Why doesn’t more rank help much? The paper compared the subspaces learned at r=8 and r=64 and found their top singular directions almost identical — the extra 56 directions of the bigger adapter are largely noise the base never needed. Turning r up past a handful mostly buys parameters, not skill. The floor is low because the ceiling is too.
LoRA isn’t teaching the model random new tricks; it is turning up features the base already has but under-uses. The paper measured it: the update ΔW amplifies specific directions of the frozen W by a factor of about 21× — directions present in the pretrained weights but not emphasized there. Adaptation is mostly re-weighting latent skills, which is exactly why a few directions suffice.
In practice r is 8–64, α around 2r, adapting the attention projections — a few million parameters, a few megabytes, an afternoon on one GPU. Bigger r for harder skills far from the base; small r for nudging tone. The dial rarely needs to go high, and the cheap first experiment is almost always r=8. Our run holds there: r=8, alpha=16, targets q_proj and o_proj — 4,194,304 numbers. Time to make it permanent.
Choosing r by task distance. The closer the new skill sits to what the base already does, the lower the rank it needs. Tone and format live happily at r=4–8, domain vocabulary at r=16–32, genuinely new reasoning patterns push toward r=64 and beyond — and past that, full fine-tuning starts to earn its cost back.
α/r vs α/√r. The original α/r scale shrinks each direction’s contribution as rank grows, which can starve high-rank adapters. rsLoRA rescales to α/√r, keeping per-direction magnitude stable — a one-line change that mainly matters when you do turn the dial up.
Spread beats depth. A counterintuitive result from the paper: with a fixed parameter budget, adapting more weight matrices at low rank beats adapting fewer at high rank — r=2 across four projection types outperformed r=8 concentrated in one. Coverage of the network matters more than capacity per matrix.
Below the megabyte. VeRA shares one pair of frozen random matrices across all layers and trains only tiny per-layer scaling vectors — kilobyte-scale adapters. DoRA splits each update into a magnitude and a direction and learns them separately, buying back some full-fine-tuning behavior at LoRA prices.
Where LoRA loses. Large domain shifts — a new language, continued pretraining on billions of tokens, heavy code or math injection — can genuinely need more directions than any small r provides. The gap shows up as LoRA plateauing while full fine-tuning keeps improving.
Rank as regularizer. Read the dial the other way: a low r doesn’t just cap capacity, it constrains the update to a small subspace, which resists overfitting and catastrophic forgetting. Part of why r=8 “just works” is that it can’t wander far enough to break things.
The adapter is trained. Now the bill comes due — or rather, doesn’t. At inference, does the extra detour cost speed? And when ten customers each have their own 8.4 MB personality, how does one frozen base serve them all at once? The answer to the first question is a fold; the answer to the second is the reason LoRA took over serving.
The detour was always parallel to W₀ and joined by addition, which means it can be folded in: compute W′ = W₀ + (α/r)·B·A once, offline, and you get a single 4096×4096 matrix identical in shape to the original. The model has no adapter, no branch, no extra multiply — it is just a normal Llama with slightly different weights. This is LoRA’s signature move: an adapter that vanishes at inference.
This is LoRA’s edge over the adapter modules that came before it. Classic adapters insert extra layers that run in sequence, adding real inference latency — measurably worse at the small batch sizes serving actually uses. A merged LoRA adds exactly zero: same matrix dimensions, same FLOPs, same speed as the base model. You paid for the new skill entirely at training time.
Merging bakes one skill in. The opposite move is more powerful: keep the base frozen in GPU memory once, and 8.4 MB adapters per request — support voice, legal voice, German, SQL — each snapping onto the same 16 GB base. S-LoRA serves thousands of concurrent adapters from a single base on one GPU, batching different customers’ adapters together, up to 4× the throughput of naive serving. One model, a whole catalog of personalities.
The frozen base is read-only during LoRA training, so it can be compressed hard without breaking the math. QLoRA quantizes it to 4-bit and back-props through it into the adapter, cutting the memory to fine-tune a 65-billion-parameter model from over 780 GB to under 48 GB — one GPU. Their Guanaco model reached 99.3% of ChatGPT’s Vicuna-benchmark score after 24 hours of fine-tuning on a single card. LoRA shrank the update; QLoRA shrank the frozen base underneath it.
When not to merge. Merging is for single-skill deploys. Multi-tenant serving, rapid iteration, and A/B-testing all want the base kept swappable — one resident model, adapters loaded per request — so the 8.4 MB file stays a file. Merging also destroys the clean rollback story: un-merging means subtracting B·A back out.
Merge drift. The fold is exact in math but not in floats: adding (α/r)·B·A into fp16 weights introduces tiny rounding differences, so a merged and an unmerged model can disagree in the last decimal places. Harmless in practice — but it is why bit-exact regression suites compare against one serving mode, not both.
Stacking LoRAs. Because adapters are additive, several can be summed or weight-averaged into one base — a style LoRA plus a language LoRA often composes surprisingly well. It breaks when the adapters fight over the same directions: interference grows with overlap, and three adapters trained on similar data can compose into nonsense.
The Stable Diffusion bazaar. The loudest proof that swapping is cheap: image-model communities trade hundreds of thousands of downloadable style and character LoRAs, megabytes each, layered and re-weighted onto multi-gigabyte checkpoints at load time — model skill as a shareable file format.
Serving stacks. S-LoRA keeps all adapters in host RAM and pages the active ones onto the GPU, batching requests that use different adapters through custom kernels; Punica’s SGMV kernel and vLLM’s multi-LoRA mode are the same idea in production form. Thousands of resident personalities per GPU is now table stakes.
QLoRA, one level down. Three tricks make the 4-bit base work. First, 4-bit NormalFloat, a data type whose quantiles match normally-distributed weights. Then double quantization, which quantizes the block-scale constants themselves and claws back ~0.37 bits per parameter; and paged optimizers, which spill optimizer state to CPU RAM on memory spikes instead of crashing the run.
The whole tutorial on one balance sheet. Eight billion weights were identified, weighed, and never touched; four million new ones — five in every ten thousand — carried the entire skill, then folded back into the base or snapped on per request.
SAVED adapter.safetensors · base untouched