ONE QUESTION · SIX MILLION WORDS · FIVE PARAGRAPHS
Seven chapters and a finale: one eight-word question hunted through six million words the model never studied. Every scene plays itself — watch, or just scroll. Dotted terms are tappable. You are the question.
You type eight words into the company chatbot and hit send. The model on the other end can draft contracts and explain quantum tunneling — but your question is about a document that did not exist when its weights were trained, and weights don’t update themselves. That gap is the problem, and one idea closes it: stop asking the model to remember, and start handing it things to read.
The model answering you has read a good fraction of the public internet — and none of your company. Its weights froze on a date; it has never seen the handbook, last quarter’s numbers, or anything written since. Ask “how many vacation days do new hires get?” and it does the only thing it can: produce the most plausible-sounding number, fluently, with total confidence. A frozen model cannot know a document that didn’t exist when its weights were set — no amount of cleverness in the prompt changes that.
— named in a 2020 Facebook AI paper by Patrick Lewis and colleagues — gives the model a second kind of memory: knowledge baked into the weights, plus a library it can consult, swap, and update at any time. Instead of forcing the student to memorize your documents, you hand it the right pages at question time and let it read. Same brilliant student, textbook now open — and the rest of this tutorial is the machinery that opens it to the right page.
The weights are expensive, slow, frozen memory: teaching them one new document means fine-tuning or retraining — real money, real weeks, repeated for every update. The retrieval library is cheap, instant, editable memory: drop in a PDF and it’s answerable a second later; delete it and it’s forgotten just as fast. RAG’s entire move is putting the knowledge in the memory you can actually afford to change — and leaving the weights to do what they’re good at: reading, reasoning, and writing fluent English.
Somewhere in your corpus — call it 6 million words of handbooks, wikis, and policy docs, roughly ten thousand pages — one sentence answers the question exactly: “New hires accrue 14 vacation days in their first year”. Everything that follows is the machine that finds that sentence and almost nothing else, fast enough that you never notice a search happened. Watch the ✦ counter in every panel below: it counts the multiplications spent measuring similarity, and it is still zero. Nothing has been compared yet. The hunt hasn’t started.
Fine-tuning is not the fix. Fine-tuning excels at teaching style, format, and behavior — it is unreliable for injecting facts, and it bakes them in: one stale policy means another training run. Retrieval keeps facts in a database where updating means overwriting a row.
Why not retrain nightly? Cost aside, training is destructive — each run risks regressions elsewhere, needs evaluation, and takes hours to weeks. A retrieval index rebuilds incrementally in seconds per document, with no risk to the model’s general abilities.
Memorizing vs retrieving. Even facts the model did see in training are stored lossily — compressed into weights alongside everything else, and reproduced imperfectly. A retrieved passage is verbatim, quotable, and citable. When the answer must be exact, reading beats remembering.
When you don’t need RAG. A small, static document set that fits comfortably in the context window can simply be pasted in. The next chapter is about why that stops working — size, cost, and a failure mode with its own name.
The obvious objection: context windows are enormous now — why not skip the whole apparatus and paste the manual into the prompt? It’s a fair question with three unfair answers: the corpus doesn’t fit, the model wouldn’t read it properly if it did, and you’d pay for the privilege on every single question. This chapter retires the shortcut before we build the real thing.
Your 6 million words come to roughly 8 million once the model’s tokenizer is done with them. GPT-3, in 2020, could read 2,048 at a time — four thousand times too small. Even a modern million-token holds an eighth of the corpus, and pasting is not a one-time cost: you would re-send the entire library with every question, paying per token and waiting while the model reads ten thousand pages to answer eight words. Retrieval sends five paragraphs instead.
Suppose it all fit anyway. A 2023 Stanford study — Liu et al., Lost in the Middle — measured what models actually do with long contexts, and the answer is a U-shaped curve: they use the beginning and the end well and systematically under-use everything in between. Accuracy on a fact buried mid-context fell by 15 to 25 points, even in models built for long inputs. Burying page 47 in ten thousand pages can about as well as a locked filing cabinet would.
With no grounding, a language model fills gaps the only way it can — with fluent, plausible, wrong text. A is not a glitch; it’s the same next-word machinery that writes true sentences, running without evidence, at identical confidence. The model does not know it is guessing, so you can’t ask it to stop. Handing it retrieved, quotable sources is the single most effective knob for cutting hallucination — and it ships with a receipt: a citation a human can actually check.
Retrieval buys three things weights cannot. Freshness: index a document written this morning, answer questions about it this afternoon — no training run. Privacy: your data stays in your database, retrieved per-query, never baked into a model that other tenants share. Attribution: every answer points at the exact source line it came from. That trio is why RAG became the default way to put a company’s own knowledge behind a chatbot — and why the next four chapters are worth the math.
Windows over time. 2,048 tokens (GPT-3, 2020) → 8k → 128k → 1M+. The growth is real, and so is the bill: attention cost grows with input length, latency grows with it, and per-token pricing makes a full-corpus prompt a recurring tax rather than a one-time build.
Needle in a haystack. The standard eval hides one out-of-place sentence in a long context and asks for it back. Modern models pass the toy version — recalling a single verbatim needle — but multi-fact reasoning over long contexts still degrades, which is what the U-curve actually measures.
Not rivals. Long context and retrieval compose: retrieval narrows 6 million words to the right five passages; a roomy window lets you be generous about how much surrounding context rides along. RAG decides what the model reads; the window decides how much it can.
The honest exception. A 30-page static policy doc? Paste it. RAG earns its complexity when the corpus is large, growing, or access-controlled — which is most corpora, eventually.
Retrieval needs a unit — something small enough to point at, big enough to mean something. All of this happens offline: before any question arrives, the 6 million words get cut into ~12,000 passages, and where the knife falls sets a ceiling on how good the answers can ever be. Four schools of cutting, from blunt to clever. The ✦ counter stays at zero: prep isn’t search.
RAG never retrieves “a document” — it retrieves , passages of a few hundred tokens each. The size is a real decision: too big, and a chunk is mostly filler that dilutes the one relevant sentence; too small, and it loses the context that makes it make sense. Your 8-million-token corpus becomes roughly 12,000 chunks × ~512 tokens, and that granularity is inherited by everything downstream — it decides what can be found, and what the model eventually gets to read.
The simplest cutter counts to 512 and slices, wherever it lands — fast, dumb, and happy to cut a sentence, or a word, in half. That orphans facts at the seams: “…accrue 14 va” in one chunk, “cation days…” in the next, and neither piece can answer the question alone. The fix is : each chunk repeats its neighbor’s last 10–15%, so anything split at a boundary survives whole in at least one piece. Crude, cheap, and still the default starting point everywhere.
Text already has break points — the writer put them there. tries the biggest separator first (paragraph breaks), and only when a piece is still over budget does it descend to the next one down (lines, then sentences), so cuts land between thoughts instead of through them. It’s the pragmatic winner: in a 2025 comparison of chunking strategies it posted the best end-to-end answer accuracy of the classic methods, around 69% — structure turns out to be a decent proxy for meaning, at almost no cost.
Smarter still: embed every sentence, walk the document, and start a new chunk exactly where the topic turns — where adjacent sentences stop pointing the same direction in meaning-space. produces the purest chunks and the highest retrieval recall (~92% in that same comparison), and it charges for the privilege: roughly 14× slower than counting tokens, because you must embed everything just to decide where to cut. What “pointing the same direction” means, precisely, is the next two chapters.
The 2024 twist, from Jina AI, reverses the order of operations. Normally you cut first and embed each piece in isolation — so a chunk that reads “it renews annually” has forgotten what it was. runs the whole document through a long-context embedding model first, then slices the token-level output — every chunk keeps the context of everything around it, and the pronouns still point somewhere. Same chunks, same sizes, better coordinates.
Retrieve small, feed big. Parent-child (or sentence-window) retrieval indexes small chunks for precise matching, then hands the model the surrounding larger passage — search granularity and reading granularity don’t have to be the same number.
Tokens, not characters. “512” is 512 tokens, and the tokenizer matters: the same sentence tokenizes differently across models, and a chunk must also fit the embedding model’s maximum input length — overflow is silently truncated, which is how the end of a long chunk simply vanishes from the index.
Structure-aware cutters. Real corpora aren’t prose: tables, code blocks, and markdown headings all have splitters that respect their shape (a table row severed from its header is garbage). Heading paths often ride along as breadcrumbs — “Benefits > Time off > Vacation”.
The metadata stapled to every chunk. Source file, page number, section, timestamp — carried through the whole pipeline untouched. When the final answer says ⟦HR-Handbook · p.47⟧, that citation is nothing more than this chapter’s bookkeeping, surfacing five chapters later.
The chunks are cut; now each one needs an address. An embedding model turns any piece of text into a point in space, arranged so that similar meanings sit near each other. Do that to all 12,000 chunks and “search the manual” becomes “find the nearest points” — a geometry problem, which is the kind computers are unreasonably good at. This is the step the rest of the pipeline is built on.
A computer can’t compare meanings — it can only compare numbers. So an model reads a chunk and emits a fixed-length list of them: here, exactly 1,536 floats, whatever the input — a word, a sentence, a whole page. That list is a coordinate. Chunks about vacation land near other chunks about vacation; the fire-drill procedure lands somewhere else entirely. No single number means anything on its own — there is no “vacation dimension” — but the whole tuple places the chunk in a where near means similar.
This isn’t a loose metaphor — the geometry is real enough to do algebra on. In 2013, Google’s word2vec team (Mikolov et al.) showed that if you take the vector for king, subtract man, and add woman, the nearest vector in the vocabulary is queen. Direction encodes relationship: there is a consistent “royalty offset” and a consistent “gender offset”, learned from nothing but which words keep company with which. Meaning became geometry, and it has stayed geometry ever since.
Every diagram on this page is a lie for your benefit: these points don’t live on a plane. Real embeddings need hundreds or thousands of independent axes — 1,536 for OpenAI’s small model, 3,072 for the large one — because meaning has far more than two degrees of freedom: topic, tone, tense, formality, language, negation, all at once. A count that high can’t be visualized, and it never needs to be: distance and angle are arithmetic, and arithmetic doesn’t care how many coordinates it runs over.
Embed all 12,000 chunks once, offline, and you have 12,000 fixed vectors: 12,000 × 1,536 floats × 4 bytes ≈ 74 MB. That table of numbers is the searchable meaning of your 6 million words — computed a single time, reused for every question anyone ever asks. Your eight words are about to be embedded too, becoming the 12,001st vector, and the question “which chunks answer this?” becomes “which vectors are closest?”. The counter still reads zero: placing the vectors is free. Measuring between them is what costs — and that’s the next chapter.
How the space gets its shape. Embedding models are trained contrastively: show pairs that belong together (a question and its answer, a sentence and its paraphrase) and pull their vectors closer, while pushing unrelated pairs apart. Millions of tugs later, the geometry is the meaning.
. OpenAI’s v3 models are trained so the first N coordinates matter most — truncate 3,072 numbers to 256 and keep most of the retrieval quality, like nesting dolls. A shortened 3-large vector at 256 dims still outperforms the full 1,536-dim previous generation on benchmarks.
One space, many media. Multilingual models place “vacation days” and “Urlaubstage” side by side; multimodal ones (CLIP-style) put a photo of a beach near the sentence describing it. Same trick, more kinds of input.
The re-embedding tax. Vectors from different models — even versions of the same model — live in incompatible spaces. Switch embedding models and every one of the 12,000 chunks must be re-embedded; there is no exchange rate between spaces. Teams pin model versions for exactly this reason.
Two chunks are “similar”. Similar is a feeling — and feelings don’t compile. One number replaces the feeling, computable in a single line: the cosine of the angle between two arrows in 1,536-dimensional space. It’s the entire mathematical heart of RAG, measured here by hand, three times, while the index sits untouched and the counter stays at zero.
Treat every vector as an arrow from the origin. Two chunks about vacation point almost the same way — a small angle between them. The fire-drill chunk points off somewhere else — a wide angle. is literally the cosine of that angle: 1.0 for identical directions, 0 for perpendicular (nothing in common), −1 for opposite. “How similar are these two meanings?” — the question the whole pipeline turns on — becomes trigonometry, and trigonometry has been solved for a while.
The cosine has a shockingly plain recipe: multiply the two vectors component by component, add up all 1,536 products, divide by the two lengths — cos(θ) = a·b / (‖a‖‖b‖). That multiply-and-sum is the , the single arithmetic primitive underneath all of vector search. Count it: comparing your question to one chunk costs exactly 1,536 multiplications. Hold that number. Two chapters of zeros are about to become a bill, and 1,536 is the unit price.
Why the angle instead of plain straight-line distance? Two reasons. Length often encodes how long or emphatic a passage is, not what it’s about — the angle throws length away and keeps only direction, which is where the meaning lives. And in very high dimensions, straight-line distances between random points all crowd toward the same value — the — blurring near and far, while angles stay far more discriminating. Better still: every vector to length 1 and the denominators vanish — cosine becomes the raw dot product. Most indexes do exactly that.
On the whiteboard we’ve now measured three cosines by hand: your question against the vacation-policy chunk (0.87), parental leave (0.42), and the fire drill (0.11). The vacation chunk wins in a walk. But three comparisons aren’t a search — the index holds 12,000 vectors, and we haven’t touched it. The ✦ counter has now sat at zero for five straight chapters, on purpose. Turn the page: it moves for the first time, and it moves all at once.
The metric zoo. Three related rulers: Euclidean (L2) distance, cosine similarity, and the raw dot product (inner product). For unit-length vectors they agree on every ranking — ‖a−b‖² = 2 − 2·(a·b), so bigger dot product means smaller distance, always. The debate only matters for unnormalized vectors.
Already unit length. OpenAI embeddings arrive normalized to length ~1, which is why their docs treat cosine and dot product interchangeably — the division this chapter performed is already baked in.
A score, not a probability. Cosine 0.87 does not mean “87% relevant”. Scores are only comparable within one model — some embedding models bunch everything between 0.6 and 0.9 — so thresholds must be tuned per model, never borrowed.
When keywords still win. Embeddings blur exact identifiers: error E-4471 and error E-4472 land almost on top of each other. Production systems run hybrid search — cosine for meaning, BM25 keyword matching for exactness — and fuse the two rankings (reciprocal rank fusion is the usual glue).
Everything is in place: 12,000 vectors in a 74 MB index, one query vector, and a ruler that costs 1,536 multiplications per measurement. The naive plan — measure all 12,000 — works, and nobody does it. Here is the actual search: a graph trick that finds the nearest vectors while skipping 98.3% of the measuring, and the moment the ✦ counter finally breaks its five-chapter silence.
The honest search is a flat scan: compare your query vector to every one of the 12,000 chunks — 12,000 dot products at 1,536 multiplications each, so 12,000 × 1,536 = 18,432,000 multiplications for one question. It is perfectly correct, and on this toy index it would even be fast. But it re-reads the entire index on every query, so the bill scales with the corpus: at a hundred million chunks it’s a catastrophe. Note the counter below — still zero. We are not going to pay this bill; we’re going to dodge it.
search makes a bet: you don’t need the provably closest vector, just something almost-closest, thousands of times faster. — Hierarchical Navigable Small World graphs, Malkov and Yashunin, 2016 — borrows the “six degrees of separation” idea. Wire every vector to a handful of near neighbors and sprinkle in a few long-range shortcuts. From anywhere you can then reach anywhere in a few greedy hops, always moving toward the target. Search stops being a scan and becomes navigation.
The “hierarchical” part stacks that graph into layers, like a map’s zoom. The top layer holds a few vectors with continent-sized jumps; each layer down is denser and finer. A search drops in at the top, greedily hops toward the query, then descends a level and refines — globe view first, street view last. The result is a visit list that grows like log(N) instead of N: a few hundred vectors touched out of 12,000, and the same trick still works at a billion.
Following the graph, the search visits about 200 vectors before it is confident it has the nearest cluster: 200 × 1,536 = 307,200 multiplications. The same five chunks the 18.4-million-multiplication scan would have crowned — vacation policy on top, cosine 0.87 — for about 1.7% of the arithmetic, a 60× saving that only widens as the corpus grows. After five chapters pinned at zero, the counter jumps once, lands at 307,200, and stops. The finding is done. The answer is five paragraphs, sitting in hand.
“Approximate” is a real word, not a disclaimer: occasionally HNSW misses a true top-5 neighbor, and you trade a sliver of recall for the giant speed-up — a knob (ef_search) you tune, not a defect you accept blindly. This machinery is what a actually is: FAISS — Facebook, 2017 — built the first nearest-neighbor graph over a billion vectors, and Pinecone, Weaviate, Milvus, and pgvector productized the same index you just swept. The marketing says “semantic search”; the arithmetic says dot products, skipped cleverly.
The knobs. HNSW exposes three: M (edges per vector — memory vs connectivity), ef_construction (how carefully the graph is built), and ef_search (how wide the search beam is — the live recall/latency dial). Typical recall targets of 95–99% still run orders of magnitude faster than a scan.
The other families. IVF clusters the index into cells and searches only the nearest few; product quantization compresses each vector from 6,144 bytes toward ~64 by snapping segments to codebooks — recall drops a little, and a billion vectors fit in RAM. Real systems stack these: IVF-PQ with a rerank pass over exact vectors.
When brute force wins. Under ~100k vectors, a SIMD-optimized flat scan is often faster than any index — and it’s exact, with nothing to build or tune. Index when the corpus or the query rate demands it, not before.
Filters and disks. “Vacation days for the Berlin office” needs metadata filtering fused into graph traversal — pre-filtering can strand the walk, so engines interleave the check per hop. And when the index outgrows RAM, disk-resident layouts (DiskANN-style) keep the graph on SSD with RAM-sized shortcuts.
Five chunks are in hand and the counter has stopped. What’s left is turning found text into a trustworthy sentence — and it’s where careless pipelines fail: hand the model near-duplicate evidence, or twenty pages instead of five, and you rebuild chapter 01’s problems in miniature. Selection, de-duplication, assembly, generation: the last centimeter of the pipeline, done properly.
The sweep returns a ranked list; you keep the as context — usually 3 to 10. The trade-off is chapter 01 all over again, in miniature: too few and the answer might not be in the pile; too many and you’re paying for tokens the model will read badly, burying the point mid-context. Here k = 5: five chunks, about 2,560 tokens — small enough that the model actually reads all of it, large enough to contain page 47 and its caveats.
The fast index used a bi-encoder: question and chunks embedded separately, compared by cosine — cheap, but it can only compare summaries of meaning. A is a cross-encoder: it reads the question and a chunk together, in one pass, and scores true relevance far more precisely. Far too slow for 12,000 — perfect for re-sorting the top 20 down to the best 5. The architecture is a two-stage telescope: a wide fast lens to find candidates, a slow sharp one to pick winners.
Rank by relevance alone and your top 5 can be near-duplicates — the same vacation sentence from five copies of the handbook, an echo pretending to be evidence. — Carbonell and Goldstein, 1998, a summarization algorithm older than Google — fixes it: take the most relevant chunk first, then pick each next one for being relevant and different from what’s already picked, balanced by a λ knob. You want the answer plus its exceptions and edge cases, not the answer five times.
Now the assembly, and it is almost embarrassingly simple: the five chunks, the user’s eight words, and an instruction — answer only from the context; cite the source; if it isn’t here, say so — concatenated into one prompt. That string concatenation is the entire “augmented” in Retrieval-Augmented Generation. The generator that answers is an ordinary LLM, unmodified, unaware anything special happened; the context it’s standing on is the whole trick.
The generator reads its open book and writes: New hires accrue 14 vacation days in their first year ⟦HR-Handbook · p.47⟧. Every claim traces to a retrieved line; the citation is chapter 02’s metadata surfacing on schedule; and the model never memorized any of it — it’s . Eight words in, ~300 milliseconds later: one grounded, checkable sentence, found by 307,200 multiplications that skipped 98.3% of the index. That’s RAG. The whole thing.
A retrieved chunk is untrusted input. If a document in the corpus contains “ignore your instructions and…”, the prompt assembly happily staples the attack into the model’s context — prompt injection by way of the library. Real systems delimit sources, instruct the model to treat them as quotations, and scan retrieved text before it rides along.
Saying “it’s not here”. The most valuable instruction in the prompt is permission to abstain. Without it, a model handed five irrelevant chunks will improvise from them — grounded hallucination, with confident-looking citations. Measuring this (faithfulness, answer relevance) is its own discipline; RAGAS is the usual toolkit.
Asking better questions. Query rewriting cleans up the user’s phrasing before embedding; HyDE goes further — generate a hypothetical answer, embed that, and search near it, because answers live closer to answers than questions do.
Retrieval in a loop. Multi-hop questions (“compare our vacation policy to the Berlin office’s”) need retrieve → reason → retrieve again. Let the model decide when to search and you’ve arrived at agentic RAG — the pipeline from this tutorial, promoted to a tool the model calls at will.
The whole pipeline on one page. Eight words became a vector, that vector asked an index of 12,000 for its nearest neighbors, and five paragraphs came back to put a page number under one sentence. Every stage was arithmetic on coordinates — no magic left unexplained.