Hi SaugaTech community,
A few years back, I was trying to make sense of an old banking regulation, we were trying to dig through old printed regulations for a clause buried in a document from almost twenty years earlier. Nobody in the room had it memorized. What happened instead was completely ordinary - one of our office assistants dug out the filing cabinet, pulled the right folder, and read the actual line back. Nobody needed to have the answer sitting in their head. They just needed to know where to look.
That small, boring moment is basically the plot of this week’s paper. Part 3 of AI Under the Hood is here, and it’s the one that explains why “chat with your documents” became the single most common thing builders in this community have shipped over the past year.
If you’re just joining the series, Part 1 covered the Transformer and Part 2 covered how GPT-3 learned to pick up new tasks from a few examples. Neither is required reading for this one, but they’ll make a few things click faster.
Grab a coffee. Let’s get into it.
🚀 First Things First
Meetup #11 is ten days out — August 22nd at IDEA Mississauga, Square One. The topic “Tech Careers in 2026: The New Rules of Getting Hired.”
We’ve got a great panel of experienced tech managers and recruiter coming in to talk about what’s actually changed on the other side of the table. If you’ve been job hunting, thinking about a switch, or just curious, join us at IDEA Mississauga on 22nd afternoon. We are also hosting our 2nd builders showcase along with the usual Open Networking session, so its going to be another day to spend with the vibrant SaugaTech community.
RSVP here
Anyway, coming back to our today’s AI paper.
⚓ The Paper: “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020)
In May 2020, a team at Facebook AI Research led by Patrick Lewis published a paper that tackled a problem every builder in this community has bumped into by now. The model sounds confident. The model is also sometimes just wrong. It has a name in the field now — hallucination — and this paper is one of the earliest and cleanest fixes for it. The shorthand everyone uses is RAG.
The problem the researchers tried to solve
Everything a language model like GPT-3 or BART knows, it learned during training. Once training ends, that knowledge is frozen. Ask about something that happened last week, something buried in your company wiki, or something that simply wasn’t common enough in the training data, and the model either admits it doesn’t know or, worse, fills the gap with something fluent and wrong.
There’d already been attempts at fixing this — REALM and ORQA both paired a language model with some kind of external lookup — but they were built specifically for extractive question answering, basically highlighting the exact span of text that contains an answer. Nobody had shown this idea working as a general-purpose approach across a wide range of tasks, including ones where the model has to actually write a sentence, not just point at one.
The core idea — two kinds of memory, working together
The paper gives a name to something worth remembering.
Parametric memory is what the model learned during training, baked into its weights. Fast, fluent, and frozen the moment training ends.
Non-parametric memory is an external, searchable collection of documents the model can look things up in — in this case, a snapshot of Wikipedia chopped into roughly 21 million chunks of about 100 words each.
RAG connects the two. Before the model writes an answer, a retriever component searches the document collection for the passages most relevant to the question and hands those passages to the generator as extra context. The generator then writes its answer with those actual documents sitting in front of it, instead of relying purely on whatever it memorized months earlier.
Our filing cabinet from the opening holds up well here. The retriever is the person who knows exactly which drawer to check. The generator is the person who reads the line back in plain English. Neither one does the whole job alone.
How it works: the library card catalogue
The retriever is built on a technique called Dense Passage Retrieval, DPR for short. Instead of matching keywords the way an old-school search engine does, DPR converts both the question and every document chunk into a vector, a long string of numbers that captures meaning rather than exact wording. A question about west GTA development ends up sitting close to a passage about Mississauga’s growth as a business hub, even when the actual words barely overlap.
Finding the closest match among 21 million vectors sounds like it should take forever. It doesn’t, thanks to a search technique called Maximum Inner Product Search, which narrows the field down in a fraction of a second using an index built ahead of time. This is the same basic machinery, evolved a fair bit since, sitting underneath every vector database you’ll hear a developer mention today.
Once the retriever hands over its top candidates, the generator — BART, a 400 million parameter model related to the ones we covered in Part 1 — simply concatenates the retrieved passages with the original question and writes the answer. The retriever and generator are trained together, end to end, with no one ever telling the retriever exactly which document is “correct.” It learns which documents are useful purely from whether they helped the generator land the right answer.
The paper tests two flavours of this. RAG-Sequence picks one set of documents and uses it for the whole answer, start to finish. RAG-Token can lean on a different document for each word it generates, which lets it blend information from more than one source into a single answer.
The evidence
RAG set a new state of the art on four open-domain question answering benchmarks — Natural Questions, TriviaQA, WebQuestions, and CuratedTrec — beating both pure “closed-book” models that only had parametric memory, and specialized extract-the-span systems that only had retrieval. On Natural Questions specifically, RAG hit 44.5% exact match, well ahead of the closed-book T5 model’s 34.5%, using a fraction of the parameters.
The most convincing evidence isn’t a benchmark score though. It’s an experiment the researchers ran almost as a side note. They built two separate document indexes — one from a Wikipedia snapshot taken in December 2016, one from December 2018 — then found 82 world leaders whose positions had changed between those dates and asked the model “Who is {position}?” against each index separately.
Using the 2016 index, the model correctly named the 2016 leader about 70% of the time. Swap in the 2018 index, and it named the 2018 leader correctly about 68% of the time. Mismatch an outdated index with a current question, and accuracy fell into single digits.
That’s the whole point of the architecture in one experiment. You can update what the model “knows” by swapping the document collection, without touching a single one of its trained weights.
Where it fell short
The paper is upfront about a failure mode they call retrieval collapse. On some tasks, especially open-ended ones like story generation, the retriever would learn to fetch the same handful of documents regardless of what was actually being asked, and the generator would learn to just ignore them, quietly reverting to acting like a plain BART model. The researchers suspect it happens more on tasks where the target output is long and the connection to any one specific fact is loose, which gives the retriever a weaker training signal to learn from. It’s a reminder that bolting on retrieval doesn’t automatically make a system reliable. The retriever still has to actually be doing something.
⚓ Why this actually matters for what we build
This paper is close to a blueprint for the majority of “AI knowledge base” products built in the last two years, including probably a few sitting in this community’s WhatsApp group right now.
The first is the most practical one for anyone reading this with a day job and limited weekends: you don’t need to train a model to build something like this. You need a way to search your own documents, and an existing language model to write the answer once the right passages are sitting in front of it. Both are sitting on the shelf already. A handful of GTA builders in this community have already stitched the two together over a weekend.
If a chatbot you’ve built keeps returning garbage, check the filing system before you blame the model. That’s usually where the fault actually sits. Chunks that are too big or too small, embeddings that miss what a question is really asking, an index that just doesn’t contain the passage you needed — nine times out of ten, that’s the failure, not the language model reading the retrieved documents wrong.
And domain expertise is what actually goes in the filing cabinet. A generic chatbot knows nothing specific about your industry. Point one at your firm’s compliance manual, your clinic’s intake procedures, or your logistics company’s routing rules, and it suddenly knows exactly what it needs to — because you’re the one who decided what goes into the index. Years spent in healthcare, manufacturing, or financial services aren’t something a prompt can replicate. That experience is the fuel. Retrieval just puts the right piece of it in front of the model at the right moment.
The world-leader experiment is also a decent argument for why you don’t need to retrain anything every time your information changes. Swap the documents, and the answers update on their own. It’s a big part of why so many teams are reaching for a vector database right now instead of waiting on the next model release to simply “know more.”
One thing worth watching before you ship, though: don’t assume retrieval is pulling its weight just because you wired it in. The paper’s own retrieval collapse finding is a decent gut check — if the connection between a question and a specific document is loose, or your outputs run long, it’s worth checking whether the system is actually using what it retrieves or quietly ignoring it.
✨ SaugaTech Epilogue — three papers, one machine
Worth pausing here, three parts in, to notice these aren’t three separate stories. They’re one machine, built in layers, and each paper we’ve covered solved the problem the last one couldn’t.
Part 1 gave us Attention — a way for a model to read an entire sentence at once and figure out which words actually matter to each other, instead of plodding through word by word and forgetting the beginning by the time it reached the end. That’s the engine. It’s what makes a model capable of understanding context in the first place.
Part 2 took that engine and asked what happens if you build it big enough. Turns out, at 175 billion parameters, something new shows up that smaller models just don’t have — the ability to learn a brand new task on the spot, from a couple of examples in the prompt, no retraining required. That’s what makes the engine flexible. One model, reused for a dozen different jobs, just by changing what you show it.
This week’s paper takes that flexible engine and gives it something neither of the first two papers had. A memory it didn’t have to be born with. Attention lets it understand what’s in front of it. Few-shot learning lets it adapt to what you’re asking. RAG lets it go check something it never saw during training at all, and come back with an answer grounded in an actual document instead of a confident guess.
Understand, adapt, look things up. That’s most of what a serious AI product needs to do well, and it’s also, not coincidentally, roughly the order these ideas arrived in the research world.
There’s a good instinct for builders buried in that order too. You don’t need to have memorized the whole industry before you start something. You need to know where the filing cabinet is, and be willing to go check it before you speak.
Next time, we go back to something the engine still doesn’t do well on its own — reliably being helpful, rather than just being fluent — and the training trick, RLHF, that taught it the difference.
See you on August 22nd.
Let’s keep building, Let’s keep learning, Together.
Team SaugaTech
CONNECT | COLLABORATE | INNOVATE