What Is RAG? Retrieval-Augmented Generation for Founders
RAG is a simple pattern: look up the passages that matter, put them in the prompt, then let the model answer from them. For most small teams the plain version is the one to beat, and the 2026 Memory Tax paper suggests many fancier memory systems do worse at a higher cost.
Key takeaways
- RAG means retrieving relevant passages from your own data and giving them to the model so the answer is grounded in them.
- The pipeline is chunk, index, retrieve, rerank and generate, and each step has cheap defaults that work.
- The October 2026 Memory Tax paper found naive chunk-based RAG remains a strong baseline against 11 agent memory systems.
- Hybrid retrieval plus a reranker beat keyword-only and embedding-only retrieval in the paper's ablations.
- Keep a path back to raw source history, and measure any added complexity against a plain RAG baseline.
RAG is a way to make a language model answer from your data instead of from memory alone. You find the passages that matter, put them in the prompt, and let the model write an answer grounded in those passages. That is the whole idea, and the plain version is still the one to beat.
TL;DR
- RAG retrieves relevant passages from your own data and passes them to the model.
- The pipeline is chunk, index, retrieve, rerank and generate.
- A 2026 paper found naive chunk-based RAG remains a strong baseline against 11 memory systems.
- Hybrid retrieval plus a reranker beat keyword-only and embedding-only retrieval in that paper's ablations.
- Keep a route back to raw source history and test every addition against a plain baseline.
Disclosure: ZeroRaised publishes this analysis and is a launch site for bootstrapped products. Facts come from the linked sources as checked on 29 Oct 2026; ZeroRaised has no commercial relationship with the companies named.
Retrieval-augmented generation (RAG) is a pattern in which a system looks up relevant passages from a body of text and passes them to a language model so the answer is grounded in those passages rather than only in the model's training data.
What does a RAG pipeline actually do?#
The pipeline has five stages, and none of them require a large team.
- Chunk. Split documents into passages small enough to retrieve precisely and large enough to carry meaning.
- Index. Store those passages so they can be searched, using keyword indexes, embedding indexes or both.
- Retrieve. Given a question, pull back the passages most likely to contain the answer.
- Rerank. Score the candidates with a second model and keep the best few.
- Generate. Put the chosen passages in the prompt and ask the model to answer from them, with citations if you need them.
The order matters less than the discipline. Each stage is a place to measure retrieval quality separately from answer quality, which is why RAG is easier to debug than asking a model to remember things.
Do you need a vector database?#
Not on its own, and not necessarily at all on day one.
The Memory Tax paper's RAG baseline is instructive: it chunked conversation turns at about 1,024 characters, used BM25 keyword search plus dense embedding retrieval, and reranked the results with a 0.6B embedding model and a 0.6B reranker (Memory Tax paper). Hybrid retrieval beat BM25-only and embedding-only retrieval in their ablations (Memory Tax paper).
That is a mundane stack. It is also the stack that many agent memory systems failed to beat. If you take one thing from the paper, take this: the retrieval quality comes from combining signals and reranking, not from buying a particular database.
What did the 2026 Memory Tax paper find?#
The paper evaluated 11 representative agent memory systems against naive chunk-based RAG on three benchmarks, LongMemEval-s, PersonaMem-v2 and MINT-Eval, across three model backbones (Memory Tax paper). Its conclusion is blunt: "naive RAG remains a strong baseline and many memory systems underperform it across diverse settings", often "despite substantially higher costs" (Memory Tax paper).
The authors give the trade a name. Memory benefit is the queries a memory system solves that RAG misses. Memory tax is the queries RAG solves that the memory system loses (Memory Tax paper). That framing is useful because it forces you to count both sides of the ledger instead of only the demos that look impressive.
They identify three causes of the tax (Memory Tax paper):
- Faithfulness. Information is lost while the memory is constructed.
- Accessibility. Evidence is fragmented, diluted or rephrased so it is harder to retrieve.
- Robustness. Behaviour breaks when the model is swapped.
The fixes are correspondingly plain. Keep a lightweight path back to the raw history and retrieve from it alongside the memory. The paper reports that this reduced the memory tax across several systems (Memory Tax paper). A concrete example: on LongMemEval-s with GPT-5.6 Luna, two-round RAG scored 90.0 against 82.0 for single-round RAG (Memory Tax paper).
How should a small team build RAG?#
Start boring, then earn your complexity.
- Begin with hybrid retrieval and a reranker. Combine keyword search with embeddings, then rerank. That is the baseline that the paper found hard to beat.
- Chunk on natural boundaries. Paragraphs, headings and conversation turns are better units than fixed windows that cut sentences in half.
- Keep a route back to the raw source. Store enough to re-open the original passage, so you can retrieve from history when the summary or memory loses something.
- Write down your questions first. A small set of real questions with known answers is the only way to tell whether a change helped.
- Measure against plain RAG before adding memory. If a memory layer does not beat the baseline on your set, it is a cost, not a feature.
The last point is where most teams go wrong. They benchmark against nothing, so every new layer looks like progress. How to evaluate LLM outputs covers building the test set, and how to choose an LLM API covers picking the model that sits at the end of the pipeline.
What this means for bootstrapped founders#
RAG is a distribution question as much as a technical one. The retrieval layer is where your product's knowledge lives, and that knowledge is often the reason a customer picks you over a generic chatbot.
Three practical consequences.
Your corpus is the moat, not the pipeline. Anyone can implement chunk, index and rerank. Few competitors have your curated documents, your support history or your customer's private data. Spend effort on ingestion quality and permissions.
Cost stays controllable. Retrieval plus a smaller model is usually cheaper than stuffing a huge context window on every request. That matters for keeping LLM costs down as usage grows.
Simplicity ships. A hybrid retrieval baseline with a reranker is a weekend of work and a clear thing to explain. A custom memory graph is a quarter of work and a harder thing to debug. For a bootstrapped team, the first version should be the one you can operate alone.
If you are building an agent rather than a search feature, the same logic applies. Retrieval is how the agent gets evidence; the harness around it decides what it does with that evidence. The agent vs workflow guide is a useful next read.
Where ZeroRaised fits#
ZeroRaised is a launch and discovery site for products built without institutional venture capital or accelerator equity. If you have shipped a product that uses retrieval to answer real questions, you can submit it here and state what data it reads and what it does with the answer. The pricing page explains the current options, and the newsletter carries the next batch of analysis.
No promises of traffic or rankings. A launch page is a place to be specific: which sources you index, how you handle permissions and what the product costs.
Sources#
Frequently asked questions
What is RAG in simple terms?
RAG, or retrieval-augmented generation, is a way of answering with a language model by first finding relevant passages in your own documents and putting them into the prompt. The model then writes an answer from that material instead of relying only on what it learned in training.
RAG vs agent memory, which is better?
That depends on the task, but a 2026 paper found that naive chunk-based RAG remains a strong baseline and that many agent memory systems underperformed it across diverse settings, often at substantially higher cost. Start with plain RAG, measure it, and add memory only where it wins.
Do I need a vector database for RAG?
Not necessarily. The Memory Tax paper's baseline combined BM25 keyword search with dense embedding retrieval and reranked the results. A vector store is one component, not the whole design, and hybrid retrieval beat either method alone in the paper's ablations.
How big should chunks be?
There is no universal number. The paper's baseline chunked on conversation turns at about 1,024 characters, which is a reasonable starting point for chat-like data. Chunk on natural boundaries first, then test sizes against questions you actually care about.
How do I know RAG is working?
Build a small set of real questions with known good answers, then compare systems on the same set. Measure whether the right passage was retrieved and whether the final answer used it correctly. Compare every added step against a plain RAG baseline.