I benchmarked plain RAG against corpus compilation on my own vault, here's what the numbers show
In Why AI Tools Fail at Your Document Archive I argued that vocabulary drift is one key reason retrieval breaks on real document archives. The same concept appears as "client onboarding friction" in one document and "sales cycle drag" in another, and if your system searches raw text, it cannot connect the two. No embedding model fixes this, because the inconsistency lives upstream of retrieval.
I had seen this in my own use, but I did not have controlled numbers, so I ran some tests. Here is what happened.
The setup
My test corpus is 181 documents, most of them from my own archive: notes and docs over my almost two decades of corp work, consulting deliverables, firm strategy memos, books, MBA class and project notes. Heavy on notes, Word and PDF, with no consistent terminology because they were written across different topics, products, projects, clients, years, and mental models. A small set of purpose-built test documents rounds out the corpus, including the dated-document chains described below.
I tested four retrieval approaches on the same corpus, with the same questions, held constant across every variable I could control:
- Arm A: Raw chunks, dense search. Standard vector RAG.
- Arm B: Compiled notes, dense search. Terminology normalized at ingest.
- Arm C: Raw chunks, hybrid search.
- Arm E: Full Elicana pipeline.
The writer model was held constant across all four arms: kimi-k2.5. The judge, deepseek-v4-pro, is from a third model family, so it was not trained to favor any arm's output style.
The questions were drawn from real queries I have tried to answer using this material. They fall into two families that matter: cross-document questions, where the answer requires combining facts from multiple documents, and temporal chains, where the answer requires knowing which dated document supersedes which.
Cross-document questions: compiled arms roughly double raw arms
The headline number on cross-document accuracy:
| Arm | Cross-document accuracy |
|---|---|
| A (raw chunks, dense) | 0.05 |
| B (compiled notes, dense) | 0.17 |
| C (raw chunks, hybrid) | 0.08 |
| E (full pipeline) | 0.21 |
Compiled arms beat raw arms by roughly 2× on the same corpus, with the same writer, judged by the same third-family model. This replicated across two independent runs.
The absolute scores are low by design. The rubrics demand complete multi-document enumerations from a mid-tier writer under a strict judge. These numbers are a ruler for comparing arms, not a product-quality grade. The point is the gap, not the height.
Arm C is worth a second look, because it is the "just improve your search" control: hybrid retrieval over the same raw chunks. It moved cross-document accuracy from 0.05 to 0.08. Arm B used the identical dense search as Arm A, on a compiled corpus, and hit 0.17. The gain lives in the corpus, not the search stack.
The gap is structural, not a tuning problem. Raw chunk retrieval finds pieces that look similar to the query. It cannot combine two separately stated facts to answer a single question. When "client onboarding friction," "deal velocity issues," and "engagement initiation" all refer to the same concept but appear in different documents, a query for any one term misses the other two. Compilation fixes this before retrieval runs. The synthesis pass extracts the concept, normalizes it across documents, and stores a representation that matches regardless of which original term the query used.
An independent benchmark published at Towards Data Science in June found directionally consistent results in a different setting: on multi-agent conversational memory, a structured context graph beat vector-only retrieval by the widest margin on join queries, where the answer requires combining two separately stated facts. Different corpus, same failure shape. I mention it only as external context. Our own numbers are what carry the claim.
Groundedness, the measure of whether answers stay anchored to retrieved context, was at parity across all arms. E scored 0.97 against raw RAG's 0.99 to 1.00. There is no hallucination tax from compilation. The answers are more accurate and just as honest.
Temporal chains: raw RAG cannot tell which document wins
The second family of questions is harder. I built nine purpose-built chains of dated documents where the newest document explicitly supersedes an earlier recommendation. These are not edge cases. They are exactly what happens in a real archive: a kickoff memo recommends one approach, a findings memo updates it, and a quarterly business review makes the final call.
On these nine chains, the accuracy numbers were:
| Arm | Accuracy |
|---|---|
| A (raw chunks, dense) | 0.03 |
| B (compiled notes, dense) | 0.42 |
| C (raw chunks, hybrid) | 0.06 |
| E (full pipeline) | 0.60 |
Raw RAG collapsed to nearly zero. It cannot tell which document wins. Compiled arms scored 0.42 to 0.60. That is not a tuning gap. It is an architecture gap.
The narrative example is a pricing recommendation chain from one of the purpose-built test chains, for a fictional company called Northwind. A kickoff deck recommended seat-based tiers. A findings memo later recommended a hybrid model. The Q1 QBR explicitly superseded both with a flat-rate structure. Only the full pipeline answered from the QBR. The raw-chunk arms retrieved pieces from all three documents and blended them into an answer that was internally contradictory and factually wrong, because nothing in the raw text told the system which date to trust.
On the full recency family of 14 queries, the pattern held: E 0.44, B 0.36, C 0.10, A 0.09. E was also the only arm with perfect groundedness on the purpose-built temporal chains: 1.00. The compiled arms knew which document was current. The raw-chunk arms did not.
What this means
If the questions you ask your archive are mostly single-document lookups, standard RAG is probably fine. Better embeddings will help.
If your questions require connecting ideas across documents, or knowing which version of a recommendation is current, retrieval-first architectures have a ceiling that no amount of retrieval tuning closes, at least not with the same model doing the answering. The problem is upstream of retrieval. Compilation normalizes the vocabulary and tracks the temporal relationships at ingest time, so the answer already exists before the query runs.
This is what I built Elicana to solve. It compiles your archive before you search it, stays on your machine, and handles the normalization and supersede tracking that raw retrieval cannot.
Related: Why AI Tools Fail at Your Document Archive, the vocabulary-drift argument this piece proves with data.