Benchmarks

These measure what agents and clients touch: the read model and the layer that serves it. They were rerun on 2026-10-01 with Bun 1.4.2 on an Apple M1 Pro, and every timing is the median of three runs. Token counts do not vary between runs.

The engine underneath, measured against the one it replaced, has its own protocol, its own acceptance rule and its own open questions. That is engine benchmarks.

Method

A corpus generated to match the shape of Google's OKF samples: v0.2 frontmatter, prose written for people, # Schema and # Joins headings, links between concepts, about 2 KB each.

Twenty fixed questions with a stated ground truth, and both paths charged for delivering the same concepts. The baseline is the OKF reference consumption pattern: read index.md, read a concept, follow its links. It runs the same BM25 this project uses over the raw Markdown, with perfect navigation and never a wrong turn.

bun bench/run.ts [profile] [bundles] [concepts per bundle]

1 500 produces the token tables. 10 500 and 40 500 produce the scale rows. A profile name — spec, handbook, handbook-untitled, scripture, scripture-coarse, book, book-coarse — runs the same harness over a real public-domain corpus, downloaded once into bench/.cache. Each run prints one JSON line.

Tokens, 500 concepts in one bundle

A session is twenty questions in one conversation. Content read once stays in context and is re-billed at 10% on every later call, the prompt cache rate. A call that would add nothing new to the context is not counted at all, since an agent would not spend the turn. Both paths pay under that same model.

The store has two strategies and the benchmark bills both. Keeping the manifest in the cached prefix costs the fewest round trips, which is what this project optimises for. Ranking first and fetching what ranked never reads the manifest at all. That is cheaper in tokens here and much cheaper on a large tenant, at one extra turn per question. Which strategy, and when has the crossover.

Scale

The tenant grows by adding bundles of 500 concepts each, which is what the 1 500, 10 500 and 40 500 runs mean. That is why the slice stays the same size.

At 500 concepts there is only one bundle, so the whole manifest and the slice are the same number.

The bundle slice stays flat as the tenant grows. That is what makes a large tenant usable at all.

Latency, median milliseconds

get is flat. A batched fetch of three sections costs the same 0.04 ms on twenty thousand concepts as on five hundred. Compiling, indexing, search and the whole-manifest read grow with the corpus.

None of it is where the time goes. One saved model round trip is worth about a second, three to four orders of magnitude more than any read in this table.

Memory is not in that table. The store keeps one BM25 index in memory per snapshot: 1.1 MB of heap at 500 concepts, 8.1 MB at 5,000 and 32 MB at 20,000. A fresh process that opens 20,000 concepts, builds the index and searches peaks at 0.36 GB resident. The index build streams the bodies off a single read of the snapshot's blob region, and serve rebuilds the index right after each sync, so the first search after a save does not pay for it.

Retrieval accuracy

Whether the concept that answers the question is in the top eight, over the same twenty questions on 500 concepts.

The last row is what you get without configuring anything. Expansion is on unless a caller passes expand: false, which is what the middle row measures. The CLI and the MCP tools offer no way to turn it off.

Queries that describe a concept rather than name it land at 95% on both sides.

Document shape

Everything above describes a warehouse catalogue. To find out what the store does to documents that are not reference tables, the same harness runs over four real public-domain corpora, converted to OKF concepts mechanically. Frontmatter holds type and title, headings are the document's own, and the converter adds no link, table or section the source does not already have.

Each profile gets twenty questions with ground truth stated the same way, and every path pays for delivering the same concepts. manifest is the manifest-in-the-prompt strategy, search is search-first, and find is search-first with the fetch replaced by a located window. The store's cost is the cheapest of the three. find only applies where the twenty questions quote the text, because a natural-language question has nothing to locate; n/a marks the other profiles.

Round trips are left out of that table because they barely vary. The manifest strategy spends 16 to 21 calls, and search-first and find spend 35 to 40. Both buy tokens with one extra turn per question, and this project treats a turn as the expensive resource.

The ordering follows how much structure the document already carries, not corpus size. Two things pay off.

• Headings. get(id, section) can only return a slice if the document names its slices. An RFC numbers every subsection, so a question about one costs 388 tokens instead of 23,439. A Bible chapter and a novel chapter have no headings, so a section read returns the whole chapter. find closes that gap: a located window costs 476 to 513 tokens where the whole document costs 841 to 3,244.
• Links. Batching saves a round trip only when the manifest knows which concepts belong together. Mrs Beeton's "No. 105" cross-references and the RFC citation graph both compile into the links column. The Bible and the novels have none, so the store spends exactly as many turns as the navigator.

When a corpus has neither, what is left is the frontmatter the compiler strips, and on real prose that is almost nothing: 17 tokens per chapter on the novels and 11 on the Bible, against the 149 a full OKF sample header costs.

Locating instead of reading

Without find, the novels save 2 percent. They have no headings to address and no links to batch. find changes the unit of retrieval instead of the format: search names the chapter, get with a literal phrase returns a window around the first occurrence plus the offset of every other one, and the chapter never enters the context. On these profiles the twenty questions are passages to locate.

All twenty questions located on every row. scripture-coarse is the loss. Its 66-row manifest, paid once and cached for the session, still beats paying for a search on every question, so find wins where the manifest or the documents are large, not everywhere. On the other rows the novels go from a 2 percent saving to 69, and the chapter-grain Bible drops from 148,029 tokens to 53,801 with nothing changed on disk.

Every find and search figure here includes the pos column described next, so the tables show its cost rather than a version without it.

Locating costs about what fetching does. A find sweeps the decompressed body with indexOf, and its median over the bench corpora is 0.03 to 0.3 ms: the same as get on the same document, and three to four orders of magnitude below the model turn it feeds.

Locating without a quote: the pos column

find needs a quote, and pos removes that requirement. Every direct search hit ends in a pos column with the offset where the query's words cluster densest in the body, so the second hop can be get(id, { offset: pos, limit: 2000 }) even when the question describes a passage instead of quoting it. The same twenty questions, asked as summaries rather than quotes, answered by capped document reads against pos windows:

The bench checks the last column rather than assuming it. A hit means the wanted concept's window contained the passage the question pointed at, which held in 113 of 135 locatable cases across the eight profiles. The book misses come from ranking, not from the window: described queries reach the top eight only 40 percent of the time on prose without headings, and the window answered every question that ranking did place. book-coarse loses for the opposite reason. Four documents spread their capped reads over twenty questions at the cache rate, and twenty fresh windows cannot match that. The column itself costs six to sixteen tokens per search result.

The MCP cap

Over MCP, get also caps a naive read. It returns at most 15,000 characters per concept by default, frames a partial slice as @@ id [start..end of total], and continues it with offset. The library and the HTTP API stay unbounded by default, because their callers are programs rather than prompts. Measured as the largest single concept of each corpus:

The most an MCP client pays for one concept is a constant, about 3,760 tokens, whatever the size of the document.

Which strategy, and when

You pay for the manifest once and spread it over the session, and you pay for a search on every question. That puts the crossover at a fixed ratio rather than a corpus size.

Over twenty questions the crossover lands near a ratio of twenty. Below it, keep the manifest in the prompt and pay for it once. Above it, never read the manifest and rank instead. The store makes that call itself: at startup the MCP server measures the manifest, estimates one search result from the manifest's own row lengths, and adds one sentence to the manifest tool's description, preferring search past a ratio of twenty and the whole manifest below it. The rule classifies all eight bench profiles the way the measured sessions came out, and the same manifest always gets the same advice.

Questions that quote the text add a third strategy, search then locate, whose fetch is a flat window of about 500 tokens whatever the document size. It wins on three of the four quote-shaped profiles. On scripture-coarse the manifest stays ahead because it is small. For questions that do not quote, the ratio above still decides between manifest and search.

Concept grain is a corpus decision with a large price

scripture and scripture-coarse are the same 1,189 chapters of the same text. The only difference is whether a chapter is a concept or a heading inside a concept.

That is 73 percent cheaper for the same text. The manifest re-read on every turn went from 1,189 rows to 66, while a read stayed one chapter, 845 tokens against 841, because chapters became addressable sections. Nearly two thirds of the original bill was re-reading the manifest.

It does not carry over to the novels. Going from book to book-coarse saves 4 percent, because their manifest was only 1,772 tokens and the cost is the 3,244-token chapter itself. Coarsening helps a corpus whose manifest is large, not one whose documents are large. The novels need an addressable unit smaller than a chapter, which is what find provides.

Retrieval across corpora

The same questions, top eight, hit rate and MRR:

The store matches or beats the raw files everywhere except MRR on the novels, 0.69 against 0.72. Prose with no headings and no links gives the compiler nothing to work with.

There is no stemming, and that is a measured decision. A minimal plural fold lifted the identifier-heavy reference corpus by five points of hit rate and 0.12 of MRR, and cost rank quality on every prose corpus: handbook lost five points and 0.06 of MRR, and scripture-coarse lost 0.08 of MRR. Those numbers are why the fold was reverted, and they are published next to the winning ones because that is the house rule.

Titles get special treatment because of handbook. Mrs Beeton numbers her recipes, so the id is recipe_181 and the words "rabbit soup" live only in the title the compiler strips. Before the index folded titles in, the store hit 50 percent on handbook against the baseline's 90. The index now puts every concept's title next to its id, weighted above the other fields, and the store hits 95.

handbook-untitled is the same bundle with the title moved out of the frontmatter into the body as an H1, which is how a downloaded document usually looks. It is the control that shows the title fold works.

Indexing the title costs zero prompt tokens and no manifest width. It rides in the snapshot directory, never in a row the agent pays for on every turn.

The serving layer

bun bench/ops.ts [profile] [bundles] [concepts per bundle]

Disk figures cover the complete tenant, source archives, revision records and import metadata included.

One query, by how you reach it. Cold is a fresh embedded invocation that builds the index first. Warm is the same question against an index already in memory in the same process. The socket row is a real serve over a unix socket, so it includes serialisation and IPC.

A daemon is 55 to 279 times faster. Compare cold with the socket row, which is the fair pair: 17.2 ms against 0.31, 157 against 0.56, 148 against 0.64. An embedded invocation rebuilds the BM25 index from nothing before it can answer anything, and that is the whole of the difference.

Transport is a fixed cost, not a proportional one. The socket adds 0.13 to 0.16 ms over the in-process figure, and the MCP layer adds another 0.13 to 0.22 ms. Neither grows much with the corpus. The query itself does. Warm search runs 0.18 to 0.48 ms here because it decompresses and scans the top hits' bodies to compute pos, and it reaches 14 ms when the hits are whole novels. That is still about two orders of magnitude below the model turn it feeds.

The write path, through the HTTP source routes where the precondition lives:

Losing a write race is cheap to discover. A write that names the version it replaces costs 14 to 44 ms end to end, because the engine syncs every new artifact to disk before it publishes the commit. A write that names a stale version is refused in 1 to 4 ms, because nothing gets written. The server refused every stale write in every run rather than merging it.

Immutability is paid in disk, linearly in edits. A snapshot is one file named by its own hash, and two snapshots share nothing, so a one-line change writes a full copy. Ten edits leave eleven snapshots and eleven times the tenant on disk at every scale tested, which is 125 MB from an 11.4 MB tenant at 5,000 concepts. Rollback is free, but gc is what keeps the design affordable. It keeps ten committed revisions by default and removes the rest.

A no-op sync is cheaper than an edit, but not free. A sync that changes nothing costs a third to a half of one that changes something: 9.5 ms against 27.6 at 500 concepts, 84 against 167 at 5,000. The store detects the no-op and writes nothing, but only after reading the whole tree. That is the watcher's steady cost per burst of filesystem events, and at 5,000 concepts it is 84 ms of work to conclude that nothing happened.

Determinism holds. Identical input compiled twice produces the same snapshot hash and the writer reuses it, at every size tested. That is what keeps the manifest in the client's prompt cache across a rebuild.

Memory is mostly the index. A tenant's BM25 index takes 1.4 to 3.5 times its snapshot size in heap after a full collection, 8.1 MB for 5,000 concepts. Resident memory per tenant is too noisy to report on Bun 1.4: across runs it moved between 0.2 and 8.9 MB per 5,000-concept tenant, because the allocator reuses pages that earlier index builds freed.

What registering the MCP server costs

That is the entry fee, paid in the client's system prompt whether or not the model ever asks about knowledge, and it does not change with corpus size. The schemas of get's four slicing parameters are 131 tokens of it, repaid the first time one window replaces one chapter. Another 156 explain the pos column, the stale status and the tenant's strategy advice, at 78, 55 and 23 tokens. One avoided capped read of 3,758 tokens repays them 24 times over.

The write side is 619 of those tokens, 55 percent on top of the read tools. write alone ties get as the most expensive definition in the set, 412 tokens against 413. Seventy of them are the paragraph that teaches a model to recover from a refused precondition, which is cheaper than shipping a separate read-the-hash tool with a definition of its own. delete is half the size because it refers to that lesson instead of restating it. A deployment that only ever reads pays all 619 for nothing, and no flag omits them.

The three database tools, transact, history and restore, are off unless a server starts with --database-tools. They add 510 tokens, measured without the tenant's advice sentence; see the agent API.

How much to trust these numbers

The real corpora are downloaded on first run and converted by parsers written against one edition of each text, and the harness fails loudly if an edition stops matching rather than benchmarking a corpus it silently mangled. Every timing comes from one machine.

Treat all of it as an order of magnitude and measure your own bundles. The method is the transferable part: twenty real questions, tokens to first correct answer, and round trips counted. Not bundle size.