Benchmarks

Method

A corpus generated to match the shape of Google's OKF samples: v0.2 frontmatter, prose written for people, # Schema and # Joins headings, links between concepts, about 2 KB each.

Twenty fixed questions with a stated ground truth, and both paths charged for delivering the same concepts. The baseline is the OKF reference consumption pattern: read index.md, read a concept, follow its links. It runs the same BM25 this project uses over the raw Markdown, with perfect navigation and never a wrong turn.

Every number below comes from one script, on one machine.

bun bench/run.ts [profile] [bundles] [concepts per bundle]

1 500 produces the token tables. 10 500 and 40 500 produce the scale rows. A profile name — spec, handbook, handbook-untitled, scripture, scripture-coarse, book, book-coarse — runs the same harness over a real public-domain corpus, downloaded once into bench/.cache. Each run prints one JSON line.

Tokens, 500 concepts in one bundle

A session is twenty questions in one conversation. Content read once stays in context and is re-billed at 10% on every later call, the prompt cache rate. A call that would add nothing new to the context is not counted at all, since an agent would not spend the turn. Both paths pay under that same model.

Scale

The tenant grows by adding bundles of 500 concepts each, which is what the 1 500, 10 500 and 40 500 runs mean. That is why the slice stays the same size.

At 500 concepts there is only one bundle, so the whole manifest and the slice are the same number.

The bundle slice stays flat as the tenant grows. That is what makes a large tenant usable at all.

Latency, median milliseconds

get is flat. Batching a fetch costs the same on a tenant of twenty thousand concepts as on one of five hundred.

None of this is where the time goes, though. One saved model round trip is worth about a second, four orders of magnitude more than any row above.

That table leaves memory mostly out on purpose. The store rebuilds the index in memory once per snapshot, and peak process memory at 20,000 concepts landed around 0.85 GB — down by roughly a third since the postings moved into typed arrays and the build stopped holding every body at once. It is still the practical ceiling on how many large tenants one daemon can hold. serve rebuilds the index right after each sync, so the first search after a save does not pay the build.

Retrieval accuracy

Whether the concept that answers the question is in the top eight, over the same twenty questions on 500 concepts.

Queries that describe a concept rather than name it land at 95% on both sides.

Document shape

Everything above describes a warehouse catalogue. The same harness runs over real public-domain corpora, converted to OKF mechanically. The saving is a property of the documents rather than of the store: it tracks how much structure — links, headings, titles — the corpus already carries. Retrieval is hit rate at 8 for queries that name a concept, store with its default expansion against the raw-file baseline.

The store meets or beats the raw files on every profile except the novels' top position (MRR 0.69 against 0.72). The savings on the prose profiles come from find, covered next: before it, prose with no headings and no links saved about two percent, because the advantage came entirely from documents that were already structured.

Locating instead of reading: before and after

Sliced reads changed the read path: offset and limit window a concept, find locates a literal phrase and returns a window around it plus the offset of every occurrence, and the MCP boundary caps a read at 15,000 characters per concept by default. The same harness ran on the commit before the change and on the commit after, same machine, so every claim below is a measured pair.

Tokens

Best strategy on each side, against the OKF navigator over the raw files:

Only the profiles whose questions are literal passages move on find, because it adds a retrieval unit rather than changing compilation or ranking. Where the questions are natural language — the catalogue, the recipes, the RFCs — find left the sessions byte-for-byte identical, which was the no-regression half of that measurement. scripture-coarse keeps its old best honestly: a 66-row manifest amortised over the session still beats paying a search per question, so find wins where the manifest is large or the documents are, not everywhere.

The Today column is the shipped tree rather than the commit find landed on, so it also carries the pos column described below. That is the one percent separating 39,274 then from 39,862 now, and it is why handbook and reference each gave back a point: pos rides in every search result, so the profiles that read through search pay for it whether or not they use it. Both manifest-strategy profiles are unchanged to the token.

Per read, on the corpora where locating applies: a novel chapter costs 3,244 tokens whole and 513 as a find window, frame and match offsets included; a Bible chapter costs 841 whole and 476 located.

Round trips

find spends exactly what search-first spends: two calls per question, 40 per session. Where it became the cheapest strategy the store now trades more calls for fewer tokens — book moves from 21 calls at 127,077 tokens to 40 at 39,862 — and the manifest strategy remains available at its old cost for a caller that prizes turns over tokens. Nothing else moved: 16 to 21 calls for the manifest strategy, 35 to 40 for search-first, before and after.

Retrieval

Unchanged in every cell, by design: ranking was not touched, and the before/after runs agree on hit rate and MRR to the last digit on all seven profiles. What find changes is what happens after the right document is found — attribution inside it becomes an exact offset rather than a read of the whole thing.

Speed

Every latency that existed before find stayed within run-to-run noise of its before value: a warm query moved by thousandths of a millisecond in both directions, and a batched get stayed flat. The new operation costs the same order as the old ones — a find is one indexOf sweep over the decompressed body. (Search itself did move later, when pos began scanning the top hits' bodies; that cost is stated in the next section rather than folded in here.)

The cap

The other half of the change is the worst case a naive MCP read can produce. get over MCP returns at most 15,000 characters per concept by default, framed as @@ id [start..end of total] and continued with offset; the library and the HTTP API stay unbounded by default. Measured as the largest single concept of each corpus:

The four read definitions grew with the three new parameters: 833 tokens per session before, 1,062 after. One window replacing one chapter repays the difference about twelve times over. The pos column's explanation, the per-tenant strategy advice and the stale status later put them at 1,122; the same window-for-chapter trade repays those sixty tokens about sixty times over.

Adding the write side took the standing bill to 1,741: write costs 413, delete 206. That is 55 percent on top of the read-only figure, and unlike the slicing parameters it does not repay itself in reads — a client that never writes pays 619 tokens per session for tools it will not call. Against a session that bills 64,355 it is under one percent; against the entry fee it is the largest single increase the tool surface has taken.

Locating without a quote: the pos column

find needs a literal phrase. Search now points instead: after BM25 ranks, the store scans the top hits' bodies for the window where the query's words cluster densest, and every direct hit ends in a pos cell — the offset to hand straight back as get(id, {offset: pos, limit: 2000}). Ranking is untouched, and the before/after runs agree on every hit rate and MRR to the last digit across all eight profiles.

The questions that needed this are the described ones — summaries rather than quotes, where find has nothing to anchor on. The same twenty questions per profile, asked descriptively, answered by capped document reads against pos windows:

The last column is checked rather than assumed: a hit means the wanted concept's window really contained the passage the question pointed at, and that held in 113 of 135 locatable cases across the eight profiles. The book misses belong to ranking, not the window — described queries reach the top eight only 40 percent of the time on headingless prose, and every question ranking did place, the window answered. book-coarse is the honest loss from the other side: four documents amortise their capped reads across twenty questions at the cache rate, and twenty fresh windows cannot.

What the column costs: six to sixteen tokens per search result, about one percent of a session, and the body scan moves a warm search from the old 0.02–0.15 ms range to 0.3–0.9 ms on ordinary corpora — up to 14 ms when the top hits are whole novels, still two orders of magnitude below the model turn it feeds.

The experiment that lost

The same round A/B-tested a minimal plural fold in the tokenizer — orders meeting order in the index. It lifted the identifier-heavy reference corpus by five points of hit rate and 0.12 of MRR, and paid for it with rank quality across every prose corpus: handbook lost five points and 0.06 of MRR, scripture-coarse 0.08 of MRR. The fold was reverted, and the losing numbers are published next to the winning ones because that is the house rule.

How much to trust these numbers

Treat all of it as an order of magnitude and measure your own bundles. The method is the transferable part: twenty real questions, tokens to first correct answer, and round trips counted. Not bundle size.