The compiled read model

Separate what humans write from what agents read

OKF fuses the two, and that is where it loses on efficiency. Humans write Markdown. The database keeps that Markdown exactly as written and, on every commit, compiles it into a compact artifact an agent reads instead. Export hands the Markdown back byte for byte, so okflint, the visualizer and Obsidian keep working on the folders it came from and goes to.

The read model is the part of the engine that did not change when it became a database. Given the same documents, the old engine and the new one produce identical manifests, whole and per bundle, and identical results for every checked read and search. The engine benchmarks compare them byte for byte.

A dense manifest that fits in context

The agent makes one hop: read the manifest, know exactly which three concepts it needs, fetch them in one call. No traversal.

Drop YAML for a positional encoding, and declare the keys once in a header. This is the manifest of the three documents in the quickstart:

# tenant: acme
# bundles: ops sales
id	bundle	kind	status	grain	summary	links
deploy	ops	runbook	-	-	How to ship the orders service.	-
customers	sales	bigquery_table	deprecated	customer_id	Registered customers, including churned.	-
orders	sales	bigquery_table	-	order_id	One row per completed customer order.	customers

The links column carries the whole graph, so the agent plans its reads without opening anything. On the reference catalogue a row costs about 41 tokens.

Constraints the compiler enforces: summaries are single-line, contain no tabs, and are truncated at 120 characters by default.

Why those seven columns and no others

id and bundle address. kind and grain are what the model actually needs to decide whether a concept answers the question. summary is the human sentence, compressed. links is the graph.

status is the only v0.2 trust field in the row, and only when it deviates. A concept with no status, or one marked current, gets -. deprecated and draft are what an agent has to see before choosing the concept, which is what earns a cell paid for on every turn. sources, generated, verified and stale_after are all real, and all cheaper to fetch on demand.

Nothing the database adds reaches the row. Source hashes, revision ids and commit times live in the database's own records, so committing never widens the manifest an agent pays for.

Section-addressable shards

get(id, section) returning one slice costs a fraction of the full document. On the reference catalogue a whole concept costs 445 tokens and its schema section 213. On an RFC, which numbers every subsection, it is 23,321 against 388.

An earlier draft invented @schema and @joins markers and assumed an offline LLM pass would rewrite prose into them. That turned out to be unnecessary. Markdown headings already are the section markers, and real bundles use them. Every Google sample concept carries # Schema, # Common query patterns, # Metrics.

The compiler slugs each heading and records where its section starts and ends, so section addressing works on unmodified OKF with no model in the build path.

Prompt caching is the multiplier

Put the manifest at the front of the prompt. It stays stable across turns, so every later call re-reads it at the cache rate, which the benchmarks bill at 10% of the full price.

That makes byte-determinism a hard requirement on the compiler rather than a nicety. Identical input must produce identical output, or every commit invalidates the cache and the multiplier disappears. A commit that leaves the compiled bytes unchanged, such as an edit to a navigation file, reuses the snapshot it already had, so the manifest an agent holds stays valid.

Concretely:

• rows sorted by bundle, then by id
• link lists sorted
• no timestamp anywhere in the file
• statistics reported to stderr rather than embedded
• sorting by plain code-unit comparison, never localeCompare, which varies by machine

What that buys, over twenty questions on 500 concepts:

The win comes from replacing model-driven traversal with deterministic selection in one hop, not from TSV being shorter than YAML. On that corpus the manifest is 20,549 tokens against 16,575 for a well-kept index.md.

Sorting by bundle is what makes narrowing cheap

Rows arrive grouped by bundle, so one bundle is one contiguous run of lines. The reader hands back a byte range instead of parsing and reassembling. It is exactly as deterministic as sorting by id alone, and it is what makes a large tenant usable: at 20,000 concepts one bundle is 20,486 tokens against 835,922 for the whole manifest.

When the whole manifest stops paying

At about 41 tokens per concept, the whole manifest stops being worth reading somewhere past a few thousand concepts. Narrow instead of paginating, to one bundle's slice or to a search.

The MCP server makes that call per tenant. At startup it measures the manifest, estimates one search result from the manifest's own row lengths, and adds one sentence to the manifest tool's description saying which strategy is cheaper. The benchmarks show where the crossover lands.

Retrieval takes the model out of the loop

BM25 returns top-k, then a deterministic one-hop expansion over the link graph. Zero model calls in the retrieval path. It is the largest single latency win and it is independent of the format.

Search returns manifest rows, never bodies, so the two-hop path holds. Narrow with search, then fetch only the chosen ids with get.

The searchable text is the concept's names — its id and its frontmatter title — the manifest row, and the body, minus the links column. Link targets are ids, and indexing them would make every concept match its neighbours' names. The title is indexed even though no manifest cell carries it: the compiler strips it from the document and stores it in the snapshot directory, so without the fold a carefully titled concept would retrieve worse than an untitled one.

Scoring

Textbook BM25 with k1 = 1.2 and b = 0.75, over a per-tenant index rebuilt in memory.

Tokenizing splits on every non-alphanumeric run, so order_id and order id produce the same tokens and a query written either way matches either form. The query goes through the same function, which is what keeps that true. Accents fold away before splitting — operações and operacoes are the same token — and non-Latin scripts are kept rather than dropped.

Two weights sit above the body text. Manifest cells count twice against a word of prose, because compiling the frontmatter away removed the id repetitions that resource and sources URLs used to contribute, so a summary match has to be told apart from an incidental body match. The concept's own names — id and title — count four times, because a query that names a concept should land on the concept itself, not on a neighbour that mentions it in passing.

Both values come from sweeps recorded next to the constants. For the cell weight, hit rate at 8 for queries that name a concept went from 65% unweighted to 75% at two, with no further gain above it, and weights of four and up started costing recall on queries that describe a concept instead of naming one. The name weight was swept over the reference, handbook and spec corpora. Hit rate is already captured at two, but the top position keeps improving up to four — reference MRR 0.39 at two, 0.43 at four — and nothing moves outside noise above that. Describing queries never moved, so the recall cost that caps the cell weight does not apply to a concept's own name.

Cap the expansion

The first version expanded every link of every hit, which reads fine on a small fixture and falls apart on real data. On the Stack Overflow sample one dataset row has sixteen outgoing links, so eight hits became twenty-six rows, most of the manifest.

Search that does not narrow is worse than reading the manifest, because it costs a round trip to learn nothing. Expansion is now capped at k and ranked by how many hits point at a target.

The index is not persisted

Measured build times: 19 ms at 500 concepts, 152 ms at 5,000, 578 ms at 20,000. The build streams every body off a single read of the snapshot's blob region and packs postings into typed arrays. The index takes 1.1 MB of heap at 500 concepts, 8.1 MB at 5,000 and 32 MB at 20,000, and a fresh process that opens 20,000 concepts, builds the index and searches peaks at 0.36 GB resident.

Persisting it would buy a few hundred milliseconds once per snapshot in exchange for a serialization format, invalidation logic, and its own collector. It is built in memory and cached by snapshot digest instead, which gives a new snapshot a fresh index with no invalidation code to get wrong. It is also rebuilt when the date changes, so a stale status appears on the day it falls due. serve rebuilds the index right after each sync, so the first search after a save does not pay for it.

Query time is 1.65 ms at 20,000 concepts, including the scan of the top hits' bodies that computes pos. There was nothing to buy.

Open questions

• Whether to keep OKF as the literal on-disk source, or accept any Markdown and treat OKF conformance as an export target. Settled: any Markdown compiles, and conformance is what --strict enforces.

Whether per-field weighting was worth it is no longer open: names weigh four, other cells weigh two, and the sweep behind each number is recorded next to the constant. Light plural stemming was measured and rejected on the same harness. It lifted the identifier-heavy catalogue by five points of hit rate and 0.12 of MRR, and cost rank quality on every prose corpus: the recipes lost five points and 0.06 of MRR.