The compiled read model

Separate what humans write from what agents read

OKF fuses the two, and that is where it loses on efficiency. Humans write Markdown. A compiler emits a compact artifact. The OKF source stays intact, so okflint, the visualizer and Obsidian keep working on the same folder.

A dense manifest that fits in context

Budget 10 to 15 tokens per concept. Then the agent makes one hop: read the manifest, know exactly which three files it needs, fetch them in parallel. No traversal.

Drop YAML for a positional encoding, and declare the keys once in a header.

# tenant: acme
# bundles: ops sales
id	bundle	kind	status	grain	summary	links
orders	sales	bigquery_table	-	order_id	One row per completed customer order.	customers
customers	sales	bigquery_table	deprecated	customer_id	Registered customers, including churned.	orders
rev_net	sales	metric	-	-	gross - refunds - tax	orders payments

Against the YAML equivalent this cuts structural overhead per record by roughly four to five times. The links column carries the whole graph, so the agent plans traversal without opening anything.

Constraints the compiler enforces: summaries are single-line, contain no tabs, and are truncated at 120 characters by default.

Why those seven columns and no others

id and bundle address. kind and grain are what the model actually needs to decide whether a concept answers the question. summary is the human sentence, compressed. links is the graph.

status is the only v0.2 trust field in the row, and only when it deviates. A concept with no status, or one marked current, gets -. deprecated and draft are what an agent has to see before choosing the concept, which is what earns a cell paid for on every turn. sources, generated, verified and stale_after are all real, and all cheaper to fetch on demand.

Section-addressable shards

get(id, section) returning one slice costs a fraction of the full document.

An earlier draft invented @schema and @joins markers and assumed an offline LLM pass would rewrite prose into them. That turned out to be unnecessary. Markdown headings already are the section markers, and real bundles use them. Every Google sample concept carries # Schema, # Common query patterns, # Metrics.

The compiler slugs each heading and records its byte range, so section addressing works on unmodified OKF with no model in the build path.

Measured on the GA4 sample: fetching the events_ concept whole costs 9,419 bytes, fetching --section schema costs 5,486. The saving grows with document size, and it costs nothing to have.

Prompt caching is the multiplier

Put the manifest at the front of the prompt. It stays stable across turns, so it hits cache at roughly 10% of the cost.

That makes byte-determinism a hard requirement on the compiler rather than a nicety. Identical input must produce identical output, or every rebuild invalidates the cache and the multiplier disappears.

Concretely:

• rows sorted by bundle, then by id
• link lists sorted
• no timestamp anywhere in the file
• statistics reported to stderr rather than embedded
• sorting by plain code-unit comparison, never localeCompare, which varies by machine

Rough order of magnitude for 500 concepts:

The win comes from replacing model-driven traversal with deterministic selection in one hop, not from TSV being shorter than YAML.

Sorting by bundle is what makes narrowing cheap

Rows arrive grouped by bundle, so one bundle is one contiguous run of lines. The reader hands back a byte range instead of parsing and reassembling. It is exactly as deterministic as sorting by id alone, and it is what makes a large tenant usable.

Scaling break point

The manifest stops fitting somewhere around 2,000 concepts, call it 25k tokens. Past that, shard by domain. A domain-level manifest of 50 lines stays cached, then the domain manifest loads. Still two hops.

Retrieval takes the model out of the loop

BM25 returns top-k, then a deterministic one-hop expansion over the link graph. Zero model calls in the retrieval path. It is the largest single latency win and it is independent of the format.

Search returns manifest rows, never bodies, so the two-hop path holds. Narrow with search, then fetch only the chosen ids with get.

The searchable text is the concept's names — its id and its frontmatter title — the manifest row, and the body, minus the links column. Link targets are ids, and indexing them would make every concept match its neighbours' names. The title is indexed even though no manifest cell carries it: the compiler strips it from the document and stores it in the snapshot directory, so without the fold a carefully titled concept would retrieve worse than an untitled one.

Scoring

Textbook BM25 with k1 = 1.2 and b = 0.75, over a per-tenant index rebuilt in memory.

Tokenizing splits on every non-alphanumeric run, so order_id and order id produce the same tokens and a query written either way matches either form. The query goes through the same function, which is what keeps that true. Accents fold away before splitting — operações and operacoes are the same token — and non-Latin scripts are kept rather than dropped.

Two weights sit above the body text. Manifest cells count twice against a word of prose, because compiling the frontmatter away removed the id repetitions that resource and sources URLs used to contribute, so a summary match has to be told apart from an incidental body match. The concept's own names — id and title — count four times, because a query that names a concept should land on the concept itself, not on a neighbour that mentions it in passing.

Both values were swept measuring hit rate at 8 for queries that name a concept. The cell weight was swept over the 500-concept catalogue; the name weight over three corpora at once:

Four is the smallest value that captures the gain. The recall cost that caps the cell weight at two does not apply here: boosting a concept's own name never hurt describing queries in the sweep.

Cap the expansion

The first version expanded every link of every hit, which reads fine on a small fixture and falls apart on real data. On the Stack Overflow sample one dataset row links to sixteen tables, so eight hits became twenty-six rows, most of the manifest.

Search that does not narrow is worse than reading the manifest, because it costs a round trip to learn nothing. Expansion is now capped at k and ranked by how many hits point at a target, which took --k 3 from 85% of the manifest down to 31%.

The index is not persisted

Measured build times: 22 ms at 500 concepts, 193 ms at 5,000, 674 ms at 20,000. The build streams every body off a single read of the snapshot's blob region and packs postings into typed arrays, which is what keeps a 20,000-concept tenant under a gigabyte of process memory.

Persisting that would buy a few hundred milliseconds once per snapshot in exchange for a serialization format, invalidation logic, and its own collector. It is built in memory and cached by snapshot digest instead, which makes a rebuilt tenant get a fresh index with no invalidation code to get wrong — and serve rebuilds it right after each sync, so the first search after a save does not pay it either.

Query time is 2.13 ms at 20,000 concepts, most of it the pos scan over the top hits' bodies rather than the ranking. There was nothing to buy.

Open questions

• Whether to keep OKF as the literal on-disk source, or accept any Markdown and treat OKF conformance as an export target. Settled: any Markdown compiles, and conformance is what --strict enforces.

Whether per-field weighting was worth it is no longer open: names weigh four, other cells weigh two, and the sweep behind each number is recorded next to the constant. Two candidate upgrades were measured and rejected on the same harness — expanding the link graph inbound as well as outbound (no gain on any profile) and light plural stemming (a ten-point win on the catalogue paid for by a five-point loss on the recipes).