[dlotterman]

llms.txt is important enough to the AX that we should re-orient some things around it

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
Datum	🟢 Ready	View Preview	Oct 30, 2026, 8:00 PM

[polylane]

Warning

Polylane could not verify the production impact of this pull request.

This change touches only documentation content and docs.json (descriptions, agent instructions, /runtime/* redirects) for the Mintlify-hosted docs site. No production resource runs it: datum-cloud/docs has no deploy edge in the graph, and docs.json parses cleanly.

View the full analysis →

Also considered · 1 refuted

• Refuted · New /runtime/* redirects shadow live docs routes · No runtime/.mdx page exists in the tree (only compute/runtimes/.mdx), the 79-entry redirect list has no duplicate source, and the docs site is not a production node in the graph, so no production resource degrades on merge.

Analysed against 2 cloud accounts and 1 repository

• Cloud accounts: datum-cloud-prod, infrastructure-control-plane-staging
• Repository: datum-cloud/docs

Polylane could not find the cloud resources this repository manages, so this review looked at the entire cloud account. Connect this repository to its resources and the next review will focus on exactly what this code deploys to.

_{Polylane analysed 5898c15 for production impact. You can ask follow-ups by mentioning @polylane in a comment.}

Did this help? React 👍 or 👎 so the next review is sharper.

Previous verdicts (2)

Head	Verdict	Analysis
7b03098	Production impact not verified	analysis
c81c668	Production impact not verified	analysis

[dlotterman]

From Claude:

 Every stage, both models

  Baseline is the unchanged repo. Round 1 added the instructions block, the new summary line and most of the description pass. Round 2 split the instructions by agent type and added a few more descriptions. Round 3 added the last two     
  descriptions (DNS plugin install, gVPC limits) and the content fixes.

  Original 8 questions

  ┌──────────┬──────────────────────────────┬─────────────────────────────────────────────┐
  │  Stage   │      Haiku (fetch only)      │                Luna (shell)                 │
  ├──────────┼──────────────────────────────┼─────────────────────────────────────────────┤
  │ Baseline │ 8/8 · 9 fetches · 1 dead end │ 8/8 · 9 fetches · 1 dead end · 46.3k tokens │
  ├──────────┼──────────────────────────────┼─────────────────────────────────────────────┤
  │ Round 1  │ 8/8 · 9 · 1 dead end         │ 8/8 · 1 (grepped the full file) · 39.4k     │
  ├──────────┼──────────────────────────────┼─────────────────────────────────────────────┤
  │ Round 3  │ 8/8 · 9 · 1 dead end         │ 8/8 · 8 · 0 dead ends · 23.6k               │
  └──────────┴──────────────────────────────┴─────────────────────────────────────────────┘

  Held-out 8 questions (written before any edits, so the docs couldn't be tuned to them)

  ┌──────────┬────────────────────────────────┬─────────────────────────────────────────┐
  │  Stage   │       Haiku (fetch only)       │              Luna (shell)               │
  ├──────────┼────────────────────────────────┼─────────────────────────────────────────┤
  │ Baseline │ 7/8 · 12 fetches · 3 dead ends │ 8/8 · 9 fetches · 24.7k tokens          │
  ├──────────┼────────────────────────────────┼─────────────────────────────────────────┤
  │ Round 2  │ 7/8 · 10 · 2 dead ends         │ 7/8 · 8 · 21.8k                         │
  ├──────────┼────────────────────────────────┼─────────────────────────────────────────┤
  │ Round 3  │ 8/8 · 7 fetches · 0 dead ends  │ 8/8 · 1 (grepped the full file) · 30.3k │
  └──────────┴────────────────────────────────┴─────────────────────────────────────────┘

  For reference, using only the docs llms-full.txt at baseline: Luna scored 8/8 in 1 fetch (27.9k tokens), and Haiku scored 8/8 but needed 11 fetches, 8 of them chunks of the same file.

  What improved, and why

  - Haiku on the held-out set went from 7/8 to 8/8, and from 12 fetches to 7. Its own feedback credits the descriptions. The gVPC limits description now names "the locations that carry Compute traffic", which answered a question it had   
    missed twice. The DNS operations description now names datumctl plugin install dns. The placeholder labels let it skip empty pages without fetching them.
  - Luna's round-2 miss is fixed. It got the DNS plugin question right this time, from the reworded description.
  - Luna now takes the right route for its abilities. In 2 of 4 runs with the instructions block, it downloaded the full docs file once and searched it. In the other 2 it fetched pages, and still scored 8/8 with no dead ends.
  - Luna's token cost is mixed. On the original set it fell from 46.3k to 23.6k. On the held-out set it rose from 24.7k to 30.3k, because searching the 740 KB file produced long output. The main gain for Luna is accuracy plus fewer       
    fetches, not cost.
  - Haiku's one remaining dead end is expected. On the original set it still opens the Tunnels page first. The new "How a tunnel connects" section fixes that, but it only goes live after deploy, because the tests read the live pages.     

  Not measured until deploy

  - The Tunnels section and the plugin placeholder fix: content changes that only show up after deploy.
  - How agents react when they fetch the real llms.txt through a summarizing tool. These tests read the index locally. The instructions block sits at the top of the file, so it's the part most likely to survive summarization.

[dlotterman]

  Original 8 questions

  ┌──────────┬──────────────────────────────┬─────────────────────────────────────┬──────────────────────────────────────┬────────────────────────────┐
  │  Stage   │    Haiku 5.5 (fetch only)    │            Luna (shell)             │    DeepSeek-V4-Flash (fetch only)    │ GLM-5.3-Flash (fetch only) │
  ├──────────┼──────────────────────────────┼─────────────────────────────────────┼──────────────────────────────────────┼────────────────────────────┤
  │ Baseline │ 8/8 · 9 fetches · 1 dead end │ 8/8 · 9 · 1 dead end · 46.3k tok    │ 8/8 · 10 · 1 dead end                │ 8/8 · 9 · 1 dead end       │
  ├──────────┼──────────────────────────────┼─────────────────────────────────────┼──────────────────────────────────────┼────────────────────────────┤
  │ Round 1  │ 8/8 · 9 · 1 dead end         │ 8/8 · 1 (grepped llms-full) · 39.4k │ —                                    │ —                          │
  ├──────────┼──────────────────────────────┼─────────────────────────────────────┼──────────────────────────────────────┼────────────────────────────┤
  │ Round 3  │ 8/8 · 9 · 1 dead end*        │ 8/8 · 8 · 0 dead ends · 23.6k       │ 8/8 · 10 (2 re-checks) · 0 dead ends │ 8/8 · 9 · 0 dead ends      │
  └──────────┴──────────────────────────────┴─────────────────────────────────────┴──────────────────────────────────────┴────────────────────────────┘

  Held-out 8 questions (written before any edits)

  ┌──────────┬────────────────────────┬────────────────────────────────────────────────────┬─────────────────────────────────────────────────────┬───────────────────────┐
  │  Stage   │       Haiku 5.5        │                        Luna                        │                  DeepSeek-V4-Flash                  │     GLM-5.3-Flash     │
  ├──────────┼────────────────────────┼────────────────────────────────────────────────────┼─────────────────────────────────────────────────────┼───────────────────────┤
  │ Baseline │ 7/8 · 12 · 3 dead ends │ 8/8 · 9 · 24.7k                                    │ 8/8 · 32 (got lost, fell back to chunked llms-full) │ 7/8 · 10              │
  ├──────────┼────────────────────────┼────────────────────────────────────────────────────┼─────────────────────────────────────────────────────┼───────────────────────┤
  │ Round 2  │ 7/8 · 10 · 2 dead ends │ 7/8 · 8 · 21.8k (fooled by the plugin placeholder) │ —                                                   │ —                     │
  ├──────────┼────────────────────────┼────────────────────────────────────────────────────┼─────────────────────────────────────────────────────┼───────────────────────┤
  │ Round 3  │ 8/8 · 7 · 0 dead ends  │ 8/8 · 1 (grepped llms-full) · 30.3k                │ 8/8 · 7 · 0 dead ends                               │ 8/8 · 7 · 0 dead ends │
  └──────────┴────────────────────────┴────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┴───────────────────────┘

  * Haiku's last dead end is the Tunnels page. Its content fix isn't deployed yet, and these tests fetch live pages.

  What the tables show:
  - Held-out questions are where the changes matter. At baseline, every fetch-only model either missed a question (Haiku, GLM) or thrashed to get it (DeepSeek, 32 fetches). At round 3, all three scored 8/8 in exactly 7 fetches with no    
    dead ends. Most of that came from one description: the gVPC limits page now names "the locations that carry Compute traffic".
  - On the original questions, dead ends went away for three of the four models. Haiku still opens the Tunnels page first.
  - Luna takes the shell route when the instructions suggest it (1 fetch in 2 of its 4 runs since the block was added). It ends at 8/8 on both sets. Its token cost is mixed: the full-file grep is sometimes larger than fetching pages.  
 

[dlotterman]

  Results

  Two runs per cell, docs-only (each model sees only a snapshot of the docs).

  ┌───────────────────┬────────────────────────┬────────────────────┐
  │       Model       │        Old docs        │     Fixed docs     │
  ├───────────────────┼────────────────────────┼────────────────────┤
  │ Haiku 5.5         │ 90% · 6.5 turns · 12 s │ 100% · 5.9 · 11 s  │
  ├───────────────────┼────────────────────────┼────────────────────┤
  │ Luna (Codex)      │ 90% · 3.6 · 14 s       │ 100% · 3.3 · 13 s  │
  ├───────────────────┼────────────────────────┼────────────────────┤
  │ DeepSeek-V4-Flash │ 75% · 16.3 · 123 s     │ 100% · 10.0 · 74 s │
  ├───────────────────┼────────────────────────┼────────────────────┤
  │ GLM-5.3-Flash     │ 95% · 9.3 · 192 s      │ 100% · 7.8 · 122 s │
  └───────────────────┴────────────────────────┴────────────────────┘

  - The six regression tasks held at or near 100% for every model; no change made any model worse.
  - The "not available" fix: with the old docs, DeepSeek and GLM told the user to configure the unavailable ClientTrafficPolicy. With the fix they don't.
  - The parked-domain question went from failing to 2 of 2 for Haiku, Luna and DeepSeek. GLM already got it right.
  - Literal names on the answer page (round two):
    - DeepSeek: "Programmed but not serving" fell from 16.5 to 11 turns, and the client-IP question from 20 to 10.5.
    - GLM: "Programmed but not serving" fell from 14.5 to 5.5 turns.
    - Streaming timeout: this one went the wrong way for DeepSeek (12 to 16 turns). With two runs per cell, treat it as noise.

  Strong agents need little help; weak ones need precision

  Luna barely needed the fixes for speed. It answered almost every task in 3–4 turns and about 13 seconds, with old docs or new. A strong model reads a few pages, connects the dots and stops; extra signposts save it almost nothing. None  
  of the fixes cost it anything measurable either: per-task turn counts moved by half a turn or so in either direction, within noise.

  The weaker models gained the most, in time and turns. With the old docs, DeepSeek averaged 16 turns and 2 minutes per question. It took 27 turns to give up on the client-IP question and 26.5 on the streaming-timeout question, because   
  nothing told it to stop looking. Weak models search by literal word and keep going while the answer is missing, so three kinds of precision helped them:
  - Explicit gaps: a line on the limitations page saying "not documented yet" or "not available" lets the model stop searching.
  - Literal names: the condition, header and field names a user actually sees (Programmed=True, X-Forwarded-For), written on the page that answers about them.
  - No confident wrong guidance: with the old Client Traffic skill page, DeepSeek and GLM told the user to configure a resource that doesn't exist for them.

  But strong models still need precision for correctness. With the old docs, Luna answered the parked-domain question "Yes, retrieve the token from status.verification.httpToken". That's quick and confident, but wrong for a parked domain.
  When the docs are silent, a strong model fills the gap with a plausible inference rather than searching longer. So precision buys a weak model time and turns, and buys a strong model correctness.

  These fixes are cheap. Each was a sentence or a description, needed no new pages, and made no model worse. That fits the earlier llms.txt finding: precise, literal, honest text works for the whole range of models, which is why it's     
  worth doing.

[dlotterman]

This is the second of hopefully only two "re-org" PRs, the first was #81 .

This PR does a couple of things:

1. Updates the "llms.txt" stuff inside this repo / mintlify
2. Takes the layout wins from docs: re-org early draft #81 and adds LLM helpers and cookie crumbs
3. Includes a couple of other minitlify opened PR suggestions

In an effort to avoid making changes for changes sake, I am try to figure out how to measure these changes as much as possible, so I've built a little local test suite that drives subagents through a series of tests related to docs understanding.

In this table, the base-6fe37d8 column is the docs repo pre the re-org commit, main is the docs repo as is, current is the commits in this PR.

  ┌──────────────────────────┬──────────────┬──────┬─────────┬──────────────────────────────────────────┐
  │          Agent           │ base-6fe37d8 │ main │ current │ Harness failures (base / main / current) │
  ├──────────────────────────┼──────────────┼──────┼─────────┼──────────────────────────────────────────┤
  │ Haiku (claude-haiku-5-5) │ 84%          │ 99%  │ 99%     │ 0 / 0 / 0                                │
  ├──────────────────────────┼──────────────┼──────┼─────────┼──────────────────────────────────────────┤
  │ gpt-5.6-luna             │ 79%          │ 92%  │ 94%     │ 0 / 0 / 0                                │
  ├──────────────────────────┼──────────────┼──────┼─────────┼──────────────────────────────────────────┤
  │ DeepSeek                 │ 87%          │ 96%  │ 99%     │ 2 / 0 / 0                                │
  ├──────────────────────────┼──────────────┼──────┼─────────┼──────────────────────────────────────────┤
  │ GLM                      │ 88%          │ 97%  │ 100%    │ 5 / 8 / 1                                │
  └──────────────────────────┴──────────────┴──────┴─────────┴──────────────────────────────────────────┘

The main wins of the changes in this repo are:

1. Helping underpowered agents not get lost by providing bread crumbs and keyword short-circuits
2. Helping capable and confident but WRONG (gpt-5.6-luna) agents turn around quickly

From what I can tell, there is real quantitative improvement in the "underpowered" agent experience with the docs repo with the contents here.

RFR / RFM

[jacobsmith928]

Excellent.

[dlotterman]

Also just a note that I tried to cover the still relevant changes in #52 as well.