[emmavdh]

Pull Request Description

What and why?

site/scripts/generate_chatbot_product_map.py lists at most six related docs per route in site/llm/chatbot-product-map.md. The ranking scored candidates by the length of the matched keyword segment, but on the workflow routes every candidate matches the same workflows segment, so they all tied and alphabetical path order picked the six. Pages from other sections that only mention workflows in their filename (/faq/, /guide/integrations/, /guide/monitoring/, /guide/risk-tiering/) sorted ahead of most of /guide/workflows/, and /guide/workflows/manage-record-stages.html dropped out of the map entirely. Nothing reported what the cap left out.

This PR changes the ranking and makes evictions visible:

• Candidates are ranked by section before the cap is applied: pages in the same directory as the route's primary docs come first, then pages in a section named by a route keyword (for example /guide/workflows/ for /workflows), then pages that only match a keyword in their filename. Keyword specificity and then path break ties, so output stays deterministic.
• When a route has more candidates than the cap, the generator prints the dropped paths per route to stderr. The cap stays at six (now the RELATED_DOCS_CAP constant), and the second related[:6] slice in the render loop is gone.
• RELATED_DOC_KEYWORDS gains "tier": ["risk-tiering"]. Without it, section ranking put /guide/risk-tiering/manage-risk-tier-templates.html in the filename-only tier on /settings/risk-tier-templates and the route lost its own guide to document-template pages. /settings/risk-tier-stages had no related docs at all before.
• Regenerated site/llm/chatbot-product-map.md with python3 site/scripts/generate_chatbot_product_map.py (the CI command), with site/installation populated the way CI does it. The frontend snapshot is unchanged.

Before and after for the related docs on the three workflow routes:

Route	Before (main)	After
/settings/workflows, /settings/workflow-states, /workflows	/faq/faq-workflows.html /guide/integrations/integrations-examples/use-webhooks-with-workflows.html /guide/monitoring/trigger-workflows-on-threshold-breach.html /guide/risk-tiering/set-up-risk-tiering-workflows.html /guide/workflows/assign-workflow-executions.html /guide/workflows/conditional-step-requirements.html	/guide/workflows/assign-workflow-executions.html /guide/workflows/conditional-step-requirements.html /guide/workflows/configure-workflows.html /guide/workflows/introduction-to-workflows.html /guide/workflows/manage-record-stages.html /guide/workflows/manage-workflow-tasks.html

All three routes had the same related list before and after. Primary docs are unchanged.

Other routes that change: seven inventory settings routes swap /faq/faq-inventory.html for a /guide/inventory/ page, four templates and documents routes (/settings/block-library, /settings/document-types, /settings/documents/general, /settings/templates) swap /guide/risk-tiering/manage-risk-tier-templates.html for a /guide/templates/ page, /settings/risk-tier-templates and /settings/risk-tier-stages now list the five /guide/risk-tiering/ guides first, /settings/attestation-templates swaps /faq/faq-attestations.html for /guide/templates/customize-document-checker.html, and /attestations and /settings/attestations only reorder.

How to test

From the repo root:

python3 site/scripts/generate_chatbot_product_map.py
git diff --exit-code -- site/llm/chatbot-product-map.md
python3 -m unittest discover -s site/scripts -p 'test_generate_chatbot_product_map.py' -v

site/installation must be populated first (CI copies it from validmind/installation; locally make -C site clone copy-installation), otherwise the SSO installation page loses its section list in the regenerated map. The first command prints the drop report to stderr. The diff should be empty and all 16 tests should pass. New tests cover same-section pages outranking cross-section filename matches, keyword-section pages outranking filename matches on routes without primary docs, the drop report listing evicted paths, a regression check that the three workflow routes keep manage-record-stages.html, and a regression check that /settings/risk-tier-templates keeps its risk-tiering guides.

What needs special review?

• FAQ pages now lose their related slot on routes whose own section has six or more matching pages (faq-workflows, faq-inventory, and faq-attestations on /settings/attestation-templates). They show up in the drop report. If FAQs should always keep a slot, that is a follow-up rule.
• Inside a section, path order still decides. A new /guide/workflows/ page that sorts before manage-record-stages would push it out again; the drop report and the regression test will flag that.
• Section ranking only helps where a route keyword names the section. The risk-tiering gap above was one case; other sections without a keyword entry would show the same pattern, and the drop report is the place to spot them.
• The drop report is long today (20 routes), because the cap was already dropping many pages silently.

Dependencies, breaking changes, and deployment notes

None.

Release notes

Internal: no release notes.

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[juanmleng]

Review: #1600 at ee6e908 (merge-base 8ed2521)

Computed by the calibrated-review skill, run by @juanmleng.

Merge with recommendations · no blocking · 2 small fixes · 1 question for you

The ranking fix works: the three workflow routes keep manage-record-stages, and the new tier keyword matches only the two risk-tier routes.

Small fixes: F1 run the generator tests on guide-only PRs (today they're skipped, so the next eviction is auto-committed and fails a later, unrelated PR), F2 drop the brittle first-position assertion in the risk-tier test.

❓ @emmavdh — should new evictions show in the PR diff? Options: commit the dropped list as a sidecar file, or warn only when a route's dropped list changes.

Fix later: F4 alphabetical tiebreak inside a section.

_{calibrated-review v2 · review · run by @juanmleng · head ee6e908}

Agent handoff · calibrated-review v2 · review · ee6e908

calibrated-review: 2
mode: review
pr: validmind/documentation#1600
head: ee6e9086ef83e10bd04acac2cee4a301242f7645
merge_base: 8ed252152bf3f99e1ccb57d3dcb6638e5540968f
run_by: juanmleng
verdict: merge-with-recs
downgraded: 1 # F1 fails gate 2: the eviction needs a future guide PR
items:
  - id: F1
    class: fix-now
    type: correctness
    direction: risky
    title: Generator tests are skipped on guide-only PRs, so the regression test fires on the wrong PR
    scenario: >-
      A PR adds site/guide/workflows/approve-workflow-steps.qmd and
      archive-workflows.qmd. The preview selector marks it targeted, so the
      "Test chatbot product map generator" step is skipped. The map step
      regenerates the map without manage-record-stages on the three workflow
      routes and the bot auto-commits it. The PR merges green. The next PR that
      runs full validation fails test_workflow_routes_keep_manage_record_stages
      for an eviction it did not cause. The PR body says the regression test
      will flag this case.
    evidence:
      - .github/workflows/validate-docs-site.yaml:419 gates the test step on targeted != 'true'
      - .github/workflows/validate-docs-site.yaml:386 map step is ungated and auto-commits the regenerated map
      - .github/scripts/select_docs_preview_targets.py exits 0 (targeted) for "added site/guide/workflows/archive-workflows.qmd"
      - with the two pages added, test_workflow_routes_keep_manage_record_stages fails for all three routes
    fix: >-
      Remove the targeted gate from the test step (no render needed, ~0.1 s)
      and run it before the map auto-commit, so a failing test stops the
      commit. If that is out of scope, correct the PR body's claim.
    done_when: a guide-only PR runs the generator tests before the map is committed
  - id: F2
    class: fix-now
    type: hygiene
    direction: safe
    title: Risk-tier regression test pins the alphabetically first page
    scenario: >-
      Adding any risk-tiering guide whose name sorts before "configure-"
      (about-..., assign-...) fails
      test_risk_tier_templates_route_keeps_risk_tiering_guide although the
      route still lists its own guides.
    evidence:
      - site/scripts/test_generate_chatbot_product_map.py:178 assertEqual(related[0], ".../configure-risk-tier-calculation.html")
      - the assertIn on the line above already covers the docstring's intent
    fix: >-
      Delete the related[0] assertion, or assert that the leading entries are
      all under /guide/risk-tiering/.
    done_when: the test asserts membership or section, not alphabetical position
  - id: F3
    class: question
    type: risk-appetite
    direction: safe
    title: The drop report is not visible during review
    scenario: >-
      The report goes to stderr in a CI step that passes, and it lists all 20
      routes the cap already trims on every run. A new eviction looks the same
      as the standing ones, and a reviewer reading the PR diff does not see it.
      The story asked for the eviction to show during review.
    evidence:
      - site/scripts/generate_chatbot_product_map.py:795 prints the full report to stderr
      - regenerated at head, the report lists 20 routes
    fix: >-
      Owner @emmavdh decides: commit the dropped list as a sidecar file next to
      the map (kept out of the LLM corpus) so evictions appear in the diff, or
      emit a ::warning:: only for routes whose dropped list changed.
  - id: F4
    class: fix-later
    type: scope-expansion
    direction: safe
    title: Within a section, alphabetical order still picks the six
    scenario: >-
      On routes whose own section has more than six matching pages (workflows,
      inventory, attestation templates), path order decides what the assistant
      sees. The author notes this under special review.
    fix: >-
      Ticket: add a small per-route pin list or rank by keyword match in the
      page title, so the canonical how-to for a screen is chosen on purpose.
accepted_risks:
  - checked: removal of "route.related_docs or" in the render loop
    why: route.related_docs is only ever assigned from suggest_related_docs, so the output is unchanged
  - checked: the "tier" keyword is a substring match
    why: across the frontend snapshot it matches only /settings/risk-tier-templates and /settings/risk-tier-stages
  - checked: FAQ pages losing their slot on crowded routes
    why: disclosed by the author under special review, and it shows in the drop report
verification:
  - run: python3 -m unittest discover -s site/scripts -p 'test_generate_chatbot_product_map.py'
    result: 16 passed at head
  - run: python3 site/scripts/generate_chatbot_product_map.py
    result: one-line diff on the installation page; site/installation not populated locally
  - run: add approve-workflow-steps.qmd and archive-workflows.qmd under site/guide/workflows, rerun the tests
    result: 3 subtest failures in test_workflow_routes_keep_manage_record_stages
  - claim: "Before and after" route table (PR body), workflow rows
    result: matches the regenerated map
  - claim: "Before and after" route table (PR body), other route swaps
    result: NOT CHECKED
  - claim: "the drop report and the regression test will flag that" (PR body)
    result: false for guide-only PRs, see F1

[github-actions]

Validate docs site

✓ INFO: A live preview of the docs site is available — Open the preview

[github-actions]

Lighthouse check results

✓ INFO: No site pages to audit in this PR.

Commit SHA: ee6e908

[cachafla]

Checked head ee6e908: related docs are now ranked by section (the primary doc's own section first, then sections a keyword names, then docs that match only by filename) before the six-entry cap. Docs the cap drops are reported on stderr. The regenerated map looks right: manage-record-stages stays on the workflow routes, and risk-tiering guides are no longer listed under the document settings routes. CI is green, and all 16 tests in test_generate_chatbot_product_map pass locally.