Expand catalog to 2,937 entries and make counts self-checking

AGENTS.md

@@ -4,10 +4,10 @@ Use this repository as a **curated pointer library** when answering software-dev
 
 ## Start here — smallest first
 
-1. Read `AI_INDEX.md`: it maps 12 broad domains to the categories worth opening and explains the file layout in one page.
+1. Read `AI_INDEX.md`: it maps 13 broad domains to the categories worth opening and explains the file layout in one page.
 2. Open only what the task needs: one category page in `categories/`, or its machine-readable shard `index/by-category/<category_id>.json`. `index/category-groups.json` lists which categories belong to each domain.
-3. For machine-readable lookup, search `resources.json` or run `python scripts/search.py "your query"`. Search ranks names and descriptive/facet fields; it intentionally ignores boilerplate review notes, link statuses, and collection provenance. Narrow results with `--topic`, `--type`, `--use-case`, `--language`, `--platform`, or `--category`. `index/resources.compact.json` is the full catalog at ~38% of the size (shortlisting fields only); `resources.csv` is the matching tabular export; `llms.txt` is a compact navigation index.
-4. Read a linked topic-specific catalog in `catalogs/` only when you need its extra provenance and cautions. For design research, see `catalogs/design-inspiration-resources.md` and `categories/ui-design-inspiration-and-reference.md`. The two 2026-10-27 development expansions are `catalogs/development-expansion-2026-10-27.md` and `catalogs/development-expansion-2026-10-27-round-2.md`; each has a linked metadata snapshot. Both snapshots are metadata-only, not adoption audits.
+3. For machine-readable lookup, search `resources.json` or run `python scripts/search.py "your query"`. Search ranks names and descriptive/facet fields; it intentionally ignores boilerplate review notes, link statuses, and collection provenance. Narrow results with `--topic`, `--type`, `--use-case`, `--language`, `--platform`, or `--category`. `index/resources.compact.json` is the full catalog at ~35% of the size (shortlisting fields only); `resources.csv` is the matching tabular export; `llms.txt` is a compact navigation index.
+4. Read a linked topic-specific catalog in `catalogs/` only when you need its extra provenance and cautions. For design research, see `catalogs/design-inspiration-resources.md` and `categories/ui-design-inspiration-and-reference.md`. The 2026-10-27 development expansions are `catalogs/development-expansion-2026-10-27.md` and `catalogs/development-expansion-2026-10-27-round-2.md`; each has a linked metadata snapshot. Both snapshots are metadata-only, not adoption audits. Later dated batches are `catalogs/expansion-2026-11-10.md` (foundations and everyday tooling) and `catalogs/expansion-2026-11-11.md` (GitHub topics, gittrend.io/hidden-gems, awesomehub.js.org).
 
 ## Selection workflow
 
@@ -23,7 +23,7 @@ Use this repository as a **curated pointer library** when answering software-dev
 
 ## Repository data and contribution conventions
 
-The canonical catalog is `resources.json`; `resources.csv` is a lossless, derived export, and `index/resources.compact.json`, `index/by-category/`, and `index/category-groups.json` are derived AI views. The current record contract is documented in `index/resource-schema.json`, with discovery facets in `index/taxonomy.json`. Stable `resource_id` and `category_id` values identify records and categories; topic, language, platform, use-case, audience, and resource-type facets are conservative discovery hints, not guarantees. `verification_level` describes recorded evidence, not present-day quality. Keep the catalog, export, category pages, `index/categories.json`, README counts, and `llms.txt` in sync; regenerate CSV and navigation with `python scripts/export_csv.py` and `python scripts/generate_indexes.py`, and the AI views with `python scripts/generate_ai_views.py`. Run `python scripts/validate.py` and `python -m unittest discover -s tests -v`. Use `CONTRIBUTING.md` for contribution rules.
+The canonical catalog is `resources.json`; `resources.csv` is a lossless, derived export, and `index/resources.compact.json`, `index/by-category/`, and `index/category-groups.json` are derived AI views. The current record contract is documented in `index/resource-schema.json`, with discovery facets in `index/taxonomy.json`. Stable `resource_id` and `category_id` values identify records and categories; topic, language, platform, use-case, audience, and resource-type facets are conservative discovery hints, not guarantees. `verification_level` describes recorded evidence, not present-day quality. Keep the catalog, export, category pages, `index/categories.json`, README counts, and `llms.txt` in sync; regenerate CSV and navigation with `python scripts/export_csv.py` and `python scripts/generate_indexes.py`, the AI views with `python scripts/generate_ai_views.py`, and the shared counts with `python scripts/catalog_stats.py --write` (which also rewrites the generated stats block in `README.md`, `AI_INDEX.md`, and `llms.txt`). Never type a count by hand; `python scripts/catalog_stats.py --check` fails when a quoted count drifts. Run `python scripts/validate.py` and `python -m unittest discover -s tests -v`. Use `CONTRIBUTING.md` for contribution rules.
 
 Record what was actually checked. A user-supplied URL is not independently verified: mark it as such, leave `http_status` empty, and avoid implying a license, current feature set, or service availability. Preserve source and review caveats. Linked resources retain their own terms and licenses; this repository contains references and original descriptions, not copied third-party documentation or project code.
 

AI_INDEX.md

@@ -1,48 +1,71 @@
 # AI index — fastest path through this repository
 
-This repository is a curated pointer library: 2,202 entries, 2,198 distinct
-URLs, 97 categories. It contains references and original notes, not
-third-party code. Entries are discovery hints, not endorsements — verify
+This repository is a curated pointer library: 2,937 entries, 2,931 distinct
+URLs, 101 categories, 13 domains. It contains references and original notes,
+not third-party code. Entries are discovery hints, not endorsements — verify
 maintenance, license, and compatibility at the primary source.
 
+<!-- BEGIN GENERATED STATS -->
+**2,937 catalog entries** · **2,931 distinct URLs** · **101 categories** · **13 domains** · **2,328 project identities**
+<!-- END GENERATED STATS -->
+
 ## Read in this order (smallest first)
 
 1. **This file** — pick a domain below.
-2. **`index/category-groups.json`** — the 97 categories rolled up into
-   12 domains, with counts and category IDs.
+2. **`index/category-groups.json`** — the 101 categories rolled up into
+   13 domains, with counts and category IDs.
 3. **One category shard** — `index/by-category/<category_id>.json` holds
    only that category's records (name, URL, purpose, facets). Most shards
    are a few KB; load one, not the whole catalog.
 4. **Only if you need provenance** — `resources.json` is the canonical
-   full catalog (2.3 MB, adds source, check date, review notes per record).
-   `index/resources.compact.json` is the same catalog at ~38% of the size
+   full catalog (3.3 MB, adds source, check date, review notes per record).
+   `index/resources.compact.json` is the same catalog at ~35% of the size
    when you genuinely need every category at once.
 
 Single-file alternative: `llms-full.txt` concatenates every category page and catalog for one-shot ingestion.
+Counts and rollups live in `index/catalog-stats.json`; it is generated, so
+quote it instead of counting records yourself.
 Local alternative: `python scripts/search.py "your query"` with
-`--topic`, `--type`, `--use-case`, `--language`, or `--platform` filters.
+`--topic`, `--type`, `--use-case`, `--language`, `--platform`, or `--category`.
 
 ## Domains
 
 | Domain | Resources | Start with |
 |---|---|---|
-| ai | 343 | `ai-mcp-servers-and-tool-use`, `ai-machine-learning-and-local-inference`, `ai-agent-frameworks-and-orchestration` |
-| windows | 232 | `windows-native-libraries`, `windows-diagnostics`, `windows-desktop-ui` (47 small Windows categories — most hold under 10 entries) |
-| languages-runtimes | 328 | `python-rust-and-go-building-blocks`, `languages-and-runtimes`, `niche-language-ecosystems` |
-| web-api-architecture | 245 | `api-contracts-clients-and-networking`, `web-interfaces-and-application-frameworks` |
-| cloud-infra-devops | 211 | `infrastructure-as-code-and-cloud-native-operations`, `cloud-infrastructure-and-deployment` |
-| security | 143 | `security-identity-and-supply-chain`, `security-testing-defense-and-supply-chain-assurance` |
-| data | 141 | `databases-data-access-and-analytics`, `data-science-and-engineering-tools` |
-| design-ux | 125 | `design-systems-and-design-engineering`, `ui-design-inspiration-and-reference` |
-| media-games-docs | 125 | `game-creation-assets-and-automation`, `documents-ocr-and-file-formats`, `world-of-warcraft-addon-development` |
-| desktop-mobile | 121 | `android-apk-development-and-analysis`, `desktop-mobile-and-device-sdks` |
-| testing-quality | 109 | `test-automation-and-quality-engineering`, `testing-verification-and-performance` |
-| operations-tools | 79 | `observability-debugging-and-operations`, `backup-storage-transfer-and-resilience` |
+| ai | 431 | `ai-mcp-servers-and-tool-use`, `ai-machine-learning-and-local-inference`, `ai-agent-frameworks-and-orchestration` |
+| windows | 249 | `windows-native-libraries`, `windows-diagnostics`, `windows-desktop-ui` (46 small Windows categories — most hold under 10 entries) |
+| languages-runtimes | 360 | `python-rust-and-go-building-blocks`, `languages-and-runtimes`, `niche-language-ecosystems` |
+| web-api-architecture | 307 | `api-contracts-clients-and-networking`, `web-interfaces-and-application-frameworks` |
+| cloud-infra-devops | 305 | `infrastructure-as-code-and-cloud-native-operations`, `cloud-infrastructure-and-deployment` |
+| media-games-docs | 234 | `game-creation-assets-and-automation`, `graphics-multimedia-scientific-and-geospatial`, `documents-ocr-and-file-formats` |
+| data | 222 | `databases-data-access-and-analytics`, `data-science-and-engineering-tools` |
+| desktop-mobile | 184 | `desktop-mobile-and-device-sdks`, `android-apk-development-and-analysis` |
+| operations-tools | 175 | `observability-debugging-and-operations`, `search-messaging-and-durable-work` |
+| design-ux | 162 | `design-systems-and-design-engineering`, `ui-design-inspiration-and-reference` |
+| security | 154 | `security-identity-and-supply-chain`, `security-testing-defense-and-supply-chain-assurance` |
+| testing-quality | 135 | `test-automation-and-quality-engineering`, `testing-verification-and-performance` |
+| foundations | 19 | `software-engineering-foundations-and-standards`, `standards-and-discovery` |
 
 Full category list with counts: `categories/README.md` or `llms.txt`.
 Human-readable category pages: `categories/<category_id>.md`.
 Long-form catalogs with provenance and cautions: `catalogs/`.
 
+## Machine-readable entry points
+
+| File | What it is |
+|---|---|
+| `resources.json` | canonical records: identity, purpose, source, check date, review scope, facets |
+| `resources.csv` | lossless tabular export of the same records |
+| `index/resources.compact.json` | every record, shortlisting fields only (~35% of canonical size) |
+| `index/by-category/<category_id>.json` | one category's compact records |
+| `index/categories.json` | category IDs, titles, counts, paths, membership |
+| `index/category-groups.json` | the 13 domains with counts and category IDs |
+| `index/catalog-stats.json` | generated counts, rollups, source and type breakdowns |
+| `index/resource-schema.json` | the record contract |
+| `index/taxonomy.json` | the controlled facet vocabulary |
+| `llms.txt` | compact navigation index |
+| `llms-full.txt` | every category page and catalog in one file |
+
 ## Do not use
 
 `archive/legacy-template/` holds generated placeholder manifests and an

CONTRIBUTING.md

@@ -2,10 +2,26 @@
 
 Add resources that solve a concrete development need. Prefer canonical upstream links and concise original descriptions: what it is useful for, where it fits, and meaningful constraints. Inclusion is not an endorsement or adoption audit.
 
-Update the appropriate full catalog and `resources.json`, preserving each existing record and its provenance. For new records, provide the original evidence and descriptions; generate stable IDs and conservative derived facets with `python scripts/enrich_resources.py --apply`, then regenerate `resources.csv` with `python scripts/export_csv.py`, navigation with `python scripts/generate_indexes.py`, and the AI views (compact catalog, category shards, domain rollup) with `python scripts/generate_ai_views.py`. Use `index/resource-schema.json` and `index/taxonomy.json` as the active metadata contract. Do not assign unsupported language/platform claims; leave uncertain facets empty or flag them for review. `scripts/enrich_resources.py --check` verifies the derived metadata is present; the explicit `--refresh-topics`, `--refresh-types`, and `--refresh-verification` modes recalculate only those derived fields when their rules change. Preserve different interfaces to the same project when they serve different purposes; do not count raw entries as unique projects. Category page names should be lowercase hyphenated names.
+Update the appropriate full catalog and `resources.json`, preserving each existing record and its provenance. For new records, provide the original evidence and descriptions; generate stable IDs and conservative derived facets with `python scripts/enrich_resources.py --apply`, then regenerate `resources.csv` with `python scripts/export_csv.py`, navigation with `python scripts/generate_indexes.py`, and the AI views (compact catalog, category shards, domain rollup) with `python scripts/generate_ai_views.py`, and the shared counts with `python scripts/catalog_stats.py --write`. Use `index/resource-schema.json` and `index/taxonomy.json` as the active metadata contract. Do not assign unsupported language/platform claims; leave uncertain facets empty or flag them for review. `scripts/enrich_resources.py --check` verifies the derived metadata is present; the explicit `--refresh-topics`, `--refresh-types`, and `--refresh-verification` modes recalculate only those derived fields when their rules change. Preserve different interfaces to the same project when they serve different purposes; do not count raw entries as unique projects. Category page names should be lowercase hyphenated names.
 
 Record the date, source, and scope of actual verification. A successful HTTP request only verifies reachability. Label archived, dormant, unmaintained, mirrored, or service-bound resources honestly and cite upstream evidence. Describe a potential revival separately from a tested release. Keep existing WR task IDs stable; the included backlog is a dated snapshot, not a promise of ongoing project work.
 
+### Finding new candidates
+
+Topic sweeps are repeatable and cheap:
+
+```sh
+gh api -X GET search/repositories -f 'q=topic:mcp-server stars:>=200 archived:false fork:false' \
+  -f sort=stars -f per_page=50 --jq '.items[] | "\(.stargazers_count)\t\(.full_name)\t\(.license.spdx_id)"'
+python3 scripts/search.py "model context protocol" --topic mcp     # check what is already covered
+```
+
+Also useful: `https://gittrend.io/hidden-gems` (repositories under 2,000 stars with
+strong engagement) and the topic indexes at `https://awesomehub.js.org/`. Before
+adding a candidate, deduplicate on GitHub `owner/repository` identity, normalized
+URL, and repository name, and drop archived, forked, or unlicensed repositories
+unless there is a specific reason to keep them with an inline license caveat.
+
 Do not include credentials, private account information, local machine paths, copied upstream documentation, or dependencies themselves. Linked projects keep their own licenses.
 
-Run `python scripts/validate.py` before proposing a change. It checks index agreement, required fields, URL syntax, category coverage, local Markdown links, and accidental local paths. It does not perform network or adoption audits.
+Run `python scripts/validate.py` before proposing a change. It checks index agreement, required fields, URL syntax, category coverage, local Markdown links, accidental local paths, the generated `index/catalog-stats.json`, and the generated count block in `README.md`, `AI_INDEX.md`, and `llms.txt`. It does not perform network or adoption audits.