Skip to content

Expand catalog to 2,937 entries and make counts self-checking - #5

Merged
SeanShaffer88 merged 1 commit into
mainfrom
arena/c1a15d05-forai
Nov 11, 2026
Merged

SeanShaffer88 merged 1 commit into
mainfrom
arena/c1a15d05-forai

Conversation

@SeanShaffer88

Copy link
Copy Markdown
Owner

What this does

Grows the catalog from 2,202 → 2,937 entries (2,931 distinct URLs, 101 categories, 13 domains) and restructures the repo so an AI can be pointed at it and reach the right few files fast — plus so future additions are mechanical rather than manual.

1. Structure for AI consumption

Change Why
index/catalog-stats.json (new, generated) One canonical source for every count, rollup, and file size. Human docs quote it instead of hard-coding numbers.
scripts/catalog_stats.py (new) --write regenerates the stats file and rewrites the stats block in README.md, AI_INDEX.md, llms.txt; --check fails on drift.
validate.py now gates counts Fails if catalog-stats.json is stale or a generated stats block drifts. Count errors can no longer ship silently.
README.md rewritten Reading-path table ("what you want → file → size"), corrected numbers, all six expansion catalogs, full regeneration order, evidence/provenance section.
AI_INDEX.md rewritten One page: read order, all 13 domains with real counts, and a table of every machine-readable entry point.
AGENTS.md, index/README.md, CONTRIBUTING.md, generated nav Consistent at 13 domains / 101 categories / ~35% compact ratio; the contribution guide now documents the discovery sweep and dedupe rules.

Entry points stay stable and small-first: AI_INDEX.md (3 KB) → index/by-category/<id>.json (1–30 KB) → index/resources.compact.json (1.2 MB, 35% of canonical) → resources.json (3.3 MB) → llms-full.txt (1.8 MB, one-shot).

2. Two dated expansions

expansion-2026-11-10 — 82 entries. Gap analysis of project entry points: SemVer, Keep a Changelog, Conventional Commits, Choose a License, SPDX, REUSE, ADRs, Diátaxis, Contributor Covenant, Twelve-Factor, C4 (new category Software engineering foundations and standards), plus Cypress/Selenium/WebdriverIO/Jest/Robot Framework, HTTP clients and mock servers, editors, package managers, and observability/BI tools.

expansion-2026-11-11 — 653 entries. The discovery channels the request named:

  • 160 GitHub topics via gh api search/repositories (ai-agents, llm, llms, ai-prompt, ai-prompts, mcp, mcp-server, reverse-engineering, vibe-coding, ai-skills, skills, copilot, claude, codex, model-context-protocol, agent-skills, llm-evaluation, guardrails, llmops, plus 140 more), ranked by stars with a 100-star floor.
  • gittrend.io/hidden-gems — today/week/month boards.
  • awesomehub.js.org — topic indexes (cli-apps, selfhosted, go, python, rust, github-actions, …).

Candidates were deduplicated against the catalog by GitHub owner/repo identity, normalized URL, and repository name, then hand-reviewed per category; archived, forked, off-topic, and unlicensed-where-material repos were dropped (the few dual-use or copyleft entries carry the trade-off inline).

Families that previously had zero entries and now have coverage: AI agent skills/memory/MCP clients/guardrails, reverse-engineering tooling, ORMs and database servers, message queues and workflow engines, OpenTelemetry SDKs and log pipelines, i18n libraries, OCR/document extraction, rich-text and note-taking engines, editor/plugin ecosystems, .NET application blocks, Android libraries.

3. Evidence and limits

Per-entry evidence in docs/expansion-2026-11-11-checks.json: GitHub repository ID, detected license (SPDX), description, stars at check time, archived/fork flags, last push, and the catalog topics that surfaced it. Every new record carries source, checked_on, review scope, and verification_level.

This is metadata and reachability evidence only — no README review, release/maintenance audit, licensing opinion, security assessment, compatibility test, or adoption recommendation. No package was installed or executed.

Verification

python scripts/validate.py              → Validated 2937 records, 101 categories, stable IDs/facets/relationships,
                                          generated indexes, CSV/JSON agreement, and local Markdown links.
python scripts/catalog_stats.py --check → current
python -m unittest discover -s tests    → Ran 7 tests ... OK

The 2026-11-10 batch had been sitting uncommitted; this PR includes it. Existing records keep their IDs, facets, and related-link symmetry (verified against git show HEAD:resources.json).

Follow-ups (not in this PR)

  • The Dev Resource Explorer snapshot still imports commit 9f4d7c8a (2,202 entries); it needs a re-import to show the new batch.
  • Further sweeps of non-English ecosystems and awesome-* lists are possible with the recipe now documented in CONTRIBUTING.md.

Adds two dated expansions and rebuilds the repository around stable,
machine-readable entry points:

- 2026-11-10 (82 entries): software engineering foundations and standards
  (SemVer, Keep a Changelog, Conventional Commits, licensing, ADRs,
  Diataxis, codes of conduct), mainstream browser/unit test frameworks,
  HTTP clients and mock servers, editors, package managers, and
  observability/BI tools.
- 2026-11-11 (653 entries): discovered from 160 GitHub topics, gittrend.io
  hidden gems, and awesomehub.js.org topic indexes. Fills whole families
  that were missing: AI agent skills, memory, MCP servers and clients,
  guardrails and red teaming, reverse-engineering tooling, ORMs and
  database servers, message queues and workflow engines, OpenTelemetry
  SDKs, i18n, OCR/document extraction, editor ecosystems, .NET
  application blocks, and Android libraries.

Structure for AI consumers:
- index/catalog-stats.json is the single generated source of counts,
  rollups, and sizes; scripts/catalog_stats.py --write regenerates it and
  rewrites the stats block in README.md, AI_INDEX.md, and llms.txt.
  validate.py now fails when that block or the JSON drifts.
- README rewritten with a reading-path table (how big each entry point is
  and when to use it), the corrected 2,937/101/13 numbers, and the new
  expansion catalogs. AI_INDEX.md lists every machine-readable entry
  point and the 13 domains with real counts. AGENTS.md, index/README.md,
  CONTRIBUTING.md, and generated navigation are consistent with 13
  domains and 101 categories.
- CONTRIBUTING.md documents the repeatable topic-sweep discovery recipe
  and the dedupe rules used for the new batch.

Evidence: each new record records source, check date, review scope, and
verification level; docs/expansion-2026-11-11-checks.json holds per-entry
repository IDs, GitHub-detected licenses, topics, and last-push times.
Metadata-only, not adoption audits.

scripts/validate.py: 2937 records, 101 categories, CSV/JSON agreement,
generated indexes, local links all pass. Unit tests: 7 passing.

Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
@SeanShaffer88
SeanShaffer88 merged commit 4ccc49a into main Nov 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant