[Cedric Vidal (cedricvidal)]

Summary

Add Taxonome, a standalone toolkit for evaluating agent trajectories against YAML-authored taxonomies:

• Reusable engine and catalog: taxonomy-engine provides loading, validation and execution; ax-taxonomy supplies the agent-behavior catalog and deterministic handlers; the taxonome CLI composes both. Custom deterministic handlers can be registered through explicit plugins.
• Typed, grounded assessments: distinguish observed, negative, indeterminate and inapplicable results; validate occurrence grouping, evidence roles, cardinality, JSONPaths, types and ordering against the input trajectory.
• Strict ATIF ingestion: validate and upcast ATIF v1.6–v1.8, retain original-source provenance separately from canonical hashes, and safely ingest existing Scope run archives.
• Ax evaluation and optimization: run prompt-based elements through the Copilot SDK with one bounded Ax correction; evaluate curated datasets, optimize prompts and replay compatible artifacts without optimizer repair loops.
• CLI and readouts: support single inputs, newline/NUL file lists, mixed plain/archive batches, bounded concurrency, resumable results, cancellation and terminal/plain progress. Generate deterministic JSON/Markdown readouts with validated evidence and explicit coverage.

The shipped catalog includes prompt-based Skipped Invocation, Wrong Discovery and Content Misused, plus deterministic Discovered, Malformed Invocation and Timeout. Taxonome runs without Scope API, Judge, database, queue or token-manager services; deterministic evaluation needs no model credentials. Production processing/report integration is not included.

Demos

Before

N/A: new standalone Taxonome CLI.

After

Earlier recording from #1529 — not newly recorded or re-executed for this PR:

terminal.mp4

21-beat walkthrough covering deterministic contracts, plugins, plain/archive provenance, prompt evaluation/optimization/replay, and readouts. Paid-call accounting and command versions belong to that earlier recording, not this PR's validation. No new inference, optimization or recording was performed.

Testing

All applicable CI checks pass for code head 2ef7ea8d5069e8d782c66886c515bdda8ed0001f: CI run includes application and Windows builds, Node 22 unit tests, lint, headers/NOTICE, CLI bundle integration, both ACP worker integrations and queue recovery. Non-applicable LLM/Gateway jobs were skipped.

Command / check	Result
pnpm exec vitest run packages/taxonomy-engine/src packages/ax-taxonomy/src apps/taxonome/src --reporter=default --testTimeout=15000	Passed: 438 tests / 42 files, including ATIF/archive conformance, hashes, typed evidence, file-list/process contracts and mocked Ax correction behavior.
pnpm --filter shared --filter telemetry --filter taxonomy-engine --filter ax-taxonomy --filter taxonome --filter taxonome-custom-handler-example build	Passed: all six packages compile.
pnpm --filter taxonomy-engine lint && pnpm --filter ax-taxonomy lint && pnpm --filter taxonome lint	Passed: strict type checks.
pnpm --filter taxonome-custom-handler-example test:handler	Passed: 2 tests.
pnpm --filter static-prompt-evals test:ts	Passed: 37 tests / 7 files, including production prompt-composition contracts.
pnpm --filter static-prompt-evals validate-data	Passed: 222 approved cases, 15 quality targets and 3 standalone SDK targets.
pnpm --filter static-prompt-evals typecheck	Passed.
pnpm install --frozen-lockfile --ignore-scripts	Passed.
pnpm notice and pnpm notice:check	Passed; notices are current.
Built-CLI file-list/TTY/archive/resume/readout acceptance harness	Passed 30/30 scenarios with 10 copied playbook trajectories, 30 primary deterministic assessments and zero model calls. Covers newline/NUL lists, whitespace/Unicode paths, aliases, distinct input origins, partial errors, overlapping archives, archive root input, resume, readout coverage and cursor restoration.

These are existing validation results; no tests, model calls or recordings were rerun for this description update. Committed model outputs and optimization evidence retain their historical provenance and are not new accuracy claims.

Documentation and compatibility

Taxonome architecture and usage, the normative trajectory taxonomy format, package READMEs and a runnable custom-handler example document authoring, APIs, commands and evidence contracts. Package architecture and the static-prompt inventory are updated.

This is an experimental authored-YAML interface. run, eval and optimize require a taxonomy; v2 optimization artifacts explicitly reject historical v1 artifacts. Plugins are opt-in and run with process permissions, not in a sandbox. Archive validation reuses the existing run-state schema through the narrow shared/schemas/request export. No database migration or existing Scope CLI behavior change is required.

Checklist

• If Portal features changed, keep CLI capabilities in sync. N/A: no Portal changes.
• If Portal components changed, update their Storybook stories. N/A.
• If database changes require a migration, include up() / down() and keep it CosmosDB-compatible. N/A: no database changes.
• If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed.
• Video showing the behavior before the suggested change. N/A: new standalone CLI.
• Video showing the behavior after the suggested change. Earlier recording linked and attributed above.

[github-actions]

Test Results (Node.js 22)

test: Run #429

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3727	3727	0	0	0	0	0	1m30s

🎉 All tests passed!

Github Test Reporter