[hsliuustc0106]

What

Implements the nightly test plan from #200: a scheduled Nightly checks workflow on main HEAD plus the in-repo documentation of the matrix and pass/fail policy.

Lanes

Lane	Content	Purpose
Gate parity	The full Release checks, re-run verbatim on main HEAD via a reusable workflow	Release-decision signal: catches flaky merges landing on main (e.g. the timing-sensitive batch that failed on main itself on 2026-11-06) and environment drift a PR cannot show
Python flake triage	Full pytest; on failure, rerun the failed subset once (--lf) and classify pass / flaky / fail; plus a 3× probe of the known-flaky watchlist	Flake rate stays visible (#17), never silently retried; data to fix the 9 known tests
Browser extended	All 37 tests/browser/*.mjs that the PR gate does not run, per-script pass/fail, one failure does not stop the rest; scripts needing --assistant-fixture run against their own fixture services, mirroring the Release checks phase	Closes the coverage gap: e.g. node_versions, multi_selection_references, paste_undo, subject_library, reference_parity never ran anywhere
Browser production	Core journeys served from the production build (vite preview of dist): locale switch, full timeline creation, assistant sessions, paste/undo, provider settings…	The artifact a release ships is exercised, not just type-checked; preview-safe scripts only (no in-page /src imports)
Perf & outputs	mp4_render_probe.mjs renders an MP4 through the app pipeline, writes it to disk and verifies with ffprobe (duration, h264/aac streams, 640×360, ~48 frames); reference_drag_performance.mjs samples frame p95 / long tasks / CPU profile	Output verification is a hard assertion; perf numbers are diagnostic sampling published per run for comparison (no invented thresholds)

Aggregated summary job writes the lane table + machine-readable nightly-summary.json artifact. Cron 0 18 * * * UTC ≈ 02:00 Asia/Shanghai.

Policy

• Release rule: cut the daily v0.1.x only when gate parity is green on the candidate SHA. Extended/production/perf lanes are advisory in v1 — failures are reported and tracked, they do not auto-block.
• No auto-tagging, no paid models, no multi-OS matrix (per [ci] Nightly test plan and scheduled workflow #200 scope).
• Release checks now lives in one reusable workflow (release-checks.yml) shared by ci.yml and the nightly — no second drifting copy. Check steps after dependency install run independently (if: always() && install ok), so a Python-test failure no longer short-circuits contracts/build/browser results — this also fixes the PR-gate behavior noted in [ci] Nightly test plan and scheduled workflow #200's context.

Deliverables from #200

• Nightly matrix + policy documented in docs/development/validation.md (Nightly 检查 section)
• schedule + workflow_dispatch workflow (nightly.yml), Release checks reusable
• Flake watchlist file scripts/nightly/flaky-watchlist.txt (9 known tests, editable without touching YAML)
• Failure notification wiring to the OpenDirector room — left to the Coordinator per [ci] Nightly test plan and scheduled workflow #200 (summary artifact + Actions summary are the machine-readable hook points)

Out of scope per #200: paid/real-provider generation, multi-OS installers, full #7 G0–G7.

Validation

• All three workflow YAMLs parse; node --check on the new probe script.
• Every watchlist pytest node id and every lane browser script verified to exist; production-lane scripts confirmed free of /src imports; vite.config preview.proxy confirmed to serve the same /api wiring as dev.
• Triage classification (pass/flaky/fail/skipped), perf aggregation and summary jq logic dry-ran locally with stub inputs.
• The PR run of this branch exercises the reusable Release checks live; scheduled dispatch can be verified after merge via workflow_dispatch.

Refs: #200

Five-level taxonomy (adopted from the vLLM-Omni-style system)

The checks are now structured as five documented levels (scope, cadence, budget, implementation per level, in docs/development/validation.md):

Level	Scope	Cadence	Budget
L1 unit, static, build	frontend/Python unit suites, i18n, browser-manifest check, layer imports, contracts, production build	every PR	≤30 min (whole job)
L2 browser E2E with model stubs	core journeys on isolated services (incl. auxiliary-panels fixture phase)	every PR	same job
L3 main-HEAD re-check + extended coverage	gate parity, Python flake triage, all PR-gate-uncovered browser scripts	nightly + manual	gate parity as L1+L2; extended ≤60 min
L4 production build, perf, outputs	built-artifact journeys (vite preview), ffprobe output assertions, diagnostic perf sampling	nightly + manual	production ≤45 min; perf ≤30 min
L5 stability & recovery (planned)	kill-restart, project reopen, long-run stability; needs authoring	weekly + pre-release	TBD

Also in this commit:

• tests/browser/manifest.json is now the single source of truth for browser-script lane membership (gate / extended / extendedFixture / production / perf) plus fixture / previewSafe requirements and per-script args; nightly derives its lane lists from the manifest — new scripts register in the manifest, not in workflow YAML.
• npm run check:browser-manifest (new Release checks step) enforces: disk ↔ manifest 1:1, fixture/previewSafe claims match source reality, and the manifest gate set equals what Release checks actually invokes.
• Advisory-lane graduation rule written down: a lane stays advisory until ~2 weeks of stable green evidence, then the Coordinator decides whether it becomes blocking.
• timeline_workflow.mjs left the production lane — main's recent update made it import /src in-page, so it is dev-server-only now.
• New docs/development/ci-failures.md: the 9 known timing-sensitive tests documented in-repo (patterns + flake-vs-regression triage), linked from the validation guide and docs index.