[ci] Nightly test plan and scheduled workflow #200

Activity

1. hsliuustc0106

   added

   type:ciBuild, test and release automationBuild, test and release automation

   area:ciContinuous integration and checksContinuous integration and checks

   priority:p0Blocks the agreed release scopeBlocks the agreed release scope

   on Nov 7, 2026
2. hsliuustc0106 commented on Nov 7, 2026

   

   hsliuustc0106

   on Nov 7, 2026

   ContributorAuthor

   More actions

   Nightly plan, updated per discussion — e2e scenarios and perf/output verification are part of the nightly, and implemented in #203.

   The design principle: the nightly catches what the PR gate structurally cannot — a possibly-red main, a changed environment, and what a 30-minute PR budget must skip. Five lanes on main HEAD, cron 0 18 * * * UTC (02:00 Asia/Shanghai):

   Lane	Content	Why nightly
   1. Gate parity	Release checks re-run verbatim (shared reusable workflow with the PR gate)	Catches flaky merges landing on main — the timing-sensitive pytest batch already failed on main itself (run 42212456290, 2026-11-06) with no code cause — plus environment drift
   2. Python flake triage	Full pytest; on failure, rerun the failed subset once and classify pass/flaky/fail; 3× probe of a watchlist file for the 9 known timing-sensitive tests	With 9 known-flaky tests, a naive nightly goes red randomly and gets ignored. Reruns are reported (flaky), never silent (#17); the watchlist (scripts/nightly/flaky-watchlist.txt) measures flake rate so the tests get fixed
   3. Browser extended	All 35 tests/browser/*.mjs that the PR gate does not run, per-script results, one failure doesn't stop the rest	e2e scenarios that today run nowhere: node_versions, multi_selection_references, paste_undo, subject_library, reference_parity, first_last_frame_connections, …
   4. Browser production	Core journeys against the production build (vite preview of dist): locale switch (zh↔en), full timeline creation, assistant sessions/navigation, paste/undo, provider settings, cross-canvas paste	Today the built artifact is only type-checked and bundled; nobody serves and exercises it before tagging. Browser scripts that import /src/*.ts in-page stay dev-only by design
   5. Perf & outputs	Output verification (hard): new mp4_render_probe.mjs renders through the app's own WebCodecs pipeline, writes the MP4 to disk, ffprobe-verifies duration (~2 s), h264/aac streams, 640×360, ~48 frames. Perf (diagnostic): reference_drag_performance.mjs samples frame p95 / long tasks + CPU profiles	Per AGENTS.md, output correctness gets real assertions; perf gets comparable per-run measurement published as nightly-perf.json — no invented thresholds, regressions become visible over time

   Policy:

   • Release decision: cut the daily v0.1.x only when gate parity is green on the candidate SHA. Lanes 2–5 are advisory in v1 — failures are reported and tracked, they don't auto-block.
   • No auto-tagging on any outcome; no paid models; no multi-OS (unchanged scope).
   • Release checks moves to one reusable workflow (release-checks.yml) shared by ci.yml and the nightly — a single implementation, no drifting copy. Side benefit: its check steps now run independently, so a Python-test failure no longer short-circuits contracts/build/browser results.
   • Each run publishes nightly-summary.json (lane results + SHA) and nightly-perf.json; the failure-notification path to the OpenDirector room hooks onto those and stays with the Coordinator as in the deliverables.
   • Matrix + policy documented in docs/development/validation.md.

   Follow-ups after this lands: stabilize or explicitly quarantine the 9 watchlist tests before the first scheduled run so nightly doesn't open red; grow the production lane as more scripts are made preview-safe; lane regression thresholds for perf once we have a baseline series.

   Implementation: #203 (includes workflow, probe script, watchlist, docs). Out of scope unchanged: paid/real-provider generation, multi-OS installers, full #7 G0–G7.
3. hsliuustc0106

   added 2 commits that reference this issue on Nov 7, 2026

   [ci] Add nightly test plan and reusable release checks

   ...

   3c25230

   [ci] Adopt five-level test taxonomy with manifest-driven lanes

   ...

   1621d6c
4. hsliuustc0106 commented on Nov 7, 2026

   

   hsliuustc0106

   on Nov 7, 2026

   ContributorAuthor

   More actions

   Follow-up on the plan above: the nightly is now framed as a five-level test system (adopting the L1–L5 pattern from the vLLM-Omni testing docs — scope/cadence/budget per level), in the same PR:

   Level	Scope	Cadence	Budget
   L1 unit, static, build	frontend/Python unit suites, i18n, browser-manifest check, layer imports, contracts, production build	every PR	≤30 min
   L2 browser E2E with model stubs	core journeys on isolated services	every PR	same job
   L3 main-HEAD re-check + extended coverage	gate parity, flake triage, all PR-uncovered browser scripts	nightly + manual pre-release	gate parity = L1+L2 bar; extended ≤60 min
   L4 production build, perf, outputs	built-artifact journeys, ffprobe output assertions, diagnostic perf sampling	nightly + manual pre-release	production ≤45 min, perf ≤30 min
   L5 stability & recovery (planned)	kill-restart, project reopen, long-run stability	weekly + pre-release	TBD

   Key mechanics borrowed and adapted:

   1. One definition, multiple cadences — L3/L4 run nightly and re-run manually on a candidate SHA before release (vllm-omni reuses its nightly L4 for pre-release; our workflow_dispatch does the same).
   2. Declarative selection instead of hardcoded lists — tests/browser/manifest.json now declares every browser script's lane (gate/extended/extendedFixture/production/perf) and requirements (fixture, previewSafe, args); the nightly derives its lists from it, and a new Release checks step (npm run check:browser-manifest) enforces disk↔manifest↔workflow consistency. New scripts register in one place.
   3. Advisory-lane graduation rule — like vllm-omni's non-blocking AMD nightly slice: extended/production/perf lanes stay advisory while they accumulate stable-run evidence (~2 weeks green → Coordinator decides blocking).
   4. Failure-pattern knowledge in-repo — docs/development/ci-failures.md documents the 9 known timing-sensitive tests with triage steps, replacing tribal knowledge.
   5. Explicit per-level time budgets — guards against nightly creep (L4 production ≤45 min, extended ≤60 min, etc.).

   One correctness finding while building the manifest: main's recent timeline_workflow.mjs update added an in-page /src import, so it no longer runs under vite preview — the production lane dropped it automatically via the manifest check rather than failing at runtime.

   Commit 2 of PR #203; Release checks re-running on it now.
5. cursor

   added 2 commits that reference this issue on Nov 7, 2026

   [ci] Add nightly test plan and reusable release checks

   ...

   188c0f3

   [ci] Adopt five-level test taxonomy with manifest-driven lanes

   ...

   5a1b3c7
6. hsliuustc0106

   added 3 commits that reference this issue on Nov 7, 2026

   [ci] Add nightly test plan and reusable release checks

   ...

   42f7d49

   [ci] Adopt five-level test taxonomy with manifest-driven lanes

   ...

   8d2e094

   [ci] Nightly test plan and scheduled workflow (#203)

   ...

   8f013a4
7. cursor

   added 2 commits that reference this issue on Nov 10, 2026

   [ci] browser: harden nightly advisory lanes

   ...

   ce83afa

   [ci] browser: harden nightly advisory lanes

   ...

   986834b
8. hsliuustc0106

   added a commit that references this issue on Nov 11, 2026

   Harden nightly advisory browser and perf lanes (#249)

   ...

   a9d02b6