[WIP] [roadmap] Q4 2026 CI: layered gates, nightly health and release evidence / 分层门禁、Nightly 健康度与发布证据 #273

Parent roadmap: #107. Proposed November–December 2026 delivery; owners and capacity require confirmation.

Goal and scope / 目标与范围

Strengthen the existing L1–L4 check system so daily v0.1.x cuts and #107 release readiness rest on one documented bar. Adopt the applicable parts of vllm-project/router#273 — a unified pre-commit static source of truth, path-aware skipping, and optional ready/merge/nightly selection — without copying GPU hardware lanes. Default automated CI continues to use model stubs and must not call paid services or require real GPUs. Suites that need GPU resources or real providers are a follow-up plan, not a Q4 delivery commitment.

把现有 L1–L4 检查做成可支撑每日 v0.1.x 与 #107 发布决策的单一契约。借鉴 router#273 中适用的部分（统一 pre-commit 作为静态检查单一来源、路径感知跳过、以及可选的 ready/merge/nightly 分层），不搬用 GPU 硬件车道。默认自动 CI 继续使用模型替身，不调用付费服务、不依赖真实 GPU。需要 GPU 或真实推理的用例只作为后续计划，不作为本季度交付承诺。

Design / 设计

0) CI layers — what runs where / 分层：PR、main 与 nightly

Target policy. Default automated CI still uses model stubs and must not call paid services or real GPUs. release-checks.yml remains the unique L1+L2 implementation; a job input selects the Python mark filter (ready on PRs, not nightly on main). Nightly does not re-run suites already covered on PR or main.

目标策略如下。默认自动 CI 仍使用模型替身，不调用付费服务、不依赖真实 GPU。L1+L2 仍以 release-checks.yml 为唯一实现；用 job 输入切换 Python 筛选（PR 跑 ready，main 跑 not nightly）。nightly 不再重复跑 PR 或 main 已经覆盖的套件。

1) Directory layout / 目录树

Today / 现状: frontend *.test.ts and Python test_*.py are mixed at the tests/ root, with tests/browser/ and tests/fixtures/ beside them. Plugin tests live under plugins/deepseek-harness/test/. There is no .pre-commit-config.yaml and no CI-layer pytest marks.

现状：前端 *.test.ts 与 Python test_*.py 混在 tests/ 根目录，旁边是 tests/browser/ 与 tests/fixtures/。插件测试在 plugins/deepseek-harness/test/。还没有 .pre-commit-config.yaml，也没有 CI 层 pytest mark。

Target (canonical) / 目标布局. Publish this convention in docs/development/validation.md and docs/architecture/test-contracts.md. Migrate incrementally in Q4; do not big-bang rewrite the tree.

目标布局如下，写进现有文档，本季度按约定增量迁移，不一次性搬完。

CI / quality files:

.pre-commit-config.yaml          # fmt, workflow lint, debug-statement, and similar static checks
.github/workflows/
  ci.yml                         # PR + push to main → release-checks.yml
  release-checks.yml             # unique L1+L2; input selects Python ready vs not nightly
  pre-commit.yml                 # CI runs that same pre-commit file only
  nightly.yml                    # advisory lanes only (no re-run of the PR/main bar)
  codeql.yml                     # or equivalent dependency/image scan (on PR and main)
  dsh-plugin.yml                 # path-filtered plugin tests
  reviewer-routing.yml           # not a test suite

Test cases:

tests/
├── frontend/                      # today's tests/*.test.ts (118 files)
│   ├── canvas/
│   ├── generation/
│   ├── agent/
│   ├── creation/
│   ├── media/
│   └── …
├── python/                        # today's tests/test_*.py (88 files)
│   ├── conftest.py
│   ├── capabilities.py              # requires_ffmpeg / requires_posix (skip gates, not CI-layer marks)
│   ├── helpers/
│   │   ├── generation_helpers.py
│   │   ├── process_helpers.py
│   │   └── legacy_fixtures.py
│   ├── creation/                    # storage, contracts, assets, API, history, layers
│   ├── agent/                       # gateway, tools, owner-rights, runtimes
│   ├── providers/                   # Kling, MiniMax, MOSS, OpenAI, vLLM-Omni, self-hosted (mocked HTTP)
│   ├── cli/                         # CLI, install, launcher
│   ├── media/                       # voice library, subject library, timeline, ffmpeg
│   └── bench/                       # daily-benchmark helpers
├── browser/                       # already here; lane membership only in manifest.json
│   ├── manifest.json
│   ├── engine.mjs
│   ├── host.mjs
│   ├── creation_server.py
│   ├── voice_library_server.py
│   └── …                            # gate / extended / production / perf / manual scripts
└── fixtures/

Plugin tests stay at plugins/deepseek-harness/test/ (path-filtered workflow). Do not merge them into tests/ in Q4.

插件测试仍留在 plugins/deepseek-harness/test/，本季度不搬进 tests/。

Marks vs directories / mark 与目录的关系: CI-layer marks are ready / merge / nightly. Folder names (frontend/ python/ creation/ …) are layout only — they are not pytest marks. Browser lanes (gate / extended / extendedFixture / production / perf / manual) stay in tests/browser/manifest.json, not as pytest marks.

CI 层 mark 是 ready / merge / nightly。目录名只表布局，不是 pytest mark。浏览器车道仍由 manifest.json 管，不用 pytest mark。

Deprecated / 迁移

2) Marker design / mark 设计

Today / 现状: only capability markers requires_ffmpeg and requires_posix are registered in tests/conftest.py. CI does not pass -m, so they are skip reasons, not lane selectors. requires_ffmpeg() is defined but unused; several tests still inline skipif. There are no ready / merge / nightly marks, and no unit / integration / gpu / slow marks.

现状只有能力门禁 mark，CI 不按 mark 筛选，也没有 CI 层 mark。

Target / 目标. Register marks in [tool.pytest.ini_options] (or pytest.ini) with --strict-markers once retagging is complete. Do not copy router/Omni hardware SKUs (cuda / L4 / H100 / cards_n). Do not register unit / integration / e2e as CI-layer marks.

在 pytest 配置里注册 mark；不搬用 GPU 硬件轴；不把 unit / integration / e2e 当作 CI 层 mark。

Env-gated real-engine tests (OPENDIRECTOR_REAL_CLI, DEEPSEEK_ENGINE_TEST, JIUWENSWARM_ENGINE_TEST) stay skipped in default CI and, when tagged, use @pytest.mark.nightly. They are not a Q4 required lane.

真实引擎环境开关用例默认仍跳过；若打 mark，归 nightly，不作为本季度必须跑通的车道。

CI selection / 筛选示例

# PR — ready subset; unmarked tests count as ready during migration
pytest -m "ready or (not merge and not nightly)" tests/python

# main — stubbed suite except nightly-only; this is the v0.1.x cut
pytest -m "not nightly" tests/python

# nightly flake triage — nightly-only cases, then the watchlist three times
pytest -m nightly tests/python

During migration, unmarked Python tests keep running on PRs (they are treated as ready) so coverage does not drop by accident. Then retag heavier stubbed cases as merge and timing-sensitive cases as nightly.

迁移期未打 mark 的 Python 用例仍算 ready，避免意外从 PR 掉落；再把较重的替身用例标为 merge，时序敏感用例标为 nightly。

Work and milestones / 工作与里程碑

• By November 30 · P0: add .pre-commit-config.yaml and a CI job that runs that file only (format, workflow lint, debug-statement and similar static checks). Stop duplicating static commands inside the 30-minute Release checks job. Add path filters so documentation-only PRs skip heavy test and browser work. Keep release-checks.yml as the unique L1+L2 implementation.

• By November 30 · P0: stabilize nightly advisory lanes (browser-extended, production, perf). The v0.1.x bar is the main push; nightly does not re-run that bar; do not auto-tag.

• By November 30 · P0: publish a tests/ directory design that states where frontend unit tests, Python tests, browser journeys, and fixtures live, how they map to L1–L4 and ready/nightly selection, and the naming and layout convention for new tests. Land it in existing docs (docs/development/validation.md and docs/architecture/test-contracts.md); migrate incrementally, and do not big-bang rewrite the tree in Q4.

• By November 30 · P0: introduce Python layer marks (or equivalent selection) so PR jobs keep a ready subset, and timing-sensitive cases stay on the flake watchlist or nightly. Update docs/development/validation.md in place; do not invent a second CI contract.

• By November 30 · P0: add security scanning on pull requests (for example CodeQL or dependency/image scanning). This is a PR gate, not a scheduled-only job.

• First half of December · P0: place [roadmap] Q4 2026: Next-generation video creation workflow #107 reliability coverage — tool failures, CLI workflows, repeated undo/redo, partial recovery, asset reuse, and representative caption/picture/audio timing — into the declared layer rather than onto every PR. For each release candidate, record SHA, environment, commands, artifacts, failures and skips ([release] Establish repeatable automated checks for release candidates #17, [RFC] First release scope and readiness checklist #7); report real-service and real-model acceptance separately.

• First half of December · P1: land the public quality-gate and read-only board ([feat] ci: 接入公共质量门禁并发布单仓看板 #135) without shrinking Release checks. Per existing policy, the Coordinator may promote an advisory lane that has been green for two weeks. Begin L5 stability checks (process interrupt/restart, project reopen, long-run) on a weekly or pre-release cadence if capacity allows.

• By December 31 · P1: document the fork and permissions model. Add package-validation hooks when [roadmap] Q4 2026 Usability: onboarding and Windows/macOS/Linux/HarmonyOS delivery / 易用性与多平台交付 #265 has distributable artifacts; this roadmap does not own installer delivery. Keep the human UX smoke ([process] Daily human UX smoke (after nightly / v0.1.x) #214) and generation-quality samples ([eval] Daily generation-quality sample set (fixed prompts × providers) #215) outside default CI. Publish only a follow-up plan for GPU/real-provider lanes (owners, hardware, credentials, and a first matrix) — those jobs are not a Q4 requirement.

• **11月底 · P0：**新增 .pre-commit-config.yaml，CI 只跑这一份静态检查（格式、workflow 校验、debug-statement 等），不再在 30 分钟 Release checks 中复制一套命令；为纯文档 PR 增加路径过滤，跳过重测试和浏览器检查。L1+L2 仍以 release-checks.yml 为唯一实现。

• **11月底 · P0：**修稳 nightly 的 advisory lane（browser-extended、production、perf）。发布卡 main 上的门槛，nightly 不再重跑该门槛，不自动打 tag。

• **11月底 · P0：**给出 tests/ 目录设计：写明前端单元、Python、浏览器旅程和夹具各自放在哪里，如何对应 L1–L4 以及 ready/nightly 筛选，以及新测试的命名与布局约定。落到现有文档（docs/development/validation.md 与 docs/architecture/test-contracts.md），按约定增量迁移，本季度不做整棵目录的一次性搬迁。

• **11月底 · P0：**为 Python 套件增加层级标记（或等价筛选），PR 只保留 ready 子集，时序敏感用例留在 flake watchlist 或 nightly。就地更新 docs/development/validation.md，不另写第二份 CI 契约。

• **11月底 · P0：**在 PR 上增加安全扫描（如 CodeQL 或依赖/镜像扫描）。这是 PR 门禁，不是只做定时任务。

• **12月上半月 · P0：**把 [roadmap] Q4 2026: Next-generation video creation workflow #107 可靠性覆盖（工具失败、CLI 流程、反复撤销/重做、局部恢复、素材复用、代表性字幕/画面/音频时序）放到声明的层级，而不是全部堆进每个 PR。每个发布候选记录 SHA、环境、命令、产物、失败与跳过（[release] Establish repeatable automated checks for release candidates #17、[RFC] First release scope and readiness checklist #7）；真实服务与真实模型验收单独报告。

• **12月上半月 · P1：**在不缩小 Release checks 的前提下落地公共质量门禁与只读看板（[feat] ci: 接入公共质量门禁并发布单仓看板 #135）。按现有政策，连续两周稳定绿色的 advisory lane 可由发布协调人升为阻断。若有容量，开始 L5 稳定性检查（进程中断重启、工程重开、长稳），按每周或发布前运行。

• **12月底 · P1：**写明 fork 与权限模型。在 [roadmap] Q4 2026 Usability: onboarding and Windows/macOS/Linux/HarmonyOS delivery / 易用性与多平台交付 #265 已有可分发安装包时补上包校验钩子，本路线图不负责安装包交付。人工 UX smoke（[process] Daily human UX smoke (after nightly / v0.1.x) #214）与生成质量样例（[eval] Daily generation-quality sample set (fixed prompts × providers) #215）不纳入默认 CI。GPU/真实推理车道只发布后续计划（负责人、硬件、凭证和首份矩阵），不作为本季度必须跑通的任务。

Implementation and dependencies / 实现影响与依赖

Reuse docs/development/validation.md, docs/development/ci-failures.md, docs/architecture/test-contracts.md, ci.yml, nightly.yml, release-checks.yml and tests/browser/manifest.json. Existing tasks: #200, #213 (landed), #214, #215, #135, #17, #7. Reliability scenarios come from the Agent (#261) and canvas/timeline (#262) roadmaps; real inference and installers stay with #264 and #265. Design reference: vllm-project/router#273. The public gate template is the agent-CI-Infra workflow described in #135.

复用现有验证文档、测试契约、失败模式说明和可复用工作流；已有 CI issue 继续承接实现。可靠性场景由 Agent 与画布/时间线路线图提供，真实推理和安装包分别由 #264、#265 负责。router#273 仅作分层与静态门禁参考，不引入 GPU queue。

Acceptance / 验收标准

• Pre-commit is the single static source of truth, and CI runs that file rather than a second command list. / pre-commit 是静态检查的单一来源，CI 只跑这一份文件，不再维护第二套命令。
• Documentation-only PRs skip heavy test and browser lanes; L1+L2 still have one implementation. / 纯文档 PR 跳过重测试和浏览器车道，L1+L2 仍只有一份实现。
• A tests/ directory design is documented (layout, layer mapping, and convention for new tests), and new tests follow it. / tests/ 目录设计已写入文档（布局、层级对应和新测试约定），新测试按该约定添加。
• Python layer marks (or equivalent selection) keep a ready subset on PRs; timing-sensitive cases stay on the flake watchlist or nightly. / Python 层级标记（或等价筛选）让 PR 只跑 ready 子集，时序敏感用例留在 flake watchlist 或 nightly。
• The v0.1.x bar is the main push; nightly runs only suites that PR and main do not already cover; advisory failures are tracked; no auto-tag; default CI makes no paid-model or real-GPU calls. / 发布卡 main 上的门槛，nightly 只跑 PR/main 尚未覆盖的套件，advisory 失败被跟踪，不自动打 tag，默认 CI 不调用付费模型、不使用真实 GPU。
• [roadmap] Q4 2026: Next-generation video creation workflow #107 reliability scenarios have automated coverage at the declared layer, with evidence recorded per release candidate. / [roadmap] Q4 2026: Next-generation video creation workflow #107 可靠性场景在声明层级有自动覆盖，每个发布候选保留证据。
• Security scanning (for example CodeQL or dependency/image scanning) runs on every pull request. / 安全扫描已落地，并在每个 PR 上运行。
• A follow-up plan for GPU/real-provider suites exists; those jobs are not required to run in Q4. / GPU/真实推理套件已有后续计划，不要求在 Q4 跑通。