Skip to content

Reduce CI queueing and repeated security review test work - #746

Merged
JacobStephens2 merged 4 commits into
mainfrom
issue-725
Oct 17, 2026
Merged

JacobStephens2 merged 4 commits into
mainfrom
issue-725

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

PR update → cancel superseded CI/Release for that PR
artifact/pipeline paths → generated Release + existing smoke checks
main/tag → independent runs; publication graph preserved
11 malformed final messages → in-process review interface
4 representative Runs → end-to-end delivery checks

Closes #725.

Evidence

  • Before: cargo test --test security_review invalid_or_missing_final_lines_hold_self_merge_but_report_only_runs_continue -- --exact --nocapture exercised 22 Runs in 72.42 s on this Linux host.
    After: the same command exercises four representative Runs in 12.28 s; all 11 malformed cases (plus missing None) pass in process in <0.01 s. Compilation is excluded from these test durations.
  • Before: workflows lacked concurrency; superseded work retained queued/active jobs.
    After: a second push at 03:36:03 UTC cancelled queued macOS work at 03:36:04, released the active Linux test runner at 03:36:42, and released the active Linux artifact runner at 03:37:04. Superseded CI and Release both concluded cancelled. The macOS aggregate rejected the cancelled shards.
  • Successor CI and Release passed, including site tests, Linux cargo test, all macOS shards and aggregate, both release artifacts, Rocky Linux and WSL smoke checks. PR publication jobs stayed skipped.

Measured macOS job timings in seconds, baseline main run vs the successor CI run. Start delay is workflow creation to runner start, including startup and any concurrency wait; execution includes setup/build/tests.

Shard Before start delay Before execution After start delay After execution
1 1528 984 879 819
2 434 856 1393 708
3 2396 860 924 775
4 1695 842 757 829

These are observations from different hosted runs, not a controlled queue benchmark or a promised savings percentage. The baseline slowest shard's nextest execution was 916.842 s, with the unauthenticated publisher matrix at 306.951 s; this run measured 763.643 s and 244.253 s respectively. That process/credential matrix remains end to end. The final-message table measured 158.687 s → 31.166 s on macOS, alongside the targeted Linux measurement above. Further performance work stays linked to #570 and the completed #174 baseline, without assuming those old targets describe this host.

Release's macOS artifact waited 724 s from workflow creation and executed in 192 s. Linux CI executed in 552 s. No live tag was created for this experiment: GitHub's @actions/expressions 0.3.61 evaluated the actual concurrency expressions for repeated PR heads, different PRs/fork branch names, main and tags. PR groups are stable per workflow/PR; non-PR groups include run IDs to preserve even pending publications. YAML-parsed Release jobs and tag triggers compare equal to main.

Validation:

  • cargo check --all-targets, cargo fmt --check, cargo clippy --all-targets -- -D warnings: passed.
  • Full cargo nextest run --no-fail-fast --status-level slow --final-status-level all: 1,768 passed, 2 intentionally skipped, 916.006 s; security validator tests: 65 passed.
  • Generator CLI test was red before implementation; python3 -m unittest discover -s scripts -p 'test_*.py' and actual python3 scripts/generate-release.py --check now pass, including stale-output detection and restoration after generator failure.
  • Four local hash partitions cover every runnable test once: 444 / 437 / 431 / 456. Hosted partitions passed 428 / 422 / 415 / 443 tests: 1,708 runnable tests, with the same 1,710-test inventory on each shard (two ignored).
  • A separate probe with the actual nextest config reported a 31.026 s passing test as slow, retained an intentional failure, and exited 100. The 30 s threshold is half the retrospective's 60 s long-tail boundary; there is no termination, failure suppression or added retry policy.
  • Main branch protection returned 404 “Branch not protected”; active branch rules returned []. The factory and release script wait for available checks/runs without requiring Release by name. Thus docs/site/test-only PRs can skip Release without stranding a current required check. Revisit this contract before making Release mandatory for every PR. Generator ownership and policy are documented in docs/ci-performance.md.

Test seams

  • Release generator CLI: generation, freshness and failure restoration.
  • Review::from_final_message → Outcome: every original malformed case and the public hold/entry contract.
  • End-to-end Run: missing/malformed final messages in both Self-merge and report-only modes, including ready/open PR and public reason.
  • Workflow event contract: concurrency identities, Release path selection, hash-partition coverage and stable aggregate failure/cancellation handling.

Review

Independent Standards and Spec reviews used main (a9e2d5353812ec3a6d6caefaa0c72838d79cfeb2) as the fixed point. Standards' duplicated concurrency policy finding was addressed by making CI the single authored definition. Spec's pending hosted-measurement finding was reproduced with its exact command, then resolved with successful successor runs and the recorded measurements. Both axes read all 12 changed files; no changed file was left unread.

Unaddressed findings

  • Standards: none.
  • Spec: none.
  • Security: none. Guidance review against pinned merge base a9e2d5353812ec3a6d6caefaa0c72838d79cfeb2 covered the diff and supporting release/delivery code. Sandboxed local checks passed for the generator CLI fixture, malformed-message unit matrix, and retained four-Run end-to-end matrix.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Workflow/config/test changes can be reverted; artifact targets and publication policy are preserved. The customized workflow uses cargo-dist's documented allow-dirty = ["ci"] support and replaces its skipped freshness check with the complete generator check in CI.

Blast Radius: CI

Path selection depends on the current package inputs and required-check contract; update it when either changes.

Built with codex · gpt-6.1-sol · high

@JacobStephens2
JacobStephens2 merged commit b58d24d into main Oct 17, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-725 branch October 17, 2026 05:30
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce CI queueing and runtime without cancelling release publication

1 participant