Skip to content

Bound CI jobs with measured execution timeouts - #1508

Merged
JacobStephens2 merged 3 commits into
mainfrom
issue-1496
Oct 28, 2026
Merged

JacobStephens2 merged 3 commits into
mainfrom
issue-1496

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

A hung Actions job inherited GitHub's six-hour default and held a Ticket run in CI watch. Every CI job and every PR-reachable Release job now has an explicit execution budget, sized from recent successful runs.

 CI / PR-reachable Release job
-  timeout-minutes: 360 (inherited)
+  timeout-minutes: 5–60 (per-job, measured)

Release's generated limits live in scripts/generate-release.py; the smoke reusable workflow sets limits on its concrete jobs because GitHub does not support timeout-minutes on reusable-workflow calls. Tag-only host, announce, custom-publish-crates and custom-release-notes cannot hold a PR's CI watch, so their timeouts are outside this change.

Closes #1496

Evidence

  • Before: python3 -m unittest discover -s scripts -p test_workflow_timeouts.py failed all seven CI-job subtests with inherits GitHub's 360-minute default. Adding Release coverage then failed for all six PR-reachable concrete Release/smoke jobs. No compilation or setup failure was counted as reproduction.
    After: the same command passes both tests, covering all 13 jobs. The confirmed cause was missing job-level timeouts; step-level bounds do not substitute for this contract.
  • Generator: python3 -m unittest discover -s scripts -p test_generate_release.py failed before the generator emitted the new execution bounds and passes afterward. The pinned real cargo-dist 0.33.0 regenerated Release; python3 scripts/generate-release.py --check passes.
  • Tooling: python3 -m unittest discover -s scripts -p 'test_*.py': 18 passed.
  • Affected checks: NEXTEST_TEST_THREADS=3 RUST_TEST_THREADS=3 /usr/bin/time -p scripts/affected-tests main passed. The selector used the full Rust suite via its cargo test fallback (nextest unavailable locally), followed by all three Release-generator tests. Wall time: 1919.02 seconds (31m59s), compilation included. All 80 Rust result summaries were successful. Three test threads follow the repository's guidance for deadline-sensitive tests on busy hosts; hosted CI independently passed nextest on the same head.
  • PR CI: CI run 40615900874 and Release run 40615900906 both completed successfully on 649c0ec7. All 18 checks finished (13 passed, five expected skips). The Linux Rust job took 971 seconds of its 60-minute budget; Console 127/900, site 82/600, macOS gate 211/1200, release builds 176–254/1200, Rocky 12/300 and WSL 50/600 seconds. PR policy uses the macOS gate; the full shards are skipped as expected.

Observed runtime and chosen limits

Measured on 2026-10-28 UTC with gh run list --workflow <workflow> --status success (PR, push and schedule events separately) and gh run view <id> --json jobs. Figures are successful jobs' completedAt - startedAt, including setup/cleanup and excluding queue time and skipped jobs. Full run links and rationale are in the runtime record.

Workflow / job Recent successful execution (seconds) Timeout (minutes)
CI / console-test 98–182 15
CI / site-test 59–102 10
CI / test, including nightly cargo test 725–983 60
CI / macos-plan 3–4 5
CI / test-macos-gate 232–265 20
CI / test-macos-shard, all four shards 627–934 60
CI / test-macos 3–4 5
Release / plan 19–26 5
Release / build-local-artifacts, Linux and macOS 180–319 20
Release / build-global-artifacts 20–30 5
Release / custom-release-smoke / gate 3–6 5
Release / custom-release-smoke / rocky 11–16 5
Release / custom-release-smoke / wsl 62–66 10

The CI sample comprises three PRs, three main pushes and one nightly run; Release comprises four PRs and three tag runs. Rocky/WSL execution figures use the tag samples because those PR samples skipped their smoke tests. The same reusable jobs can execute on pipeline-changing PRs, including this one. Full suites get at least 3.6 times the slowest observed runtime, browser/gate jobs at least 4.5 times, and Release builds at least 3.7 times. Five-minute orchestration floors and ten minutes for WSL leave room for setup and transfers. These budgets need remeasurement as job workloads grow.

Test seams

  • The job-level GitHub Actions configuration in CI and PR-reachable Release jobs.
  • The Release generator CLI with an external cargo-dist stand-in and the pinned real generator.

Review coverage

Independent Standards and Spec reviews against main found 0 Standards / 0 Spec findings. Each reviewer read all seven changed files, including generated Release hunks. Unread changed files: none. Generated-output exclusions: none. Reports and each axis's final files-read list are retained in the Session review artifacts (.thirdshift-review-zTkRVU/standards.md, spec.md, summary.json).

Unaddressed findings

  • Standards: none.
  • Spec: none.
  • Security: none.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Revert the workflow limits and generator policy to restore previous behavior.

Blast Radius: CI

A healthy job slower than its budget will end early. Limits include several times the observed execution time; Rust suites retain 60 minutes. This changes job execution bounds and leaves CI watch's behavior intact.

Built with codex codex-cli 0.164.0 · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit b5ff109 into main Oct 28, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1496 branch October 28, 2026 02:47
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI jobs have no timeout-minutes, so a hung job holds a Run in CI watch for 6 hours

1 participant