Repository navigation
Bound CI jobs with measured execution timeouts - #1508
Merged
Merged
Conversation
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A hung Actions job inherited GitHub's six-hour default and held a Ticket run in CI watch. Every CI job and every PR-reachable Release job now has an explicit execution budget, sized from recent successful runs.
Release's generated limits live in
scripts/generate-release.py; the smoke reusable workflow sets limits on its concrete jobs because GitHub does not support timeout-minutes on reusable-workflow calls. Tag-onlyhost,announce,custom-publish-cratesandcustom-release-notescannot hold a PR's CI watch, so their timeouts are outside this change.Closes #1496
Evidence
python3 -m unittest discover -s scripts -p test_workflow_timeouts.pyfailed all seven CI-job subtests withinherits GitHub's 360-minute default. Adding Release coverage then failed for all six PR-reachable concrete Release/smoke jobs. No compilation or setup failure was counted as reproduction.After: the same command passes both tests, covering all 13 jobs. The confirmed cause was missing job-level timeouts; step-level bounds do not substitute for this contract.
python3 -m unittest discover -s scripts -p test_generate_release.pyfailed before the generator emitted the new execution bounds and passes afterward. The pinned real cargo-dist 0.33.0 regenerated Release;python3 scripts/generate-release.py --checkpasses.python3 -m unittest discover -s scripts -p 'test_*.py': 18 passed.NEXTEST_TEST_THREADS=3 RUST_TEST_THREADS=3 /usr/bin/time -p scripts/affected-tests mainpassed. The selector used the full Rust suite via itscargo testfallback (nextest unavailable locally), followed by all three Release-generator tests. Wall time: 1919.02 seconds (31m59s), compilation included. All 80 Rust result summaries were successful. Three test threads follow the repository's guidance for deadline-sensitive tests on busy hosts; hosted CI independently passed nextest on the same head.649c0ec7. All 18 checks finished (13 passed, five expected skips). The Linux Rust job took 971 seconds of its 60-minute budget; Console 127/900, site 82/600, macOS gate 211/1200, release builds 176–254/1200, Rocky 12/300 and WSL 50/600 seconds. PR policy uses the macOS gate; the full shards are skipped as expected.Observed runtime and chosen limits
Measured on 2026-10-28 UTC with
gh run list --workflow <workflow> --status success(PR, push and schedule events separately) andgh run view <id> --json jobs. Figures are successful jobs'completedAt - startedAt, including setup/cleanup and excluding queue time and skipped jobs. Full run links and rationale are in the runtime record.The CI sample comprises three PRs, three main pushes and one nightly run; Release comprises four PRs and three tag runs. Rocky/WSL execution figures use the tag samples because those PR samples skipped their smoke tests. The same reusable jobs can execute on pipeline-changing PRs, including this one. Full suites get at least 3.6 times the slowest observed runtime, browser/gate jobs at least 4.5 times, and Release builds at least 3.7 times. Five-minute orchestration floors and ten minutes for WSL leave room for setup and transfers. These budgets need remeasurement as job workloads grow.
Test seams
Review coverage
Independent Standards and Spec reviews against
mainfound 0 Standards / 0 Spec findings. Each reviewer read all seven changed files, including generated Release hunks. Unread changed files: none. Generated-output exclusions: none. Reports and each axis's final files-read list are retained in the Session review artifacts (.thirdshift-review-zTkRVU/standards.md,spec.md,summary.json).Unaddressed findings
Security review outcome
No unaddressed introduced findings.
Merge Danger
Door: two-way
Revert the workflow limits and generator policy to restore previous behavior.
Blast Radius: CI
A healthy job slower than its budget will end early. Limits include several times the observed execution time; Rust suites retain 60 minutes. This changes job execution bounds and leaves CI watch's behavior intact.
Built with codex codex-cli 0.164.0 · gpt-6.1-sol · xhigh