Repository navigation
Conversation
A "come together" pull request that bumps every module hash in
known_good.json at once usually fails for a reason visible long before the
hour-long image builds finish: a module no longer resolves, a target was
renamed, a documentation mount broke. Today every push to such a PR spends
a full pipeline to find that out.
Adding the 'staged-ci' label now skips the expensive jobs and leaves the
cheap, known_good.json-derived validation running:
runs - known_good_correct, bzlmod-lock, format, copyright
runs - test_and_docs.yml documentation preflight
skipped - test_and_docs.yml unit tests, coverage, feature integration tests
skipped - build_and_test_{linux,qnx,autosd,ebclfsa}.yml
skipped - codeql-multiple-repo-scan.yml
Two details make this safe rather than a foot-gun:
1. staged_ci_guard.yml fails while the label is set. GitHub reports a job
skipped via `if:` as *skipped*, and branch protection treats a skipped
required check as satisfied - so without this guard a staged PR would be
mergeable although none of the heavy checks ever ran. The guard has to be
configured as a required status check for that protection to exist.
2. Every gated workflow now also listens for the `labeled` and `unlabeled`
pull-request events. Without them, adding or removing the label would not
trigger a new run and the gate would appear to do nothing.
On push, merge_group and release events the label list is empty, so the
condition evaluates to true and all jobs run unchanged.
Staging is a manual override for a supervised integration pull request, not
a way to merge unvalidated changes - the full pipeline still has to pass
before the merge.
|
The created documentation from the pull request is available at: docu-html |
Staging CI buys time to look at an integration failure; it does not remove
the failure. When a single module blocks the whole cycle, it can now be taken
out of the integration without losing its pinned state:
"enabled": false,
"disabled_reason": "blocked by eclipse-score/example#123"
Commenting the entry out is not an option - known_good.json is parsed with a
strict json.loads and JSON has no comments - and deleting it discards the
hash, the bazel_patches and the metadata, which is exactly the state needed to
put the module back. The flag keeps all of it, so re-enabling is a two-line
revert. The reason is mandatory so a disabled module cannot decay into an
unnoticed permanent state, and the generator prints every disabled module on
each run.
A disabled module contributes no bazel_dep/git_override entry, no coverage
target, no docs mount, no unit-test run and no SBOM entry.
The flag only governs the generated artefacts, so the two references that
would otherwise fail far away from known_good.json now abort the regeneration
naming the offending lines:
- a --@module//... flag left in .bazelrc, which Bazel resolves on every
invocation including 'query', making the whole workspace unusable;
- another module's extra_test_config/exclude_test_targets, or a leftover
sbom.tracked_modules entry, pointing into the disabled module.
Both were verified against the real known_good.json: disabling
score_config_management reports the .bazelrc flag, disabling score_logging
reports the SBOM entry and score_persistency's test config. Regeneration with
nothing disabled is byte-identical to the committed files.
Hand-written references (images/*/BUILD, showcases, the ITF scenarios) are not
detected and still have to be removed manually; the documentation says so
explicitly rather than implying the flag is sufficient for every module.
The enabled flag from the previous commit removed a module from the generated Bazel fragments, but three things still pointed at a module that was gone. All three are now handled. .bazelrc flags are the worst case: Bazel resolves --@score_module//... on every invocation, so a leftover flag breaks bazel query and bazel mod too - the very commands needed to diagnose it. The previous commit only detected this and aborted, leaving the integrator to edit .bazelrc by hand. The 12 module-specific flags now live in bazel_common/bazelrc/<module>.bazelrc, grouped by the repository the flag names rather than by the config it belongs to, which is what makes removal a pure deletion. update_module_from_known_good.py generates bazel_common/module_flags.bazelrc with one import per enabled fragment; the root .bazelrc imports that single file. A disabled module is simply absent from the list and comes back unchanged when re-enabled. sbom.tracked_modules was a hard error for no good reason - the SBOM should not be able to veto a deactivation. It is filtered automatically now, and the generator reports which modules it dropped. Wiring this up revealed that generate_sbom_module_content() had never been called from main(); score_sbom.MODULE.bazel was hand-maintained and carried a timestamp from 2026-08-21. Regenerating it changed only the timestamp, confirming the generator had been correct all along. The third case cannot be fixed, only detected. Dropping our bazel_dep removes this repository's request for a module, not the module: if another module declares its own dependency, Bazel falls back to the registry and resolves a released version instead of the pinned commit. The build succeeds and says nothing. check_disabled_modules.py resolves the real graph and fails with the list of modules pulling a disabled one back in. It returns before invoking Bazel when nothing is disabled, so it costs nothing on an ordinary pull request. Only score_config_management, score_kyron and score_time are leaves today. Verified with bazel build --announce_rc that all 12 flags still apply under --config=_common and --config=ferrocene-coverage-per, and end-to-end by disabling score_time in the real workspace: 8 -> 7 git_overrides, SBOM 7 -> 6, and a real bazel mod graph confirming the module was gone. 161 tests pass (was 156).
The previous commit wired generate_sbom_module_content() into main() but shipped the stale hand-maintained file: an end-to-end test restored the workspace with git checkout, which reverted the regenerated fragment, and it was staged without regenerating again. score_sbom.MODULE.bazel therefore still carried the 2026-08-21 timestamp while the generator produces the one from known_good.json. Caught by the Known Good Matches with Bazel job, which is exactly what it is for.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A "come together" integration PR that bumps every module hash in
known_good.jsonat once usually fails for a reason that is visible long before the hour-long image builds finish - a module no longer resolves, a target was renamed, a documentation mount broke. Today every push to such a PR spends a full pipeline (QNX, AutoSD, EBcLfSA, Linux images, unit tests, coverage, FIT, CodeQL) just to surface that.What this PR does
Adds a
staged-cilabel. While it is set on a PR, the expensive jobs skip themselves and only the cheap,known_good.json-derived validation runs:staged-ciknown_good_correct,bzlmod-lock,format,copyrighttest_and_docs.yml- documentation preflighttest_and_docs.yml- unit tests, coverage, FITbuild_and_test_{linux,qnx,autosd,ebclfsa}.ymlcodeql-multiple-repo-scan.ymlstaged_ci_guard.yml(new)Intended flow: label the integration PR, iterate until the docs preflight and the generator checks are green, then remove the label and let the full pipeline run.
Two details that make this safe rather than a foot-gun
1.
staged_ci_guard.ymlblocks the merge. GitHub reports a job skipped viaif:as skipped, and branch protection treats a skipped required check as satisfied. Without the guard, a staged PR would be mergeable even though none of the heavy checks ever ran - exactly the PR that needs them most. The guard always runs and fails while the label is present.2. Every gated workflow now also listens for
labeled/unlabeled. The existing triggers were[opened, reopened, synchronize], so a label change would not have started a new run and the gate would have appeared to do nothing.On
push,merge_groupandreleaseevents the label list is empty, the condition evaluates to true, and all jobs run unchanged.Staged CI Guardmust be added to the required status checks in the repository's branch-protection settings. Without that, the guard is informational and staging would weaken the merge gate instead of just deferring it.The
staged-cilabel also needs to be created in the repo (applying it requires write access, so external contributors cannot stage their own PRs).Scope / non-goals
This is the manual-override half of the problem. It deliberately does not add automatic fail-fast gating (a cheap
bazel mod graph/--nobuildresolution check that every PR runs before the heavy jobs) - that is a separate change and would benefit every PR, not just the bi-weekly integration one. This PR is intentionally small and fully revertible by deleting a handful ofif:lines.