Repository navigation
Fix JUnit subtest reports and require opt-in for live B2 CI #702
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
84fe3c1
fix: emit individual JUnit cases for subtests
sophiecarreras 39dc7b5
ci: keep draft pull requests free of live B2 operations
sophiecarreras 190ef77
test: interpret CI gate expressions without executing code
sophiecarreras 95e3a17
fix: preserve JUnit teardown totals and subtest diagnostics
sophiecarreras 099b642
fix: keep interrupted JUnit reports inconclusive
sophiecarreras 53522ac
ci: require explicit opt-in for live B2 operations
sophiecarreras File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Keep draft pull request CI credential-free: skip bucket cleanup and SDK/CLI live tests while retaining unit and build checks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Keep JUnit case totals consistent when a failed subtest parent also has a teardown error. Preserve requested captured output on subtest cases while respecting passing-test logging settings. Leave interrupted reports unreconciled rather than manufacturing passing outcomes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Emit individual JUnit cases for pytest and unittest subtests so report totals match their case entries, including failures and skips. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Require the B2_LIVE_TESTS_ENABLED repository variable to opt into live tests and bucket cleanup, including ready pull requests and master pushes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| ###################################################################### | ||
| # | ||
| # File: test/junit_subtests.py | ||
| # | ||
| # Copyright 2026 Backblaze Inc. All Rights Reserved. | ||
| # | ||
| # License https://www.backblaze.com/using_b2_code.html | ||
| # | ||
| ###################################################################### | ||
| """Keep pytest 9's JUnit subtest counts consistent with its case elements. | ||
|
|
||
| Only the XML reporter sees copied subtest reports with distinct identities. | ||
| Other plugins, test execution and pytest's terminal/exit outcomes are unchanged. | ||
| This compatibility layer deliberately uses pytest's internal reporter API; | ||
| its subprocess regressions must pass when upgrading pytest. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from copy import copy | ||
| from itertools import count | ||
|
|
||
| import pytest | ||
| from _pytest.junitxml import LogXML, xml_key | ||
| from _pytest.subtests import SubtestReport | ||
|
|
||
|
|
||
| class SubtestLogXML(LogXML): | ||
| def __init__(self, *args, **kwargs): | ||
| super().__init__(*args, **kwargs) | ||
| self._subtest_ids = count(1) | ||
|
|
||
| @pytest.hookimpl(trylast=True) | ||
| def pytest_runtest_logreport(self, report): | ||
| if not isinstance(report, SubtestReport): | ||
| return super().pytest_runtest_logreport(report) | ||
| report = copy(report) | ||
| report.nodeid += f'::subtest-{next(self._subtest_ids)}' | ||
| super().pytest_runtest_logreport(report) | ||
| # The normal reporter writes output at teardown, which subtests lack. | ||
| # Failure-only logging and regular skips have already written theirs. | ||
| if ( | ||
| report.passed | ||
| or (report.failed and self.log_passing_tests) | ||
| or (report.skipped and hasattr(report, 'wasxfail')) | ||
| ): | ||
| self.node_reporter(report).write_captured_output(report) | ||
| # Subtests have a call report, but no later teardown to close the case. | ||
| self.finalize(report) | ||
| self.open_reports = [entry for entry in self.open_reports if entry is not report] | ||
|
|
||
| def pytest_sessionfinish(self): | ||
| # Upstream subtracts a logical test when call and teardown both fail, | ||
| # although it emits two XML cases. Compensate from the actual case list | ||
| # so every outcome remains present and the declared count matches it. | ||
| # An interrupted call can leave a bare, unfinished XML case. Do not | ||
| # reconcile such a report: that would manufacture a passing outcome. | ||
| if not self.node_reporters: | ||
| self.cnt_double_fail_tests = sum(self.stats.values()) - len(self.node_reporters_ordered) | ||
| super().pytest_sessionfinish() | ||
|
|
||
|
|
||
| @pytest.hookimpl(trylast=True) | ||
| def pytest_configure(config): | ||
| original = config.stash.get(xml_key, None) | ||
| if original is None: | ||
| return # No --junitxml, or an xdist worker: leave reporting untouched. | ||
| replacement = SubtestLogXML( | ||
| original.logfile, | ||
| original.prefix, | ||
| original.suite_name, | ||
| original.logging, | ||
| original.report_duration, | ||
| original.family, | ||
| original.log_passing_tests, | ||
| ) | ||
| config.pluginmanager.unregister(original) | ||
| config.stash[xml_key] = replacement | ||
| config.pluginmanager.register(replacement) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| ###################################################################### | ||
| # | ||
| # File: test/static/test_ci_draft_safety.py | ||
| # | ||
| # Copyright 2026 Backblaze Inc. All Rights Reserved. | ||
| # | ||
| # License https://www.backblaze.com/using_b2_code.html | ||
| # | ||
| ###################################################################### | ||
| """Exercise CI's draft, credential and dependency gates without running jobs.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import ast | ||
| import re | ||
| from pathlib import Path | ||
|
|
||
| import pytest | ||
| import yaml | ||
|
|
||
| JOBS = yaml.safe_load((Path(__file__).parents[2] / '.github/workflows/ci.yml').read_text())['jobs'] | ||
|
|
||
|
|
||
| def expression(value, context): | ||
| """Evaluate only the boolean/string expression subset used by these gates. | ||
|
|
||
| actionlint validates the complete workflow separately. Unknown contexts or | ||
| expression syntax fail this check rather than being assumed safe. | ||
| """ | ||
| if not isinstance(value, str) or not value.startswith('${{'): | ||
| return value | ||
| code = value[3:-2].strip().replace('cancelled()', repr(context['cancelled'])) | ||
| code = re.sub( | ||
| r'[a-zA-Z_][\w-]*(?:\.[a-zA-Z_][\w-]*)+', lambda match: repr(context[match.group()]), code | ||
| ) | ||
| code = code.replace('&&', ' and ').replace('||', ' or ') | ||
| code = re.sub(r'!(?!=)', 'not ', code) | ||
| code = re.sub(r'\btrue\b', 'True', code) | ||
| code = re.sub(r'\bfalse\b', 'False', code) | ||
| tree = ast.parse(code.strip(), mode='eval') | ||
|
|
||
| def resolve(node): | ||
| if isinstance(node, ast.Constant) and isinstance(node.value, (str, bool)): | ||
| return node.value | ||
| if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.Not): | ||
| return not resolve(node.operand) | ||
| if isinstance(node, ast.BoolOp): | ||
| result = resolve(node.values[0]) | ||
| for child in node.values[1:]: | ||
| if isinstance(node.op, ast.And) and not result: | ||
| return result | ||
| if isinstance(node.op, ast.Or) and result: | ||
| return result | ||
| result = resolve(child) | ||
| return result | ||
| if isinstance(node, ast.Compare): | ||
| left = resolve(node.left) | ||
| for operator, child in zip(node.ops, node.comparators): | ||
| right = resolve(child) | ||
| assert isinstance(operator, (ast.Eq, ast.NotEq)), code | ||
| if (left == right) != isinstance(operator, ast.Eq): | ||
| return False | ||
| left = right | ||
| return True | ||
| raise AssertionError(f'Unsupported workflow expression: {code}') | ||
|
|
||
| return resolve(tree.body) | ||
|
|
||
|
|
||
| def context( | ||
| draft=True, | ||
| credentials=True, | ||
| event='pull_request', | ||
| lint='success', | ||
| cleanup='skipped', | ||
| live_enabled='true', | ||
| ): | ||
| return { | ||
| 'github.event_name': event, | ||
| 'github.event.pull_request.draft': draft, | ||
| 'needs.lint.result': lint, | ||
| 'needs.cleanup_buckets.result': cleanup, | ||
| 'cancelled': False, | ||
| 'vars.B2_LIVE_TESTS_ENABLED': live_enabled, | ||
| 'secrets.B2_TEST_APPLICATION_KEY': 'test-key' if credentials else '', | ||
| 'secrets.B2_TEST_APPLICATION_KEY_ID': 'test-id' if credentials else '', | ||
| } | ||
|
|
||
|
|
||
| @pytest.mark.parametrize('credentials', [True, False]) | ||
| def test_draft_never_starts_bucket_cleanup(credentials): | ||
| assert not expression( | ||
| JOBS['cleanup_buckets'].get('if', '${{ true }}'), context(credentials=credentials) | ||
| ) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize('job_id', ['test', 'cli-compat']) | ||
| @pytest.mark.parametrize( | ||
| 'draft,event', [(True, 'pull_request'), (False, 'pull_request'), (False, 'push')] | ||
| ) | ||
| def test_b2_credentials_are_empty_only_for_drafts(job_id, draft, event): | ||
| values = context(draft=draft, event=event) | ||
| for name in ('B2_TEST_APPLICATION_KEY', 'B2_TEST_APPLICATION_KEY_ID'): | ||
| actual = expression(JOBS[job_id]['env'][name], values) | ||
| assert actual == ('' if draft else values[f'secrets.{name}']) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize('job_id', ['test', 'cli-compat']) | ||
| @pytest.mark.parametrize( | ||
| 'draft,credentials,allowed', | ||
| [(True, True, False), (True, False, False), (False, True, True), (False, False, False)], | ||
| ) | ||
| def test_live_steps_require_non_draft_and_credentials(job_id, draft, credentials, allowed): | ||
| values = context(draft=draft, credentials=credentials) | ||
| values.update( | ||
| { | ||
| f'env.{name}': expression(value, values) | ||
| for name, value in JOBS[job_id]['env'].items() | ||
| if name.startswith('B2_TEST_') | ||
| } | ||
| ) | ||
| values.update({'matrix.os': 'ubuntu-latest', 'matrix.python-version': '3.14'}) | ||
| steps = [ | ||
| step | ||
| for step in JOBS[job_id]['steps'] | ||
| if step.get('name') in ('Run integration tests', 'Run CLI integration tests (with secrets)') | ||
| ] | ||
| assert len(steps) == 1 | ||
| assert bool(expression(steps[0]['if'], values)) == allowed | ||
|
|
||
|
|
||
| @pytest.mark.parametrize('job_id', ['test', 'cli-compat']) | ||
| @pytest.mark.parametrize( | ||
| 'lint,cleanup,allowed', | ||
| [ | ||
| ('success', 'skipped', True), | ||
| ('success', 'success', True), | ||
| ('failure', 'skipped', False), | ||
| ('success', 'failure', False), | ||
| ('success', 'cancelled', False), | ||
| ], | ||
| ) | ||
| def test_unit_jobs_survive_skipped_cleanup_but_not_failed_dependencies( | ||
| job_id, lint, cleanup, allowed | ||
| ): | ||
| assert 'lint' in JOBS[job_id]['needs'] | ||
| assert 'cleanup_buckets' in JOBS[job_id]['needs'] | ||
| values = context(lint=lint, cleanup=cleanup) | ||
| assert bool(expression(JOBS[job_id]['if'], values)) == allowed | ||
| values['cancelled'] = True | ||
| assert not expression(JOBS[job_id]['if'], values) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize('event', ['pull_request', 'push']) | ||
| def test_non_draft_cleanup_is_retained(event): | ||
| assert expression(JOBS['cleanup_buckets']['if'], context(draft=False, event=event)) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize('event', ['pull_request', 'push']) | ||
| @pytest.mark.parametrize('live_enabled', ['', 'false', 'true']) | ||
| def test_non_draft_live_operations_require_explicit_repository_opt_in(event, live_enabled): | ||
| values = context(draft=False, event=event, live_enabled=live_enabled) | ||
| allowed = live_enabled == 'true' | ||
| assert bool(expression(JOBS['cleanup_buckets']['if'], values)) == allowed | ||
| for job_id in ('test', 'cli-compat'): | ||
| job_values = dict(values) | ||
| for name in ('B2_TEST_APPLICATION_KEY', 'B2_TEST_APPLICATION_KEY_ID'): | ||
| actual = expression(JOBS[job_id]['env'][name], values) | ||
| assert actual == (values[f'secrets.{name}'] if allowed else '') | ||
| job_values[f'env.{name}'] = actual | ||
| job_values.update({'matrix.os': 'ubuntu-latest', 'matrix.python-version': '3.14'}) | ||
| steps = [ | ||
| step | ||
| for step in JOBS[job_id]['steps'] | ||
| if step.get('name') | ||
| in ('Run integration tests', 'Run CLI integration tests (with secrets)') | ||
| ] | ||
| assert len(steps) == 1 | ||
| assert bool(expression(steps[0]['if'], job_values)) == allowed |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do not treat completed collection reports as interrupted tests.
If a module skips during collection, pytest 9.0.3 retains its reporter in
node_reporters. If another test fails during call and teardown, this guard prevents compensation: XML contains three cases but declares two tests. This still violates the report-count requirement. Distinguish completed collection reporters from unfinished runtime reporters, and add this combination to the regressions. Preserve the interrupted-call behavior. (raw.githubusercontent.com)