Repository navigation
Azure Pipelines: macOS CI Tests and Ubuntu CI Tests (coverage) jobs are always skipped #109395
Description
Activity
macOS CI TestsThe macOS tests were disabled 3.5 years ago, when GitHub Actions CI was newer: PR #18818.
Ubuntu CI Tests (coverage)This is skipped because the condition:
and( and( succeeded(), eq(variables['coverage'], 'true') ), eq(dependencies.Prebuild.outputs['tests.run'], 'true') )evaluates to false:
and( and( True, eq('false', 'true') ), eq(dependencies['Prebuild']['outputs']['tests.run'], 'true') )Because
variables['coverage']is'false'.There is a
coverage: truefor the job's only step, but it looks like the above condition is evaluating to false before the step can even run. I've not checked if this is intentional.
Are these Azure Pipelines runs doing anything that is not covered by the GitHub Actions CI?
If not, a better course of action would be to remove them from Azure Pipelines. GitHub Actions is easier to manage, especially when it comes to restarting and maintenance. (Compare: PR #105823 that removed docs testing from AP.)
(The same applies to the other Ubuntu and Windows jobs that are running, but there's at least some unique things run in patchcheck that hasn't been migrated over, so we can deal with these later.)
In the issue that led to disabling the macOS jobs, @zooba outlined the required steps (#84018 (comment)):
Deleting the files is not the right first step.
First, it needs to be changed to a non-required check.
Then, I can use the web UI to disable it starting.
Then, we can remove some of the files in the directory. Others are used for the official release, and have to stay.
@zooba, for macOS:
- change to a non-required check (bpo-39837: Disable macOS tests on Azure Pipelines #18818)
- disable in web UI - does this need doing?
- remove some files: can we delete
.azure-pipelines/macos-steps.yml(and the matching jobs inci.ymlandpr.yml)?
For Ubuntu coverage:
- First off, do we want to fix coverage?
- If so, should we add something to GitHub Actions first?
- In any case, can we remove this skipped job now?
- added a commit that references this issue
on Sep 14, 2023 Ubuntu CI Tests (coverage)A quick experiment that hardcodes
coverage: trueto check if the run completes shows it core dumped after 59 minutes:Compared to 15 minutes for the regular
Ubuntu CI Testsbuild which passes.Here's a PR to remove the skipped coverage job: #109412
Yeah, coverage just never worked, unfortunately. It was a nice idea. It's possible that the segfault at 59 minutes is just the 1 hour timeout sending a Ctrl+C at a bad time, but the tests are so slow I don't think we'd want it enabled anyway. Maybe as a scheduled (daily) job.
Removing the files is fine. They don't do any harm by staying there, and might be helpful to someone trying to run tests on their fork, but if people are going to keep suggesting we remove them then I guess we may as well make them happy.
Reacted by Hugo van KemenadeAt work, the coverage is also slow on the CI. But it's faster locally, so we sometimes use it for local development. And you might just run a subset of tests for the bit of code you're working on, and check you're properly covering your new code.
So something for the future may be to set tooling up for local development, if people would find that useful.
I cannot find any CI runs where these two are completed / failed. They are always skipped.
Looks like Azure only stores build result for ~1 month or so, so I cannot trace it back further.
Examples:
Why does it happen?
Linked PRs