Repository navigation
[BUG]: Windows CI Tests False-Positive of Passing #604
Description
Description
After reviewing your Actions, it is clear tests are not running on Windows machines. On Windows, the linter and test script does not output any results and takes 0 or 1s to complete, whereas on Linux/MacOS they take >1 minute and output pytest results.
Your .github/workflows/pythonpackage.yml worflow does not specify a shell to use in the Run tests and linters step. Hence, on Windows, it defaults to using Powershell. Thus, what is run (which you can see in this example workflow) is simply:
pwsh -Command ./scripts/testThis does nothing more than ask Windows to attempt to open a file it does not know how to handle, which is easily repeated locally:
Commitizen_Testing_Error.mp4
Cloning the repo, changing the shebang to a portable variant, and running pytest, it is clear many tests are failing on Windows. This was first noticed in #602
https://github.com/commitizen-tools/commitizen/files/9790428/pytest_report.zip
Steps to reproduce
- Follow the Contributing steps to:
(a) Fork the repo, install dependencies through poetry, and setup the pre-commit hooks - Run pre-commit run --all
Current behavior
Tests are failing silently on GitHub Actions (i.e. CI), yielding false positives on prior commits. Tests fail explicitly when the repo is cloned and tested on Windows, which also prevents developers using Windows to contribute (pre-commit tests fail, preventing pushing to PRs).
Desired behavior
- Windows tests run on GitHub Actions
- Windows users can contribute to your repo
If you continue to package for Windows, you need to confirm that your tests are actually running and passing. Your tests are failing at this point.
Screenshots
Please see the Description.
Environment
Commitizen Version: 2.35.0
Python Version: 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]
Operating System: Windows
Local pytest Report
Adding the pytest run on Windows, first referenced in #602 . This report was generated using the pytest-html plugin.
Activity
[-]Tests Not Running on Windows[/-][+][BUG]: Windows CI Tests False-Positive of Passing[/+]Lee-W commentedon Oct 17, 2022
Hi @adam-grant-hendry , thanks so much for the detail report and testing! Without your report we'll not be able to find such issue. However, both Santi and I does not have a Windows machine. Wondering would you be interested in sending us a PR for fixing this issue. I'm happy to cowork with you to see how this can be fixed! (I'm a bit busy these days but will try to reply ASAP)
adam-grant-hendry commentedon Oct 17, 2022
@Lee-W Thank you for your kindness! It is so greatly appreciated.
Yes, I would be happy to help. There are two things to pursue here. The first is a quick fix and the second addresses the larger issue:
Quick Fix:
bashin yourpythonpackage.ymlGitHub Actions workflow.--no-verifylike you suggested if developing on Windows not inbashThat should get tests running on Windows in your CI.
Long-Term Fix:
powershellsupport.This is going to require fixing several failing tests. One challenge is
subprocessin Python on Windows references the environment variableComSpecto determine which shell to run commands in, which defaults to Command Prompt (C:\WINDOWS\system32\cmd.exe).This shouldn't be changed because several older Windows OS utilities may rely on running batch scripts that expect the old behavior of Command Prompt, so changing it to PowerShell or Bash could cause problems. Instead, PowerShell should be called out explicitly. Traditional PowerShell (i.e. not PowerShell Core) ships with Windows OS's since Windows 7, and GitHub Actions uses (traditional) PowerShell on Windows by default, so this shouldn't be a problem.
I recommend a wrapper around
subprocessas recommended in this realpython.com article and implemented in thedropboxignoreGitHub repo.fix(pythonpackage.yml): use `bash`
fix(pythonpackage.yml): use `bash`
fix(pythonpackage.yml): use `bash`
fix(pythonpackage.yml): use `bash`
fix(pythonpackage.yml): use `bash`
fix(pythonpackage.yml): use `bash`
woile commentedon Apr 28, 2023
Fixed in pr #605