Windows CI builds broken

Currently all Windows CI builds seem to be broken.

Example failure: https://github.com/python/mypy/actions/runs/2044190861

This might be caused by the switch to a new CI image with Windows Server 2022: actions/runner-images#4856

We could try falling back to Windows-2019 temporarily while we investigate what is going on.

Any help with investigating the failures would be appreciated, since I don't do much development in Windows.

Attempt to fix Windows CI by falling back to windows-2019

The only suspicious thing I saw in the output is cl : Command line warning D9002 : ignoring unknown option '/O0'. https://docs.microsoft.com/en-us/cpp/build/reference/o-options-optimize-code?view=msvc-170 doesn't mention O0. Maybe it's a setuptools issue?

Try to fix Windows build

I can take a look at the issue a little later.

last pass was https://github.com/python/mypy/runs/5694572863?check_suite_focus=true
first fail was https://github.com/python/mypy/runs/5700430887?check_suite_focus=true

The last pass was on windows server 2022, so I don't think that is the root cause.

Sample failed output

__________________________________ testAsync __________________________________
[gw0] win32 -- Python 3.7.9 D:\a\mypy\mypy\.tox\py37\Scripts\python.EXE
data: D:\a\mypy\mypy\mypyc\test-data\run-misc.test:3:
D:\a\mypy\mypy\mypyc\test\test_run.py:137: in run_case
    self.run_case_inner(testcase)
D:\a\mypy\mypy\mypyc\test\test_run.py:153: in run_case_inner
    self.run_case_step(testcase, step)
D:\a\mypy\mypy\mypyc\test\test_run.py:266: in run_case_step
    assert glob.glob('native.*.{}'.format(suffix))
E   AssertionError: assert []
E    +  where [] = <function glob at 0x041B36A8>('native.*.pyd')
E    +    where <function glob at 0x041B36A8> = glob.glob
E    +    and   'native.*.pyd' = <built-in method format of str object at 0x05D9D1B0>('pyd')
E    +      where <built-in method format of str object at 0x05D9D1B0> = 'native.*.{}'.format
---------------------------- Captured stdout call -----------------------------
running build_ext
building 'native' extension
creating build\temp.win32-3.7
creating build\temp.win32-3.7\Release
creating build\temp.win32-3.7\Release\build
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\bin\HostX86\x86\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -ID:\a\mypy\mypy\mypyc\lib-rt -ID:\a\mypy\mypy\.tox\py37\include -Ic:\hostedtoolcache\windows\python\3.7.9\x86\include -Ic:\hostedtoolcache\windows\python\3.7.9\x86\Include "-IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\ATLMFC\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" /Tcbuild\__native.c /Fobuild\temp.win32-3.7\Release\build\__native.obj /O0 /wd4102 /wd4101 /wd4146
__native.c

creating C:\Users\RUNNER~1\AppData\Local\Temp\mypy-test-wpgouzfj\tmp\build\lib.win32-3.7
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\bin\HostX86\x86\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:D:\a\mypy\mypy\.tox\py37\libs /LIBPATH:c:\hostedtoolcache\windows\python\3.7.9\x86\libs /LIBPATH:c:\hostedtoolcache\windows\python\3.7.9\x86 /LIBPATH:D:\a\mypy\mypy\.tox\py37\PCbuild\win32 "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\ATLMFC\lib\x86" "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\lib\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22000.0\\um\x86" /EXPORT:PyInit_native build\temp.win32-3.7\Release\build\__native.obj /OUT:build\lib.win32-3.7\native.pyd /IMPLIB:build\temp.win32-3.7\Release\build\native.lib
   Creating library build\temp.win32-3.7\Release\build\native.lib and object build\temp.win32-3.7\Release\build\native.exp

Generating code

Finished generating code

copying build\lib.win32-3.7\native.pyd -> 
---------------------------- Captured stderr call -----------------------------
cl : Command line warning D9002 : ignoring unknown option '/O0'

Here are the details from "Set up job":

Successful job

Current runner version: '2.289.1'
Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Virtual Environment
  Environment: windows-2022
  Version: 20220316.1
  Included Software: https://github.com/actions/virtual-environments/blob/win22/20220316.1/images/win/Windows2022-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win22%2F20220316.1
Virtual Environment Provisioner
  1.0.0.0-main-20220307-1

Failed job

Current runner version: '2.289.1'
Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Virtual Environment
  Environment: windows-2022
  Version: 20220316.1
  Included Software: https://github.com/actions/virtual-environments/blob/win22/20220316.1/images/win/Windows2022-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win22%2F20220316.1
Virtual Environment Provisioner
  1.0.0.0-main-20220307-1

Additional analysis

unknown option '/O0'`

There is a warning: cl : Command line warning D9002 : ignoring unknown option '/O0'

/O0 is being passed, which is the MYPYC_OPT_LEVEL being set here:

mypy/mypyc/test/test_run.py

Line 246 in 1480344

opt_level = int(os.environ.get('MYPYC_OPT_LEVEL', 0))

And being used here:

mypy/mypyc/build.py

Lines 525 to 529 in 1480344

    
           elif compiler.compiler_type == 'msvc': 
        
               if opt_level == '3': 
        
                   opt_level = '2' 
        
               cflags += [ 
        
                   '/O{}'.format(opt_level),

#12462 Changed build to not pass the argument at all if the opt level was 0, which removed the warning but didn't resolve the failures, I still this this is a good change though and have re-raised the change here #12468, slightly modified to map 0 to 1

In #12462 I tried to fix the /O0 issue; unfortunately it didn't fix the build.

setuptools had some releases around the time CI started failing, and since the bug seems to be related to the names of the generated extension modules and I think we use setuptools to handle the compilation, a setuptools change could cause a failure there.

The last pass in #12460 (comment) seems to be using setuptools 61.0.0, and the first fail seems to be using 61.1.0.

I haven't looked too much into what the cause of the failures is, but it seems worth trying to pin setuptools to 61.0.0 for now to see if that fixes the CI.

I started looking into it last night, but it seems that the test suite hangs and I cannot Ctrl+C when it fails for me, which is unfortunate. I will try to run things with failfast and see if using an older setuptools fixes things. (Hopefully it fixes the test hanging as well, but that may be another issue).

https://github.com/python/mypy/runs/5714129295?check_suite_focus=true

Doesn't seem like its setuptools?

virtualenv==20.13.4 went to virtualenv==20.14.0?

#12471 that didn't help either

I'm doing some experiments in #12476. The first result is that a running single test case still fails (-k testRunDataclass), which makes it faster to iterate using a PR. I also disabled all the other builds in this PR to avoid using lots of CI capacity.

Fix Windows test failures in CI

Fix Windows test failures in CI (#12477)

Seems to be fixed now. Thanks for the help!

It would be great if somebody could install the latest Windows wheel from https://github.com/mypyc/mypy_mypyc-wheels (once it's ready) and check that it works and is compiled.

It looks like this is an instance of pypa/setuptools#3219, and setuptools upgrading was what broke the CI. I think #12471 didn't help because the CI run still appears to be using setuptools version 61.2.0 because there are 2 pip install commands that try to install setuptools (here and here) and for some reason pip doesn't appear to be reinstalling setuptools for the second one.

It would be great if somebody could install the latest Windows wheel from https://github.com/mypyc/mypy_mypyc-wheels (once it's ready) and check that it works and is compiled.

I will test this once I am back in front of my computer in about an hour.

Things seem to be working as expected!

Navigation Menu

Repository navigation

Windows CI builds broken #12460

Additional analysis

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	elif compiler.compiler_type == 'msvc':
	if opt_level == '3':
	opt_level = '2'
	cflags += [
	'/O{}'.format(opt_level),

Windows CI builds broken #12460

Description

Activity

JelleZijlstra commented on Mar 26, 2022

emmatyping commented on Mar 26, 2022

KotlinIsland commented on Mar 27, 2022

Additional analysis

JelleZijlstra commented on Mar 27, 2022

pranavrajpal commented on Mar 27, 2022

emmatyping commented on Mar 27, 2022

jhance commented on Mar 28, 2022

KotlinIsland commented on Mar 28, 2022

KotlinIsland commented on Mar 28, 2022

JukkaL commented on Mar 28, 2022

JukkaL commented on Mar 28, 2022

pranavrajpal commented on Mar 28, 2022

emmatyping commented on Mar 28, 2022

emmatyping commented on Mar 28, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions