Repository navigation
Missing unit test dependencies on Linux? #12903
Description
Bug Report
It appears that there are several missing dependencies in test-requirements.txt, including:
- gcc and make.
- Some other dependency, whose symptom is that
#include <Python.h>fails. For example:
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/edreamleo/mypy/mypyc/lib-rt -I/home/edreamleo/mypy/venv/include -I/usr/include/python3.8 -c build/__native.c -o build/temp.linux-x86_64-3.8/build/__native.o -O0 -g1 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option -Wno-unused-but-set-variable
---------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------
In file included from build/__native.c:1:
/home/edreamleo/mypy/mypyc/lib-rt/init.c:1:10: fatal error: Python.h: No such file or directory
1 | #include <Python.h>
| ^~~~~~~~~~
compilation terminated.To Reproduce
Rather than including multiple (huge) console logs, I'll summarize. I'll be happy to run any tests that you might want.
apt-get remove gcc
apt-get remove make
<< install mypy >>
source venv/bin/activate
python3 -m pytestHundreds of tests will fail, complaining either that x86_64-linux-gnu-gcc does not exist or that make does not exist.
After:
apt-get install gcc
apt-get install makeI still get many failures: 346 failed, 9620 passed, 370 skipped, 9 xfailed,
with many tests failing because #include <Python.h> fails, as shown above.
Expected Behavior
All unit tests pass (in the venv) after installing mypy.
Actual Behavior
See above.
Your Environment
mypy installation:
Details
(venv) edreamleo@edreamleo-MS-7D09:~/mypy$ python3 -m pip install -r test-requirements.txt
Ignoring typed-ast: markers 'python_version < "3.8"' don't match your environment
Ignoring typed-ast: markers 'python_version < "3.8"' don't match your environment
Ignoring filelock: markers 'python_version < "3.7"' don't match your environment
Collecting typing_extensions>=3.10
Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting mypy_extensions>=0.4.3
Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting tomli>=1.1.0
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting types-typed-ast<1.6.0,>=1.5.0
Using cached types_typed_ast-1.5.6-py3-none-any.whl (5.9 kB)
Collecting attrs>=18.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting filelock>=3.0.0
Using cached filelock-3.7.0-py3-none-any.whl (10 kB)
Collecting flake8==3.9.2
Using cached flake8-3.9.2-py2.py3-none-any.whl (73 kB)
Collecting flake8-bugbear==22.3.20
Using cached flake8_bugbear-22.3.20-py3-none-any.whl (19 kB)
Collecting flake8-pyi>=20.5
Using cached flake8_pyi-22.5.1-py37-none-any.whl (20 kB)
Collecting lxml>=4.4.0
Using cached lxml-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.9 MB)
Collecting psutil>=4.0
Using cached psutil-5.9.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (284 kB)
Collecting pytest>=6.2.4
Using cached pytest-7.1.2-py3-none-any.whl (297 kB)
Collecting pytest-xdist>=1.34.0
Using cached pytest_xdist-2.5.0-py3-none-any.whl (41 kB)
Collecting pytest-forked<2.0.0,>=1.3.0
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting pytest-cov<3.0.0,>=2.10.0
Using cached pytest_cov-2.12.1-py2.py3-none-any.whl (20 kB)
Collecting py>=1.5.2
Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting typed_ast<2,>=1.5.4
Using cached typed_ast-1.5.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (897 kB)
Collecting virtualenv>=20.6.0
Using cached virtualenv-20.14.1-py2.py3-none-any.whl (8.8 MB)
Requirement already satisfied: setuptools!=50 in ./venv/lib/python3.8/site-packages (from -r test-requirements.txt (line 19)) (44.0.0)
Collecting importlib-metadata<5.0.0,>=4.6.1
Using cached importlib_metadata-4.11.4-py3-none-any.whl (18 kB)
Collecting pyflakes<2.4.0,>=2.3.0
Using cached pyflakes-2.3.1-py2.py3-none-any.whl (68 kB)
Collecting pycodestyle<2.8.0,>=2.7.0
Using cached pycodestyle-2.7.0-py2.py3-none-any.whl (41 kB)
Collecting mccabe<0.7.0,>=0.6.0
Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting ast-decompiler<0.5.0,>=0.4.0; python_version < "3.9"
Using cached ast_decompiler-0.4.0-py2.py3-none-any.whl (12 kB)
Collecting packaging
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pluggy<2.0,>=0.12
Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting iniconfig
Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting toml
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting coverage>=5.2.1
Using cached coverage-6.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)
Collecting platformdirs<3,>=2
Using cached platformdirs-2.5.2-py3-none-any.whl (14 kB)
Collecting six<2,>=1.9.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting distlib<1,>=0.3.1
Using cached distlib-0.3.4-py2.py3-none-any.whl (461 kB)
Collecting zipp>=0.5
Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Installing collected packages: typing-extensions, mypy-extensions, tomli, types-typed-ast, attrs, filelock, pyflakes, pycodestyle, mccabe, flake8, flake8-bugbear, ast-decompiler, flake8-pyi, lxml, psutil, pyparsing, packaging, py, pluggy, iniconfig, pytest, execnet, pytest-forked, pytest-xdist, toml, coverage, pytest-cov, typed-ast, platformdirs, six, distlib, virtualenv, zipp, importlib-metadata
Successfully installed ast-decompiler-0.4.0 attrs-21.4.0 coverage-6.4 distlib-0.3.4 execnet-1.9.0 filelock-3.7.0 flake8-3.9.2 flake8-bugbear-22.3.20 flake8-pyi-22.5.1 importlib-metadata-4.11.4 iniconfig-1.1.1 lxml-4.8.0 mccabe-0.6.1 mypy-extensions-0.4.3 packaging-21.3 platformdirs-2.5.2 pluggy-1.0.0 psutil-5.9.1 py-1.11.0 pycodestyle-2.7.0 pyflakes-2.3.1 pyparsing-3.0.9 pytest-7.1.2 pytest-cov-2.12.1 pytest-forked-1.4.0 pytest-xdist-2.5.0 six-1.16.0 toml-0.10.2 tomli-2.0.1 typed-ast-1.5.4 types-typed-ast-1.5.6 typing-extensions-4.2.0 virtualenv-20.14.1 zipp-3.8.0
(venv) edreamleo@edreamleo-MS-7D09:~/mypy$ python3 -m pip install -e .
Obtaining file:///home/edreamleo/mypy
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Requirement already satisfied: typing-extensions>=3.10 in ./venv/lib/python3.8/site-packages (from mypy==0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879) (4.2.0)
Requirement already satisfied: tomli>=1.1.0; python_version < "3.11" in ./venv/lib/python3.8/site-packages (from mypy==0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879) (2.0.1)
Requirement already satisfied: mypy-extensions>=0.4.3 in ./venv/lib/python3.8/site-packages (from mypy==0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879) (0.4.3)
Installing collected packages: mypy
Attempting uninstall: mypy
Found existing installation: mypy 0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879
Not uninstalling mypy at /home/edreamleo/mypy, outside environment /home/edreamleo/mypy/venv
Can't uninstall 'mypy'. No files were found to uninstall.
Running setup.py develop for mypy
Successfully installed mypy
(venv) edreamleo@edreamleo-MS-7D09:~/mypy$ hash -rEnvironment:
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linuxpip list (outer):
Details
Package Version Editable project location
---------------------- -------------------- --------------------------
appdirs 1.4.3
apturl 0.5.2
astroid 2.11.5
attrs 21.4.0
bcrypt 3.1.7
blinker 1.4
Brlapi 0.7.0
certifi 2019.11.28
chardet 3.0.4
Click 7.0
colorama 0.4.3
command-not-found 0.3
cryptography 2.8
cupshelpers 1.0
dbus-python 1.2.16
defer 1.0.6
dill 0.3.4
distlib 0.3.0
distro 1.4.0
distro-info 0.23ubuntu1
docutils 0.18.1
duplicity 0.8.12.0
entrypoints 0.3
fasteners 0.14.1
filelock 3.0.12
future 0.18.2
gitdb 4.0.9
GitPython 3.1.27
httplib2 0.14.0
idna 2.8
importlib-metadata 1.5.0
iniconfig 1.1.1
isort 5.10.1
keyring 18.0.1
language-selector 0.1
launchpadlib 1.10.13
lazr.restfulclient 0.14.2
lazr.uri 1.0.3
lazy-object-proxy 1.7.1
lockfile 0.12.2
louis 3.12.0
macaroonbakery 1.3.1
Mako 1.1.0
MarkupSafe 1.1.0
mccabe 0.7.0
monotonic 1.5
more-itertools 4.2.0
netifaces 0.10.4
oauthlib 3.1.0
olefile 0.46
packaging 21.3
paramiko 2.6.0
pexpect 4.6.0
Pillow 7.0.0
pip 22.0.4
platformdirs 2.5.2
pluggy 1.0.0
protobuf 3.6.1
py 1.11.0
py-cpuinfo 8.0.0
pycairo 1.16.2
pycups 1.9.73
pyenchant 3.2.2
PyGObject 3.36.0
PyJWT 1.7.1
pylint 2.14.0b1 /home/edreamleo/ekr-pylint
pymacaroons 0.13.0
PyNaCl 1.3.0
pyparsing 3.0.8
PyQt6 6.3.0
PyQt6-Qt6 6.3.0
PyQt6-sip 13.3.1
PyQt6-WebEngine 6.3.0
PyQt6-WebEngine-Qt6 6.3.0
pyRFC3339 1.1
pytest 7.1.2
pytest-benchmark 3.4.1
pytest-timeout 2.1.0
python-apt 2.0.0+ubuntu0.20.4.7
python-dateutil 2.7.3
python-debian 0.1.36ubuntu1
pytz 2019.3
pyxdg 0.26
PyYAML 5.3.1
reportlab 3.5.34
requests 2.22.0
requests-unixsocket 0.2.0
SecretStorage 2.3.1
setuptools 62.1.0
simplejson 3.16.0
six 1.14.0
smmap 5.0.0
systemd-python 234
toml 0.10.0
tomli 2.0.1
tomlkit 0.10.2
tox 3.13.2
typing_extensions 4.2.0
ubuntu-advantage-tools 27.8
ubuntu-drivers-common 0.0.0
ufw 0.36
unattended-upgrades 0.1
urllib3 1.25.8
usb-creator 0.3.7
virtualenv 20.0.17
wadllib 1.3.3
wheel 0.37.1
wrapt 1.14.1
xkit 0.0.0
zipp 1.0.0(venv) pip list:
Details
edreamleo@edreamleo-MS-7D09:~$ cd mypy
edreamleo@edreamleo-MS-7D09:~/mypy$ source venv/bin/activate
(venv) edreamleo@edreamleo-MS-7D09:~/mypy$ pip list
Package Version Location
------------------ -------------------------------------------------- --------------------
ast-decompiler 0.4.0
attrs 21.4.0
coverage 6.4
distlib 0.3.4
execnet 1.9.0
filelock 3.7.0
flake8 3.9.2
flake8-bugbear 22.3.20
flake8-pyi 22.5.1
importlib-metadata 4.11.4
iniconfig 1.1.1
lxml 4.8.0
mccabe 0.6.1
mypy 0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879 /home/edreamleo/mypy
mypy-extensions 0.4.3
packaging 21.3
pip 20.0.2
pkg-resources 0.0.0
platformdirs 2.5.2
pluggy 1.0.0
psutil 5.9.1
py 1.11.0
pycodestyle 2.7.0
pyflakes 2.3.1
pyparsing 3.0.9
pytest 7.1.2
pytest-cov 2.12.1
pytest-forked 1.4.0
pytest-xdist 2.5.0
setuptools 44.0.0
six 1.16.0
toml 0.10.2
tomli 2.0.1
typed-ast 1.5.4
types-typed-ast 1.5.6
typing-extensions 4.2.0
virtualenv 20.14.1
zipp 3.8.0
(venv) edreamleo@edreamleo-MS-7D09:~/mypy$ I have spent several hours trying various permutations. This report is far from the last word. To repeat, I'll be happy to run any tests that you might want.
Activity
JelleZijlstra commentedon May 29, 2022
Thanks for reporting! You probably need to install something like
python3-devforPython.hto be available. Agree that we should make it explicit that gcc and make are required.edreamleo commentedon May 29, 2022
@JelleZijlstra Thanks. I'll try python3-dev later today.
edreamleo commentedon May 29, 2022
@JelleZijlstra
apt-get install python3-devfixed all but one of the test failures:Details
So it looks like g++ must be installed too.
Indeed, all tests pass for me after
apt-get install g++:9966 passed, 370 skipped, 9 xfailedemmatyping commentedon May 29, 2022
I think a helper to try to compile a simple file that includes Python.h in conftest that can be used to skip the mypyc tests makes sense? Unless we think that the mypyc tests should always be run, in which case we can just show a helpful error.