Repository navigation
Multiple python versions in one environment #98
Description
Activity
I'd like to use
tox --parallelfor linting py2 and py3 as it's faster than separate machines. So I'd also like to be able to install both at onceReacted by Alex Lowe and Joshua ShieldsI would go for installing x86 and x64 python's at same time as well, expecially if it means generating python c extension modules depending on the platform.
Reacted by wookie184 and AraHaanAny practical workarounds? I need this in order to make use of combined coverage reporting and I really want to avoid going towards collecting artifacts and combining them later, it would add a lot of complexity to the pipelines and increase the lock-in effect.
My goal is to read the python versions I want to test against from a config file, so I don't have to update the workflow(s) later on, when I only want to add or remove a python version. As I currently cannot just pass the list of python versions to
actions/setup-pythonaction I separate them in another job beforehand and put them in different variables like so:jobs: pyproject-config: name: Get config from pyproject.toml runs-on: ubuntu-latest outputs: test_python_version_1: ${{ steps.get-config.outputs.test_python_version_1 }} test_python_version_2: ${{ steps.get-config.outputs.test_python_version_2 }} ... steps: - uses: actions/checkout@v2 - uses: actions/setup-python@v2 - name: Install dependencies run: | python -m pip install -U pip pip install -U tomlkit - name: Get config id: get-config shell: python run: | import tomlkit with open("pyproject.toml") as pyproject_file: PYPROJECT = tomlkit.parse(pyproject_file.read()) test_ver = PYPROJECT["tool"]["_testing"]["ci_test_python_versions"] if len(test_ver) >= 1: print(f"::set-output name=test_python_version_1::{test_ver[0]}") if len(test_ver) >= 2: print(f"::set-output name=test_python_version_2::{test_ver[1]}") ...
The test job than depends on the upper job and initializes the python versions like so:
jobs: test-code: needs: pyproject-config ... steps: - uses: actions/setup-python@v2 with: - uses: actions/setup-python@v2 with: python-version: ${{ needs.pyproject-config.outputs.test_python_version_1 }} if: needs.pyproject-config.outputs.test_python_version_1 - uses: actions/setup-python@v2 with: python-version: ${{ needs.pyproject-config.outputs.test_python_version_2 }} if: needs.pyproject-config.outputs.test_python_version_2 ...
I currently does that for 10 possible python versions and have 7 specified (incl. PyPy) so 3 steps are skipped.
I was able to use the Miniconda action as shown below to accomplish the OP's mention for tox with multiple Pythons in one invocation, where I am testing a Python library for 3.7, 3.8, and 3.9. Note the
bash -l {0}is required for the Miniconda action to work properly across steps. Below, thetestenvenvironment is automatically active at each step. Essentially, I just create the extra Python environments using conda and append the paths for the Python binaries in those environments to the PATH when running tox in thetestingenvironment.name: build_tests on: [push] jobs: build: strategy: matrix: os: [ubuntu-latest, macos-latest, windows-latest] name: Run tox build tests runs-on: ${{ matrix.os }} defaults: run: shell: bash -l {0} steps: - name: Setup repo uses: actions/checkout@v2 - uses: conda-incubator/setup-miniconda@v2 with: miniconda-version: latest python-version: 3.8 activate-environment: testenv channels: conda-forge - name: Setup conda python environments run: | conda config --set always_yes yes --set changeps1 no conda install -q -c conda-forge tox conda create -q -n py37 python=3.7 conda create -q -n py39 python=3.9 - name: Run pytest env: TOXENV: py37,py38,py39 run: | export PY37_PATH=$CONDA/envs/py37/bin export PY39_PATH=$CONDA/envs/py39/bin export PATH=$PATH:$PY37_PATH:$PY39_PATH tox - name: Run code checks if: ${{ matrix.os == 'ubuntu-latest' }} env: TOXENV: typing,pypi-description,manifest,precom run: tox
Reacted by Niklas RosensteinHello everyone. We released a new version with support to install multiple python versions and updated the major tag.
For now I'm going to close the issue. If you have any concerns feel free to ping us.
Reacted by Chris Barnes, Sorin Sbarnea, Kamil Dybicz, AraHaan and Carter FendleyHello everyone. We released a new version with support to install multiple python versions and updated the major tag.
For now I'm going to close the issue. If you have any concerns feel free to ping us.
Does this include the ability to specify multiple architectures as well in it too? like x86 and x64 where it will install each version with each architecture listed?
Some tools are designed to have access to multiple python interpreters, and then create their own virtualenvs. Tox is a common example (although there are other ways to use tox in Actions). Less common, but my use case, is maturin https://github.com/PyO3/maturin , which is used to build rust crates with python bindings. It's a lot quicker to do them all at once rather than in different jobs, as most of the compiled code is shared.
It would be helpful if
setup-pythonallowed users to pass an array of python versions and installed all of them (which one takespythonis up for debate, butpython3.7,python3.8etc should be fine).