Repository navigation
Bug: git+https requirements fail to install when using python3.12 container #675
Copy link
Copy link
Open
Description
Description:
sam build fails when requirements.txt contains git+https links, python3.12 only
Steps to reproduce:
- SAM init hello world example
- Runtime: python3.12; Architectures: - arm64
- Add to requirements.txt:
gendercomputer@git+https://github.com/tue-mdse/genderComputer.git
sam build --use-container --debug
Observed result:
Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image......
2024-08-14 13:41:13,759 | Mounting .../sam-app/hello_world as /tmp/samcli/source:ro,delegated, inside runtime container
Using the request object from command line argument
...
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
pip stdout: b"Collecting gendercomputer@ git+https://github.com/tue-mdse/genderComputer.git (from -r /tmp/samcli/source/requirements.txt (line 1))\n Cloning https://github.com/tue-mdse/genderComputer.git to /tmp/pip-download-cnqo230n/gendercomputer_83fc8aa3dae241db97874f2c91a268bd\n Resolved https://github.com/tue-mdse/genderComputer.git to commit f6267615517913e53cb0b882b248f1c2e11b8bbc\n Installing build dependencies: started\n Installing build dependencies: finished with status 'done'\n Getting requirements to build wheel: started\n Getting requirements to build wheel: finished with status 'done'\n Preparing metadata (pyproject.toml): started\n Preparing metadata (pyproject.toml): finished with status 'done'\nCollecting unidecode==1.3.2 (from gendercomputer@ git+https://github.com/tue-mdse/genderComputer.git->-r /tmp/samcli/source/requirements.txt (line 1))\n Downloading Unidecode-1.3.2-py3-none-any.whl.metadata (13 kB)\nCollecting nameparser==1.0.6 (from gendercomputer@ git+https://github.com/tue-mdse/genderComputer.git->-r /tmp/samcli/source/requirements.txt (line 1))\n Downloading nameparser-1.0.6-py2.py3-none-any.whl.metadata (5.9 kB)\nDownloading nameparser-1.0.6-py2.py3-none-any.whl (23 kB)\nDownloading Unidecode-1.3.2-py3-none-any.whl (235 kB)\n \xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81 235.7/235.7 kB 5.9 MB/s eta 0:00:00\nSaved /tmp/samcli/scratch/gendercomputer-0.1.zip\nSaved /tmp/samcli/scratch/nameparser-1.0.6-py2.py3-none-any.whl\nSaved /tmp/samcli/scratch/Unidecode-1.3.2-py3-none-any.whl\nSuccessfully downloaded gendercomputer nameparser unidecode\n"
pip stderr: b"WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.\nPlease see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.\nTo avoid this problem you can invoke Python with '-m pip' instead of running pip directly.\n Running command git clone --filter=blob:none --quiet https://github.com/tue-mdse/genderComputer.git /tmp/pip-download-cnqo230n/gendercomputer_83fc8aa3dae241db97874f2c91a268bd\n\n[notice] A new release of pip is available: 24.0 -> 24.2\n[notice] To update, run: pip install --upgrade pip\n"
Non zero rc (1) from the setup.py egg_info command: b'Traceback (most recent call last):\n File "<string>", line 1, in <module>\nModuleNotFoundError: No module named \'setuptools\'\n'
Using fallback location for PKG-INFO file in package directory: /tmp/tmpdamop0ow/gendercomputer
PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 374, in run
action.execute()
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 71, in execute
package_builder.build_dependencies(
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 262, in build_site_packages
wheels, packages_without_wheels = self._download_dependencies(scratch_directory, requirements_filepath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 286, in _download_dependencies
deps = self._download_all_dependencies(requirements_filename, directory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 370, in _download_all_dependencies
Package(directory, filename, self.python_exe)
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 522, in __init__
self._name, self._version = self._calculate_name_and_version()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 564, in _calculate_name_and_version
name, version = info_fetcher.get_package_name_and_version(sdist_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 706, in get_package_name_and_version
pkg_info_filepath = self._get_pkg_info_filepath(package_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 618, in _get_pkg_info_filepath
raise UnsupportedPackageError(self._osutils.basename(package_dir))
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: gendercomputer
Builder workflow failed
Traceback (most recent call last):
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 374, in run
action.execute()
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 71, in execute
package_builder.build_dependencies(
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 262, in build_site_packages
wheels, packages_without_wheels = self._download_dependencies(scratch_directory, requirements_filepath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 286, in _download_dependencies
deps = self._download_all_dependencies(requirements_filename, directory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 370, in _download_all_dependencies
Package(directory, filename, self.python_exe)
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 522, in __init__
self._name, self._version = self._calculate_name_and_version()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 564, in _calculate_name_and_version
name, version = info_fetcher.get_package_name_and_version(sdist_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 706, in get_package_name_and_version
pkg_info_filepath = self._get_pkg_info_filepath(package_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 618, in _get_pkg_info_filepath
raise UnsupportedPackageError(self._osutils.basename(package_dir))
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: gendercomputer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/__main__.py", line 117, in main
builder.build(
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/builder.py", line 172, in build
return workflow.run()
^^^^^^^^^^^^^^
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 116, in wrapper
func(self, *args, **kwargs)
File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 385, in run
raise WorkflowUnknownError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowUnknownError: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: gendercomputer
2024-08-14 13:41:17,306 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for
package: gendercomputer"}}
2024-08-14 13:41:17,634 | Exception raised during the execution
Build Failed
Expected result:
Build successful.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
Notice No module named \'setuptools\' in the logs.
The issue is not reproducible when changing Runtime to python3.11.
{
"version": "1.121.0",
"system": {
"python": "3.12.5",
"os": "macOS-14.6.1-arm64-arm-64bit"
},
"additional_dependencies": {
"docker_engine": "27.1.1",
"aws_cdk": "Not available",
"terraform": "1.9.4"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
Reactions are currently unavailable
Activity
hawflau commentedon Aug 15, 2024
@zhemaituk thanks for raising the issue. I verified that the same issue happened if I ran
sam build --use-containerfor Python3.12. However, there's no issue if I switch Python3.11 or if I runsam buildwithout--use-container.Marking it as bug.
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Use Makefile to get around SAM issue
Dizotoff commentedon Dec 17, 2024
I'm experiencing a similar issue when not using the
--use-containerflag. It fails a certain percentage of the time, but not alwayskru2710shna commentedon Feb 4, 2025
Quick Questions: Do you need to assign the issue to me , or i can hop on directly ?
johnc44 commentedon Mar 3, 2025
I am also seeing the same problem when using "sam build" without "--use-container". This happens every time. We have observed it on different machines (Mac and Windows/WSL) including with fresh versions of our code, sno nothing should be getting cached. It also fails on python 3.13 but works fine on 3.11.
--use-container doesn't get that far as we are using a private repo which needs access to our ssh keys.
For now we are sticking to 3.11.
nowei commentedon Mar 3, 2025
I'm also running into this issue with
--use-containeronpython3.12.It also breaks when I try it with python 3.11 when I only have a
pyproject.toml, but this seems to work properly on 3.11 when I have apyproject.tomlwith asetup.py.cmurtaugh commentedon Apr 19, 2025
I'm also seeing this with
sam buildonpython3.13.