not compatible with Google Cloud Containers

I'm using the google/python-runtime Docker image to run an app on Google Managed VMs, and need to access the datastore. The obvious solution seems to be gcloud-python as it's so much easier to use than the datastore python library.

So here's what happens. In my requirements.txt file I have:
httplib2
tornado
gcloud

PIP installs everything OK except it fails when installing cffi. Here is a log:

Processing cffi-0.8.6.tar.gz

Writing /tmp/easy_install-MCEL5F/cryptography-0.6.1/temp/easy_install-ZPcCkB/cffi-0.8.6/setup.cfg

Running cffi-0.8.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-MCEL5F/cryptography-0.6.1/temp/easy_install-ZPcCkB/cffi-0.8.6/egg-dist-tmp-fmCpCO

compiling '_configtest.c':

__thread int some_threadlocal_variable_42;



gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -c _configtest.c -o _configtest.o

success!

removing: _configtest.c _configtest.o

c/_cffi_backend.c:13:17: fatal error: ffi.h: No such file or directory

compilation terminated.

Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/env/build/gcloud/setup.py", line 49, in <module>

    'Topic :: Internet',

  File "/usr/lib/python2.7/distutils/core.py", line 112, in setup

    _setup_distribution = dist = klass(attrs)

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 260, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 285, in fetch_build_eggs

    replace_conflicting=True

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 631, in resolve

    dist = best[req.key] = env.best_match(req, ws, installer)

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 871, in best_match

    return self.obtain(req, installer)

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 883, in obtain

    return installer(requirement)

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 335, in fetch_build_egg

    return cmd.easy_install(req)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 595, in easy_install

    return self.install_item(spec, dist.location, tmpdir, deps)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 625, in install_item

    dists = self.install_eggs(spec, download, tmpdir)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 822, in install_eggs

    return self.build_and_install(setup_script, setup_base)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1028, in build_and_install

    self.run_setup(setup_script, setup_base, args)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1013, in run_setup

    run_setup(setup_script, args)

  File "/env/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 50, in run_setup

    lambda: execfile(

  File "/env/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 100, in run

    return func()

  File "/env/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 52, in <lambda>

    {'__file__':setup_script, '__name__':'__main__'}

  File "setup.py", line 338, in <module>



  File "/usr/lib/python2.7/distutils/core.py", line 112, in setup

    _setup_distribution = dist = klass(attrs)

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 260, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 285, in fetch_build_eggs

    replace_conflicting=True

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 631, in resolve

    dist = best[req.key] = env.best_match(req, ws, installer)

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 871, in best_match

    return self.obtain(req, installer)

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 883, in obtain

    return installer(requirement)

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 335, in fetch_build_egg

    return cmd.easy_install(req)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 595, in easy_install

    return self.install_item(spec, dist.location, tmpdir, deps)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 625, in install_item

    dists = self.install_eggs(spec, download, tmpdir)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 822, in install_eggs

    return self.build_and_install(setup_script, setup_base)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1028, in build_and_install

    self.run_setup(setup_script, setup_base, args)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1016, in run_setup

    raise DistutilsError("Setup script exited with %s" % (v.args[0],))

distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /env/build/gcloud
Storing debug log for failure in /root/.pip/pip.log

This is kind of a road block for me, is there anything that can be done to make it installable on google containers?

FWIW, cffi is an indirect dependency (via pyOpenSSL and cryptography), which are required to make the OAuth flows work.

@artooro

@artooro You need to apt-get install libffi-dev (and maybe also libffi6).

As for this library:

We could (should) make the crypto stuff a conditional dependency just like it is in oauth2client:
https://github.com/google/oauth2client/blob/7d3b1ae4f7a87de20dcf2a7e9517479cda6311dc/oauth2client/client.py#L43

Compare:
https://github.com/google/oauth2client/blob/7d3b1ae4f7a87de20dcf2a7e9517479cda6311dc/setup.py#L34
https://github.com/GoogleCloudPlatform/gcloud-python/blob/01a1eeab3d6d66acf4a6b83fb476a48a4c8c0247/setup.py#L20

With that fix, you'd need to get a token from the environment, which might be possible?

@artooro

#335 will fix all of this, also enabling support for Compute Engine Service Accounts (no need for keys).

@artooro for the time being you should use a custom dockerfile and follow @dhermes instructions.

@dhermes

I'll use the suggestion from @dhermes to install the required packages. Thanks!

@artooro

@artooro That's great!

The missing ffi.h file may not be the only issue, though adding libffi-dev will likely fix the cffi install issue. If you encounter more, please post here so we can have a better idea of what we're up against.

I added this to my Dockerfile with success:

RUN apt-get install -y libffi-dev
RUN /env/bin/pip install gcloud

Note: Can't have gcloud in requirements.txt because it is processed by the docker image before it runs the apt-get install.

chore(python): update .pre-commit-config.yaml to use black==22.3.0 (#348

chore: use gapic-generator-python 0.56.2 (#348)

fix: disable always_use_jwt_access (#348)

Test and impl for recursive and inclusive resource path helper funcs (#…

fix: resolve issue where marshal fails with cross api dependency (#348)

build: use trampoline_v2 for python samples and allow custom dockerfi…

chore(deps): update dependency google-cloud-datastore to v2.8.1 (#348)

chore(python): update unittest workflow template (#348)

chore(main): release 1.5.3 (#348)

chore(deps): update dependency backoff to v1.11.1 (#348)

feat: `Key.to_legacy_urlsafe()` (#348)

chore(deps): update dependency backoff to v1.11.1 (#348)

Add support for imersonated_credentials.Sign, IDToken (#348)

build(deps): bump urllib3 from 1.26.12 to 1.26.18 (#348)

build(deps): Bump django from 2.2.28 to 3.2.24 in /.kokoro (#348)

Navigation Menu

Repository navigation

not compatible with Google Cloud Containers #348

25 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

not compatible with Google Cloud Containers #348

Description

Activity

tseaver commented on Nov 6, 2014

dhermes commented on Nov 6, 2014

silvolu commented on Nov 6, 2014

artooro commented on Nov 6, 2014

dhermes commented on Nov 6, 2014

artooro commented on Nov 6, 2014

25 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions