RTD uses a cached virtualenv, which makes subpackage content stale

@jonparrott

(I recently noticed this, though it's been happening since we switched to subpackages.) For example, in a recent build:

Processing ./core
  Requirement already satisfied (use --upgrade to upgrade):
  google-cloud-core==0.20.0 from file:///.../checkouts/latest/core in
  /home/.../envs/latest/lib/python2.7/site-packages
  (from -r docs/requirements.txt (line 1))
Processing ./bigquery
  Requirement already satisfied (use --upgrade to upgrade):
  google-cloud-bigquery==0.20.0 from file:///.../checkouts/latest/bigquery in
  /home/.../envs/latest/lib/python2.7/site-packages
  (from -r docs/requirements.txt (line 2))

If we were building these on a machine we owned, it would be no big deal, we'd force the packages to reinstall (--ignore-installed) or we'd just create a brand new virtual environment. However, on RTD the only toggles we have are docs/requirements.txt, docs/conf.py and setup.py (at the root).

I spent about 30m messing with the requirements.txt trying to convince it to accept the flag, and trying to see if there was any other hack (by specifying a git URL instead of a filesystem path). The "recommeneded" RTD fix is to install as editable, but pip and setuptools are b0rken with editable installs in namespace packages. Yay!

/cc @jonparrott

pip and setuptools are b0rken with editable installs in namespace packages.

I don't believe that is universally true: pip -e . screws them up, but python setup.py develop does not.

@tseaver

@tseaver I'll look into it right now.

@tseaver

@tseaver AFAIK pip -e calls setup.py develop. Either way, using an isolated setup.py develop the "last one wins" / "first egg link found wins" and the namespace packages are not actually namespace packages.

$ virtualenv venv
$ source venv/bin/activate
$ export ROOT=$(pwd)
(venv) $ (cd ${HOME} && ${ROOT}/venv/bin/pip freeze)
(venv) $ (cd core && ${ROOT}/venv/bin/python setup.py develop)
(venv) $ (cd ${HOME} && ${ROOT}/venv/bin/pip freeze)
-e git+git@github.com:GoogleCloudPlatform/google-cloud-python.git@392f0f8b2ffaee98766f2851df72221051124100#egg=google_cloud_core&subdirectory=../../../google-cloud-python/core
six==1.10.0
(venv) $ (cd datastore && ${ROOT}/venv/bin/python setup.py develop)
(venv) $ (cd ${HOME} && ${ROOT}/venv/bin/pip freeze)
-e git+git@github.com:GoogleCloudPlatform/google-cloud-python.git@392f0f8b2ffaee98766f2851df72221051124100#egg=google_cloud_core&subdirectory=../../../google-cloud-python/core
-e git+git@github.com:GoogleCloudPlatform/google-cloud-python.git@392f0f8b2ffaee98766f2851df72221051124100#egg=google_cloud_datastore&subdirectory=../../../google-cloud-python/datastore
six==1.10.0
(venv) $ ls -1 ${ROOT}/venv/lib/python2.7/site-packages/google*
venv/lib/python2.7/site-packages/google-cloud-core.egg-link
venv/lib/python2.7/site-packages/google-cloud-datastore.egg-link
(venv) $ (cd ${HOME} && ${ROOT}/venv/bin/python)
>>> import google
>>> google
<module 'google' from '.../google-cloud-python/datastore/google/__init__.py'>
>>> import google.cloud
>>> google.cloud
<module 'google.cloud' from '.../google-cloud-python/datastore/google/cloud/__init__.py'>
>>> import google.protobuf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named protobuf
(venv) $ deactivate
$ unset ROOT
$ rm -fr venv  # Good riddance

Mixing pip and direct setup.py <anything> has never worked properly for namespace packages. If you install via setup.py develop alone, with no pip in sight, then namespace packages work. E.g.:

$ /opt/Python-2.7.11/bin/virtualenv --no-pip /tmp/gcp-2582
New python executable in /tmp/gcp-2582/bin/python
Installing setuptools, wheel...done.
$ cd /tmp/gcp-2582/
$ mkdir src
$ cd src
$ git clone git@github.com:zopefoundation/zope.component
Cloning into 'zope.component'...
remote: Counting objects: 1420, done.
remote: Total 1420 (delta 0), reused 0 (delta 0), pack-reused 1420
Receiving objects: 100% (1420/1420), 384.54 KiB | 0 bytes/s, done.
Resolving deltas: 100% (719/719), done.
Checking connectivity... done.
$ cd zope.component/
$ ../../bin/python setup.py develop
running develop
running egg_info
creating src/zope.component.egg-info
...
Installed /tmp/gcp-2582/lib/python2.7/site-packages/zope.interface-4.3.2-py2.7-linux-x86_64.egg
Searching for setuptools==28.6.1
Best match: setuptools 28.6.1
Adding setuptools 28.6.1 to easy-install.pth file
Installing easy_install-3.5 script to /tmp/gcp-2582/bin
Installing easy_install script to /tmp/gcp-2582/bin

Using /tmp/gcp-2582/lib/python2.7/site-packages
Finished processing dependencies for zope.component==4.4.0.dev0
$ cd ../..
$ bin/python
Python 2.7.11 (default, Feb 22 2016, 07:44:44) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import zope.interface
>>> import zope.component
>>> import zope
>>> zope.__path__
['/tmp/gcp-2582/src/zope.component/src/zope', '/tmp/gcp-2582/lib/python2.7/site-packages/zope.event-4.2.0-py2.7.egg/zope', '/tmp/gcp-2582/lib/python2.7/site-packages/zope.interface-4.3.2-py2.7-linux-x86_64.egg/zope']

@tseaver

@tseaver We got down the rabbit hole here. We're not going to change the fact that RTD uses pip.

Also, your example doesn't (explicitly, not sure what the deps do) use co-tenant namespace packages, which is where the problem presents itself.

chore: bump minimum version of google-cloud-access-context-manager (#…

Navigation Menu

Repository navigation

RTD uses a cached virtualenv, which makes subpackage content stale #2582

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RTD uses a cached virtualenv, which makes subpackage content stale #2582

Description

Activity

tseaver commented on Oct 21, 2016

dhermes commented on Oct 21, 2016

dhermes commented on Oct 21, 2016

tseaver commented on Oct 21, 2016

dhermes commented on Oct 21, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions