Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Travis's curl is too old, causing spurious fetch-test.sh issues #4645

Closed

Description

@mnordhoff

Spun off from comments on pull #4619. CC @J0WI.

Travis runs Ubuntu 14.04, which is 1.9 years old and apparently hopelessly out-of-date in Internet-years. ;-) Its (|lib|py)curl version is too old for modern features that we need, causing spurious fetch-test.sh failures.

ECDSA

First and critically, it fails on ECDSA websites, such as an ever-growing number of CloudFlare clients. For example, the Archive.is ruleset updated in #4599, which passed Travis a week ago, would fail fetch-test.sh if proposed today:

INFO Finished comparing http://archive.today/ -> https://archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://blog.archive.today/ -> https://blog.archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.today/ -> https://www.archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://archive.is/ -> https://archive.is/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.is/ -> https://www.archive.is/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://archive.li/ -> https://archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://blog.archive.li/ -> https://blog.archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.li/ -> https://www.archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://archive.today/ => https://archive.today/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://blog.archive.today/ => https://blog.archive.today/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://www.archive.today/ => https://www.archive.today/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://archive.li/ => https://archive.li/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://blog.archive.li/ => https://blog.archive.li/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://www.archive.li/ => https://www.archive.li/: (35, 'gnutls_handshake() failed: Handshake failed')
INFO Finished in 4.51 seconds. Loaded rulesets: 1, URL pairs: 8.
Test URL test failed.

This would work fine on a current Ubuntu 15.10 system:

$ ./fetch-test.sh src/chrome/content/rules/Archive.is.xml 
INFO Finished comparing http://archive.today/ -> https://archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://blog.archive.today/ -> https://blog.archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.today/ -> https://www.archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://archive.is/ -> https://archive.is/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.is/ -> https://www.archive.is/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://archive.li/ -> https://archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://blog.archive.li/ -> https://blog.archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.li/ -> https://www.archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished in 10.79 seconds. Loaded rulesets: 1, URL pairs: 8.

unrecognized_name

Second and extremely obscurely, it errors out on receiving an unrecognized_name TLS warning alert. This came up in #4619 today, due to https://apt.linode.com/. Older Apache servers will send an unrecognized_name warning if the SNI host sent by the client doesn't match ServerAlias or ServerName:

$ openssl s_client -connect apt.linode.com:443 -servername apt.linode.com -state
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:unknown state
SSL3 alert read:warning:unrecognized name
SSL_connect:SSLv3 read server hello A
...

The obsolete RFC 4366 § 3.1 encouraged this:

   If the server understood the client hello extension but does not
   recognize the server name, it SHOULD send an "unrecognized_name"
   alert (which MAY be fatal).

The current RFC 6066 § 3 does not:

               If the server understood the ClientHello extension but
   does not recognize the server name, the server SHOULD take one of two
   actions: either abort the handshake by sending a fatal-level
   unrecognized_name(112) alert or continue the handshake.  It is NOT
   RECOMMENDED to send a warning-level unrecognized_name(112) alert,
   because the client's behavior in response to warning-level alerts is
   unpredictable.

For that reason, the alert was removed in Apache 2.2.30 and 2.4.10 (Apache bug 56241). Still, as we can see, there is at least one unfortunate 2.2.22 installation in the wild.

Chrome and Firefox both ignore the warning alert. (SSL Labs reports "Incorrect SNI alerts".) Ubuntu 14.04's version of curl fails hard on encountering it. From a reduced testcase:

INFO Finished comparing http://apt.linode.com/ -> https://apt.linode.com/. Rulefile: src/chrome/content/rules/apt.linode.com.xml.
ERROR src/chrome/content/rules/apt.linode.com.xml: Fetch error: http://apt.linode.com/ => https://apt.linode.com/: (51, 'gnutls_handshake() warning: The server name sent was not recognized')
INFO Finished in 0.28 seconds. Loaded rulesets: 1, URL pairs: 1.
Test URL test failed.

Again, the version of curl in Ubuntu 15.10 ignores it:

$ ./fetch-test.sh src/chrome/content/rules/apt.linode.com.xml
INFO Finished comparing http://apt.linode.com/ -> https://apt.linode.com/. Rulefile: src/chrome/content/rules/apt.linode.com.xml.
INFO Finished in 0.19 seconds. Loaded rulesets: 1, URL pairs: 1.

I'm uncertain, but i believe the curl fix relates to Debian bug 685402 (2012), and curl commits curl/curl@ee3551e, curl/curl@2045d83, curl/curl@41eec4e (all from 2012) and curl/curl@d5aab55 (2015-05). I don't know the releases involved.

This Linode rule is very unimportant, and i'm not attached to it, but we will run into more ECDSA problems in the future.

What do?

Activity

  1. mnordhoff commented on Apr 14, 2016

    @mnordhoff
    ContributorAuthor

    By the way, it's easy to try it from a python shell:

    $ sudo aptitude install python-pycurl
    $ python
    import pycurl
    c = pycurl.Curl()
    c.setopt(c.URL, 'https://apt.linode.com/')
    c.perform()
    

    Good: Screenful of HTML.
    Bad: Traceback with pycurl.error: (51, 'gnutls_handshake() warning: The server name sent was not recognized').

    To avoid the "screenful of HTML" part:

    import cStringIO, pycurl
    s = cStringIO.StringIO()
    c = pycurl.Curl()
    c.setopt(c.URL, 'https://apt.linode.com/')
    c.setopt(c.WRITEFUNCTION, s.write)
    c.perform()
    

    (s.getvalue() will return the HTML.)

  2. J0WI commented on Apr 15, 2016

    @J0WI
    Contributor

    Thanks for debugging this failures!
    I think @fuglede is also interested in this issue.

    Can we already switch to Ubuntu 16.04 on Travis?

  3. mnordhoff commented on Apr 16, 2016

    @mnordhoff
    ContributorAuthor

    I don't think so. They wrote this three weeks ago in travis-ci/travis-ci#5821:

    We do not have immediate plans to update the build environment's release. You can use our Docker feature (https://docs.travis-ci.com/user/trusty-ci-environment) to pull a reasonable image and use that instead.

    Another option is whipping up a backported curl package and configuring Travis to install it. I gave it a shot, but i'm a Debian packaging amateur and got stymied by a trivial dependency issue. I might take another stab at it later today.

  4. mnordhoff commented on Apr 16, 2016

    @mnordhoff
    ContributorAuthor

    Got it. ✨ Successful Travis run of the ECDSA Archive.is ruleset using backported curl and pycurl from Ubuntu 15.10:

    https://travis-ci.org/mnordhoff/https-everywhere/builds/123522486

    .travis.yml branch: https://github.com/mnordhoff/https-everywhere/tree/travis-curl-backport
    .travis.yml diff: mnordhoff/https-everywhere@9ac2a33
    PPA: https://launchpad.net/~mnordhoff/+archive/ubuntu/curl-backport

    The pycurl package is a straight copy of the one in Wily.

    The curl package was a hassle, but it only took two small debian/control changes to fix: I disabled RTMP support because dependency hell, and i adjusted a dependency on a newer version of a GnuTLS package because it was only there to work around dependency hell in a newer OS version (Debian bugs 784009 and 787960).

    (How many times can i use "dependency" and "version" in one sentence?)

    Gist with control and diff: https://gist.github.com/mnordhoff/0941ef2219f62762581cdf33c8c5929f

    (I don't have a bzr or git repo for the packages, sorry.)

    (By the way, the gpg key i used to sign the packages hasn't been signed by anyone. It's listed on my Launchpad account and their keyserver, but you can't web-of-trust it.)

    I'm not saying maintaining backported packages is a good option, or the best option, but it is an option.

  5. mnordhoff commented on Apr 22, 2016

    @mnordhoff
    ContributorAuthor

    If my backporting thing is still interesting, i have an Ubuntu 16.04 Xenial version up. (It was just released.)

    Branch: https://github.com/mnordhoff/https-everywhere/tree/travis-curl-backport-xenial
    .travis.yml diff vs. the above (ignore the Wikimedia stuff from master): https://github.com/mnordhoff/https-everywhere/compare/travis-curl-backport...mnordhoff:travis-curl-backport-xenial?expand=1
    .travis.yml diff vs. master: https://github.com/EFForg/https-everywhere/compare/master...mnordhoff:travis-curl-backport-xenial?expand=1
    PPA (likely to be renamed): https://launchpad.net/~mnordhoff/+archive/ubuntu/curl-backport-xenial
    Travis test: https://travis-ci.org/mnordhoff/https-everywhere/builds/124923745
    Travis Archive.is test: https://travis-ci.org/mnordhoff/https-everywhere/builds/124921813
    Travis Linode test: https://travis-ci.org/mnordhoff/https-everywhere/builds/124923072

    Packages:

    • curl. I adjusted a (different) GnuTLS dependency and left RTMP enabled.
    • pycurl. I diked out its build dependency on the newer package python-flaky to avoid backporting half the modern Python infrastructure. flaky was only used by pycurl's test suite — it helps you rerun unreliable unit tests — and it should be fine without it (but i didn't check).
    • rtmpdump to provide librtmp1 for curl. (Unmodified.)
  6. J0WI commented on Apr 22, 2016

    @J0WI
    Contributor

    It seems that this issue breaks most recent submissions.
    cc @fuglede @Hainish @jsha

  7. Hainish commented on Apr 22, 2016

    @Hainish
    Member

    I'm inclined to adopt the backport solution in #4645 (comment), but I think we probably want to create a new ppa of our own so we're not dependent on @mnordhoff's. I'll prevent travis-ruleset-fetch.sh from being run until I get a chance to create our own ppa

  8. Hainish commented on Apr 22, 2016

    @Hainish
    Member

    @J0WI the travis fetch test has been disabled. Next week I'll have some time to create our own ppa off of @mnordhoff's template and re-enable with backported curl

  9. fuglede commented on Apr 22, 2016

    @fuglede
    Contributor

    Note also that whatever breakage this might cause will affect https-everywhere-checker as a whole, so if we are catching a large ratio of false negatives here, a lot of rulesets may have potentially been incorrectly disabled on the general database-wide https-everywhere-checker run(s).

    When the present issues are fixed, I suppose we may want to run https-everywhere-checker against all the rulesets that have been disabled by it to see if that is indeed the case?

  10. mnordhoff commented on Apr 22, 2016

    @mnordhoff
    ContributorAuthor

    @Hainish FWIW, Launchpad's most recent package diffs cover everything i changed in the 16.04 packages:

    curl: https://launchpadlibrarian.net/254163766/curl_7.47.0-1ubuntu2~trusty2_7.47.0-1ubuntu2~trusty3.diff.gz

    Ubuntu had renamed libgnutls28-dev to libgnutls-dev between 14.04 and 16.04 after dropping older versions:

    http://packages.ubuntu.com/search?keywords=libgnutls-dev
    http://packages.ubuntu.com/search?keywords=libgnutls28-dev

    (I disabled RTMP in ~trusty1 and reenabled it in ~trusty2, so there were no net changes.)

    pycurl: https://launchpadlibrarian.net/254168601/pycurl_7.43.0-1ubuntu1~trusty1_7.43.0-1ubuntu1~trusty2.diff.gz

    I just grepped for "flaky" and removed everything. 😀 It would've been a hassle to backport: python-flaky wants things like pypy-setuptools and python-genty, which also wants pypy-six. It would've just been a mess, or required some other variety of dependency surgery (for example, to remove the modern PyPy parts).

    (~trusty1 was an unmodified copy.)

    (Again, i didn't change rtmpdump.)

    To be honest, i didn't verify my approach was entirely necessary — it might have worked to only backport pycurl, skipping curl. But 14.04's curl is older, and built against an older version of GnuTLS, so backporting everything was most likely a good idea.

    Edit: For trust, i'm not sure how to verify i didn't insert a backdoor into an earlier version of the packages, other than downloading the upstream package and diffing by hand. For "@mnordhoff bus factor", you can copy the packages to a different PPA (help).

  11. jsha commented on Apr 22, 2016

    @jsha
    Member

    Quick question: Why a PPA vs downloading a built library from curl's website, or downloading source and building it in Travis? We used the latter approach for protocol-compiler in Boulder recently and have been fairly happy with it.

  12. Hainish commented on Apr 26, 2016

    @Hainish
    Member
  13. mnordhoff commented on Apr 26, 2016

    @mnordhoff
    ContributorAuthor

    No reason, honestly. Using a PPA was the first approach that came to mind.

    Something similar to this would probably work:

    sudo apt-get build-dep curl pycurl
    sudo apt-get install python-pip
    curl -O https://curl.haxx.se/download/curl-7.48.0.tar.bz2
    tar xf curl-7.48.0.tar.bz2
    cd curl-7.48.0
    ./configure --with-gnutls --without-ssl
    make
    # make test
    sudo make install
    sudo pip install pycurl
    

    Compiling it could add some minutes to the build time, if Travis's VMs aren't very fast. Especially if you run curl's make test.

    downloading a built library from curl's website

    Do they make any? For Ubuntu packages, https://curl.haxx.se/download.html just links to packages.ubuntu.com.

  14. Hainish commented on Apr 26, 2016

    @Hainish
    Member

    The problem seems to be not that we're using an outdated version of curl and libcurl, but that the version of pycurl that is installed by default in the travis build system uses gnutls, which apparently as @mnordhoff states does not support ECDSA keys.

    I found that running the fetch test on https://github.com/EFForg/https-everywhere/blob/master/src/chrome/content/rules/Archive.is.xml on my local instance of Ubuntu 14.04 did not result in an error. Thinking it curious, I noticed that the errors that we encounter are referencing gnutls, but that my own pycurl uses openssl. I had both libraries installed, but it seems that pycurl prefers openssl when it is available, unless explicitly told not to. When travis hits the pip install -r test/rules/requirements.txt step, it skips over pycurl because the build environment already has it installed:

    $ pip install -r test/rules/requirements.txt
    
    Requirement already satisfied (use --upgrade to upgrade): pycurl in /usr/lib/python2.7/dist-packages (from -r test/rules/requirements.txt (line 1))

    This system pycurl in /usr/lib/python2.7/dist-packages must have had gnutls explicitly preferred. We can fix this by following up the pip install -r commands with a PYCURL_SSL_LIBRARY=openssl pip install --upgrade --force-reinstall --compile pycurl command. This seems to work, using openssl instead: https://travis-ci.org/Hainish/https-everywhere/jobs/125960621

    I've opened a PR with the fix #4683

    Fortunately, this is much quicker than building curl and libcurl on travis - in another branch I saw it adding ~2.5 mins to our test time.

  15. Hainish commented on Apr 26, 2016

    @Hainish
    Member

    I've distilled this to an issue for travis-ci to consider: travis-ci/travis-ci#5957

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions