Repository navigation
Travis's curl is too old, causing spurious fetch-test.sh issues #4645
Description
Spun off from comments on pull #4619. CC @J0WI.
Travis runs Ubuntu 14.04, which is 1.9 years old and apparently hopelessly out-of-date in Internet-years. ;-) Its (|lib|py)curl version is too old for modern features that we need, causing spurious fetch-test.sh failures.
ECDSA
First and critically, it fails on ECDSA websites, such as an ever-growing number of CloudFlare clients. For example, the Archive.is ruleset updated in #4599, which passed Travis a week ago, would fail fetch-test.sh if proposed today:
INFO Finished comparing http://archive.today/ -> https://archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://blog.archive.today/ -> https://blog.archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.today/ -> https://www.archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://archive.is/ -> https://archive.is/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.is/ -> https://www.archive.is/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://archive.li/ -> https://archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://blog.archive.li/ -> https://blog.archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.li/ -> https://www.archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://archive.today/ => https://archive.today/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://blog.archive.today/ => https://blog.archive.today/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://www.archive.today/ => https://www.archive.today/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://archive.li/ => https://archive.li/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://blog.archive.li/ => https://blog.archive.li/: (35, 'gnutls_handshake() failed: Handshake failed')
ERROR src/chrome/content/rules/Archive.is.xml: Fetch error: http://www.archive.li/ => https://www.archive.li/: (35, 'gnutls_handshake() failed: Handshake failed')
INFO Finished in 4.51 seconds. Loaded rulesets: 1, URL pairs: 8.
Test URL test failed.
This would work fine on a current Ubuntu 15.10 system:
$ ./fetch-test.sh src/chrome/content/rules/Archive.is.xml
INFO Finished comparing http://archive.today/ -> https://archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://blog.archive.today/ -> https://blog.archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.today/ -> https://www.archive.today/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://archive.is/ -> https://archive.is/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.is/ -> https://www.archive.is/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://archive.li/ -> https://archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://blog.archive.li/ -> https://blog.archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished comparing http://www.archive.li/ -> https://www.archive.li/. Rulefile: src/chrome/content/rules/Archive.is.xml.
INFO Finished in 10.79 seconds. Loaded rulesets: 1, URL pairs: 8.
unrecognized_name
Second and extremely obscurely, it errors out on receiving an unrecognized_name TLS warning alert. This came up in #4619 today, due to https://apt.linode.com/. Older Apache servers will send an unrecognized_name warning if the SNI host sent by the client doesn't match ServerAlias or ServerName:
$ openssl s_client -connect apt.linode.com:443 -servername apt.linode.com -state
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:unknown state
SSL3 alert read:warning:unrecognized name
SSL_connect:SSLv3 read server hello A
...
The obsolete RFC 4366 § 3.1 encouraged this:
If the server understood the client hello extension but does not
recognize the server name, it SHOULD send an "unrecognized_name"
alert (which MAY be fatal).
The current RFC 6066 § 3 does not:
If the server understood the ClientHello extension but
does not recognize the server name, the server SHOULD take one of two
actions: either abort the handshake by sending a fatal-level
unrecognized_name(112) alert or continue the handshake. It is NOT
RECOMMENDED to send a warning-level unrecognized_name(112) alert,
because the client's behavior in response to warning-level alerts is
unpredictable.
For that reason, the alert was removed in Apache 2.2.30 and 2.4.10 (Apache bug 56241). Still, as we can see, there is at least one unfortunate 2.2.22 installation in the wild.
Chrome and Firefox both ignore the warning alert. (SSL Labs reports "Incorrect SNI alerts".) Ubuntu 14.04's version of curl fails hard on encountering it. From a reduced testcase:
INFO Finished comparing http://apt.linode.com/ -> https://apt.linode.com/. Rulefile: src/chrome/content/rules/apt.linode.com.xml.
ERROR src/chrome/content/rules/apt.linode.com.xml: Fetch error: http://apt.linode.com/ => https://apt.linode.com/: (51, 'gnutls_handshake() warning: The server name sent was not recognized')
INFO Finished in 0.28 seconds. Loaded rulesets: 1, URL pairs: 1.
Test URL test failed.
Again, the version of curl in Ubuntu 15.10 ignores it:
$ ./fetch-test.sh src/chrome/content/rules/apt.linode.com.xml
INFO Finished comparing http://apt.linode.com/ -> https://apt.linode.com/. Rulefile: src/chrome/content/rules/apt.linode.com.xml.
INFO Finished in 0.19 seconds. Loaded rulesets: 1, URL pairs: 1.
I'm uncertain, but i believe the curl fix relates to Debian bug 685402 (2012), and curl commits curl/curl@ee3551e, curl/curl@2045d83, curl/curl@41eec4e (all from 2012) and curl/curl@d5aab55 (2015-05). I don't know the releases involved.
This Linode rule is very unimportant, and i'm not attached to it, but we will run into more ECDSA problems in the future.
What do?
Activity
mnordhoff commentedon Apr 14, 2016
By the way, it's easy to try it from a
pythonshell:Good: Screenful of HTML.
Bad: Traceback with
pycurl.error: (51, 'gnutls_handshake() warning: The server name sent was not recognized').To avoid the "screenful of HTML" part:
(
s.getvalue()will return the HTML.)J0WI commentedon Apr 15, 2016
Thanks for debugging this failures!
I think @fuglede is also interested in this issue.
Can we already switch to Ubuntu 16.04 on Travis?
mnordhoff commentedon Apr 16, 2016
I don't think so. They wrote this three weeks ago in travis-ci/travis-ci#5821:
Another option is whipping up a backported curl package and configuring Travis to install it. I gave it a shot, but i'm a Debian packaging amateur and got stymied by a trivial dependency issue. I might take another stab at it later today.
mnordhoff commentedon Apr 16, 2016
Got it. ✨ Successful Travis run of the ECDSA
Archive.isruleset using backported curl and pycurl from Ubuntu 15.10:https://travis-ci.org/mnordhoff/https-everywhere/builds/123522486
.travis.ymlbranch: https://github.com/mnordhoff/https-everywhere/tree/travis-curl-backport.travis.ymldiff: mnordhoff/https-everywhere@9ac2a33PPA: https://launchpad.net/~mnordhoff/+archive/ubuntu/curl-backport
The
pycurlpackage is a straight copy of the one in Wily.The
curlpackage was a hassle, but it only took two smalldebian/controlchanges to fix: I disabled RTMP support because dependency hell, and i adjusted a dependency on a newer version of a GnuTLS package because it was only there to work around dependency hell in a newer OS version (Debian bugs 784009 and 787960).(How many times can i use "dependency" and "version" in one sentence?)
Gist with
controland diff: https://gist.github.com/mnordhoff/0941ef2219f62762581cdf33c8c5929f(I don't have a bzr or git repo for the packages, sorry.)
(By the way, the gpg key i used to sign the packages hasn't been signed by anyone. It's listed on my Launchpad account and their keyserver, but you can't web-of-trust it.)
I'm not saying maintaining backported packages is a good option, or the best option, but it is an option.
mnordhoff commentedon Apr 22, 2016
If my backporting thing is still interesting, i have an Ubuntu 16.04 Xenial version up. (It was just released.)
Branch: https://github.com/mnordhoff/https-everywhere/tree/travis-curl-backport-xenial
.travis.ymldiff vs. the above (ignore theWikimediastuff frommaster): https://github.com/mnordhoff/https-everywhere/compare/travis-curl-backport...mnordhoff:travis-curl-backport-xenial?expand=1.travis.ymldiff vs.master: https://github.com/EFForg/https-everywhere/compare/master...mnordhoff:travis-curl-backport-xenial?expand=1PPA (likely to be renamed): https://launchpad.net/~mnordhoff/+archive/ubuntu/curl-backport-xenial
Travis test: https://travis-ci.org/mnordhoff/https-everywhere/builds/124923745
Travis
Archive.istest: https://travis-ci.org/mnordhoff/https-everywhere/builds/124921813Travis Linode test: https://travis-ci.org/mnordhoff/https-everywhere/builds/124923072
Packages:
curl. I adjusted a (different) GnuTLS dependency and left RTMP enabled.pycurl. I diked out its build dependency on the newer packagepython-flakyto avoid backporting half the modern Python infrastructure. flaky was only used by pycurl's test suite — it helps you rerun unreliable unit tests — and it should be fine without it (but i didn't check).rtmpdumpto providelibrtmp1for curl. (Unmodified.)J0WI commentedon Apr 22, 2016
It seems that this issue breaks most recent submissions.
cc @fuglede @Hainish @jsha
Hainish commentedon Apr 22, 2016
I'm inclined to adopt the backport solution in #4645 (comment), but I think we probably want to create a new ppa of our own so we're not dependent on @mnordhoff's. I'll prevent
travis-ruleset-fetch.shfrom being run until I get a chance to create our own ppaDisable travis ruelset fetch for now (per EFForg#4645 (comment))
Hainish commentedon Apr 22, 2016
@J0WI the travis fetch test has been disabled. Next week I'll have some time to create our own ppa off of @mnordhoff's template and re-enable with backported
curlfuglede commentedon Apr 22, 2016
Note also that whatever breakage this might cause will affect
https-everywhere-checkeras a whole, so if we are catching a large ratio of false negatives here, a lot of rulesets may have potentially been incorrectly disabled on the general database-widehttps-everywhere-checkerrun(s).When the present issues are fixed, I suppose we may want to run
https-everywhere-checkeragainst all the rulesets that have been disabled by it to see if that is indeed the case?mnordhoff commentedon Apr 22, 2016
@Hainish FWIW, Launchpad's most recent package diffs cover everything i changed in the 16.04 packages:
curl: https://launchpadlibrarian.net/254163766/curl_7.47.0-1ubuntu2~trusty2_7.47.0-1ubuntu2~trusty3.diff.gzUbuntu had renamed
libgnutls28-devtolibgnutls-devbetween 14.04 and 16.04 after dropping older versions:http://packages.ubuntu.com/search?keywords=libgnutls-dev
http://packages.ubuntu.com/search?keywords=libgnutls28-dev
(I disabled RTMP in
~trusty1and reenabled it in~trusty2, so there were no net changes.)pycurl: https://launchpadlibrarian.net/254168601/pycurl_7.43.0-1ubuntu1~trusty1_7.43.0-1ubuntu1~trusty2.diff.gzI just grepped for "
flaky" and removed everything. 😀 It would've been a hassle to backport:python-flakywants things likepypy-setuptoolsandpython-genty, which also wantspypy-six. It would've just been a mess, or required some other variety of dependency surgery (for example, to remove the modern PyPy parts).(
~trusty1was an unmodified copy.)(Again, i didn't change
rtmpdump.)To be honest, i didn't verify my approach was entirely necessary — it might have worked to only backport
pycurl, skippingcurl. But 14.04'scurlis older, and built against an older version of GnuTLS, so backporting everything was most likely a good idea.Edit: For trust, i'm not sure how to verify i didn't insert a backdoor into an earlier version of the packages, other than downloading the upstream package and diffing by hand. For "@mnordhoff bus factor", you can copy the packages to a different PPA (help).
jsha commentedon Apr 22, 2016
Quick question: Why a PPA vs downloading a built library from curl's website, or downloading source and building it in Travis? We used the latter approach for protocol-compiler in Boulder recently and have been fairly happy with it.
Hainish commentedon Apr 26, 2016
cc @mnordhoff #4645 (comment)
mnordhoff commentedon Apr 26, 2016
No reason, honestly. Using a PPA was the first approach that came to mind.
Something similar to this would probably work:
Compiling it could add some minutes to the build time, if Travis's VMs aren't very fast. Especially if you run curl's
make test.Do they make any? For Ubuntu packages, https://curl.haxx.se/download.html just links to packages.ubuntu.com.
Hainish commentedon Apr 26, 2016
The problem seems to be not that we're using an outdated version of
curlandlibcurl, but that the version ofpycurlthat is installed by default in the travis build system usesgnutls, which apparently as @mnordhoff states does not support ECDSA keys.I found that running the fetch test on https://github.com/EFForg/https-everywhere/blob/master/src/chrome/content/rules/Archive.is.xml on my local instance of Ubuntu 14.04 did not result in an error. Thinking it curious, I noticed that the errors that we encounter are referencing
gnutls, but that my own pycurl usesopenssl. I had both libraries installed, but it seems that pycurl prefersopensslwhen it is available, unless explicitly told not to. When travis hits thepip install -r test/rules/requirements.txtstep, it skips over pycurl because the build environment already has it installed:$ pip install -r test/rules/requirements.txt Requirement already satisfied (use --upgrade to upgrade): pycurl in /usr/lib/python2.7/dist-packages (from -r test/rules/requirements.txt (line 1))This system pycurl in
/usr/lib/python2.7/dist-packagesmust have hadgnutlsexplicitly preferred. We can fix this by following up thepip install -rcommands with aPYCURL_SSL_LIBRARY=openssl pip install --upgrade --force-reinstall --compile pycurlcommand. This seems to work, usingopensslinstead: https://travis-ci.org/Hainish/https-everywhere/jobs/125960621I've opened a PR with the fix #4683
Fortunately, this is much quicker than building
curlandlibcurlon travis - in another branch I saw it adding ~2.5 mins to our test time.Hainish commentedon Apr 26, 2016
I've distilled this to an issue for travis-ci to consider: travis-ci/travis-ci#5957