You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Error running fetch-test.sh: "Unknown cipher in list" #5628
I'm getting Fetch error messages such as you see below when I run fetch-test.sh against any ruleset I've tried so far. The example I've given below is run with commit 45083c6. This is using Vagrant with a Windows 10 host/Fedora 24 Cloud guest (see #5584). bash test.sh seems to finish without error.
What's causing this error and how can I fix it? Thanks.
[vagrant@localhost vagrant]$ ./fetch-test.sh src/chrome/content/rules/Amnesty_USA.org.xml
INFO Finished comparing http://amnestyusa.org/ -> https://amnestyusa.org/. Rulefile: src/chrome/content/rules/Amnesty_USA.org.xml.
INFO Finished comparing http://www.amnestyusa.org/ -> https://www.amnestyusa.org/. Rulefile: src/chrome/content/rules/Amnesty_USA.org.xml.
ERROR src/chrome/content/rules/Amnesty_USA.org.xml: Fetch error: http://amnestyusa.org/ => https://amnestyusa.org/: (59, 'Unknown cipher in list: RC4-MD5:RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-S')
ERROR src/chrome/content/rules/Amnesty_USA.org.xml: Fetch error: http://www.amnestyusa.org/ => https://www.amnestyusa.org/: (59, 'Unknown cipher in list: RC4-MD5:RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-S')
INFO Finished in 0.29 seconds. Loaded rulesets: 1, URL pairs: 2.
[vagrant@localhost vagrant]$
The tests use pycurl, so a first guess would be that pycurl or (more likely) one of its dependencies are outdated on your setup. Here's what I see on a system which normally runs the tests without problems:
In [1]: import pycurl
In [2]: pycurl.version
Out[2]: 'PycURL/7.19.5.3 libcurl/7.38.0 OpenSSL/1.0.1t zlib/1.2.8 libidn/1.29 libssh2/1.4.3 librtmp/2.3'
Can you check the corresponding versions on your system?
Sure. This is on Windows 10/Fedora 24 Cloud/Vagrant. I had to separately install pycurl because the default pycurl from pip did not work (see pull request #5584).
The main difference between our configurations seems to be my NSS vs your OpenSSL, which makes sense given the problem I documented in the pull request.
I see. We've had similar issues with gnutls; I can see how that's annoying but something you can always do would be to double-check the problematic hosts, and let Travis carry out its OpenSSL based tests afterwards.
I'm not sure we can do much else here. Perhaps have the script output a specific error in case pycurl uses libraries that have issues. @Hainish?
I dug into this some more. Also, I'm checking this with Fedora 24 Workstation using a GUI in VirtualBox, instead of using Vagrant, just to simplify things.
These ciphers are ultimately passed to pycurl. The error Unknown cipher in list means that pycurl sees a cipher in this list that it doesn't recognize. The cipher list logged with the error (by this code) is a truncated version of the above list.
The problem is that NSS and OpenSSL use different, possibly incompatible names for any given cipher, assuming they both support that cipher. I'm guessing a little here, but as an example, consider what OpenSSL labels RC4-MD5. If you look at this list in the Mozilla Servo repository, it looks like RC4-MD5 matches TLS_RSA_WITH_RC4_128_MD5 among others. (I have no idea how reliable this Servo list is, by the way.) curl documentation for CURLOPT_SSL_CIPHER_LISThere leads us to mod_nss documentation here. (The content type for the mod_nss page is text/plain so you might need to save it to your desktop and then open the file with your browser to see rendered HTML.) In the mod_nss documentation we see that TLS_RSA_WITH_RC4_128_MD5 is labeled rsa_rc4_128_md5.
So if we're using NSS instead of OpenSSL, we need to tell pycurl to use rsa_rc4_128_md5 instead of RC4-MD5. Lucky for us, we can override the cipherList by adding the following line to the [http] section of test/rules/manual.checker.config:
cipherList = rsa_rc4_128_md5
The code that enables this override is here. With this option set, we no longer get Unknown cipher in list when running ./fetch-test.sh src/chrome/content/rules/Amnesty_USA.org.xml. Of course, I guess this only works because https://amnestyusa.org/ happens to support rsa_rc4_128_md5.
So a possible fix for this issue is to provide an alternate default cipherList for NSS, corresponding to the existing default list for OpenSSL, and to somehow switch to the new cipherList when using NSS instead of OpenSSL. If you agree this is an appropriate fix, please let me know and I can work on it.
Thanks a lot for the digging! Ultimately what is appropriate and what is not is not for me to decide, but that certainly sounds like a reasonable approach to me.
I'm inclined to just standardize on OpenSSL for testing. When we were testing in travis' build environment rather than a full travis container, we had to specify pycurl to build with OpenSSL support. It seems to me that this is the most appropriate solution, and the one that's likely to give us the most standardized results across all dev environments.
@jeremyn I suggest for development & testing to use docker and follow the standard steps for testing specified in .travis.yml. Barring that, you can build pycurl with OpenSSL in virtualenv using the steps here: http://pycurl.io/docs/latest/install.html
@Hainishpycurl is an interface to libcurl, and both need to use the same SSL backend. On Red Hat systems, libcurl uses the NSS SSL backend. libcurl is also a dependency of dnf and rpm, so it can't be removed or modified as far as I know. So requiring OpenSSL means that you don't support development on Red Hat systems. Is that right?
Also, for what it's worth, this seems to be a maximal cipherList based on the mod_nss documentation linked above, pulled from the table under NSSCipherSuite and removing some ciphers that were not recognized on Fedora 24:
@Hainish The install-dev-dependencies.sh script optionally uses dnf to install packages (see here) which suggests that development is supported on Fedora. If Red Hat/Fedora is not officially supported, that should be noted somewhere that's easy to find.
Also, I submitted PR #5584 in June with some commits and documentation to improve Fedora support. Please feel free to review and comment there.
@jeremyn now that #5584 is merged, your comment in #5628 (comment) seems like the appropriate fix - if you want to start working on providing a default ciphersuite for NSS that sgtm.
Activity
fuglede commentedon Jul 18, 2016
The tests use
pycurl, so a first guess would be thatpycurlor (more likely) one of its dependencies are outdated on your setup. Here's what I see on a system which normally runs the tests without problems:Can you check the corresponding versions on your system?
jeremyn commentedon Jul 18, 2016
Sure. This is on Windows 10/Fedora 24 Cloud/Vagrant. I had to separately install pycurl because the default pycurl from pip did not work (see pull request #5584).
Output:
The main difference between our configurations seems to be my NSS vs your OpenSSL, which makes sense given the problem I documented in the pull request.
jeremyn commentedon Jul 18, 2016
Also, I'm able to use pycurl to fetch https://amnestyusa.org/ in the Python interpreter using the Python 2 directions at http://pycurl.io/docs/latest/quickstart.html .
fuglede commentedon Jul 27, 2016
I see. We've had similar issues with
gnutls; I can see how that's annoying but something you can always do would be to double-check the problematic hosts, and let Travis carry out its OpenSSL based tests afterwards.I'm not sure we can do much else here. Perhaps have the script output a specific error in case
pycurluses libraries that have issues. @Hainish?jeremyn commentedon Jul 28, 2016
I dug into this some more. Also, I'm checking this with Fedora 24 Workstation using a GUI in VirtualBox, instead of using Vagrant, just to simplify things.
Hardcoded in test/rules/src/https_everywhere_checker/http_client.py is a list of default ciphers, currently:
These ciphers are ultimately passed to pycurl. The error
Unknown cipher in listmeans that pycurl sees a cipher in this list that it doesn't recognize. The cipher list logged with the error (by this code) is a truncated version of the above list.The problem is that NSS and OpenSSL use different, possibly incompatible names for any given cipher, assuming they both support that cipher. I'm guessing a little here, but as an example, consider what OpenSSL labels
RC4-MD5. If you look at this list in the Mozilla Servo repository, it looks likeRC4-MD5matchesTLS_RSA_WITH_RC4_128_MD5among others. (I have no idea how reliable this Servo list is, by the way.) curl documentation forCURLOPT_SSL_CIPHER_LISThere leads us to mod_nss documentation here. (The content type for the mod_nss page istext/plainso you might need to save it to your desktop and then open the file with your browser to see rendered HTML.) In the mod_nss documentation we see thatTLS_RSA_WITH_RC4_128_MD5is labeledrsa_rc4_128_md5.So if we're using NSS instead of OpenSSL, we need to tell pycurl to use
rsa_rc4_128_md5instead ofRC4-MD5. Lucky for us, we can override the cipherList by adding the following line to the [http] section oftest/rules/manual.checker.config:The code that enables this override is here. With this option set, we no longer get
Unknown cipher in listwhen running./fetch-test.sh src/chrome/content/rules/Amnesty_USA.org.xml. Of course, I guess this only works because https://amnestyusa.org/ happens to supportrsa_rc4_128_md5.So a possible fix for this issue is to provide an alternate default cipherList for NSS, corresponding to the existing default list for OpenSSL, and to somehow switch to the new cipherList when using NSS instead of OpenSSL. If you agree this is an appropriate fix, please let me know and I can work on it.
fuglede commentedon Jul 28, 2016
Thanks a lot for the digging! Ultimately what is appropriate and what is not is not for me to decide, but that certainly sounds like a reasonable approach to me.
Hainish commentedon Jul 28, 2016
I'm inclined to just standardize on OpenSSL for testing. When we were testing in travis' build environment rather than a full travis container, we had to specify pycurl to build with OpenSSL support. It seems to me that this is the most appropriate solution, and the one that's likely to give us the most standardized results across all dev environments.
@jeremyn I suggest for development & testing to use docker and follow the standard steps for testing specified in
.travis.yml. Barring that, you can buildpycurlwith OpenSSL in virtualenv using the steps here: http://pycurl.io/docs/latest/install.htmljeremyn commentedon Jul 28, 2016
@Hainish
pycurlis an interface tolibcurl, and both need to use the same SSL backend. On Red Hat systems,libcurluses the NSS SSL backend.libcurlis also a dependency ofdnfandrpm, so it can't be removed or modified as far as I know. So requiring OpenSSL means that you don't support development on Red Hat systems. Is that right?jeremyn commentedon Jul 28, 2016
Also, for what it's worth, this seems to be a maximal cipherList based on the mod_nss documentation linked above, pulled from the table under
NSSCipherSuiteand removing some ciphers that were not recognized on Fedora 24:Hainish commentedon Jul 29, 2016
I'm not interested in putting effort into supporting Red Hat, but if you want to open a PR to add support I'd be happy to review / merge, just @ me.
jeremyn commentedon Jul 29, 2016
@Hainish The
install-dev-dependencies.shscript optionally usesdnfto install packages (see here) which suggests that development is supported on Fedora. If Red Hat/Fedora is not officially supported, that should be noted somewhere that's easy to find.Also, I submitted PR #5584 in June with some commits and documentation to improve Fedora support. Please feel free to review and comment there.
Hainish commentedon Jul 29, 2016
I'll get a chance to look at #5584 next week
Hainish commentedon Aug 10, 2016
@jeremyn now that #5584 is merged, your comment in #5628 (comment) seems like the appropriate fix - if you want to start working on providing a default ciphersuite for NSS that sgtm.
jeremyn commentedon Aug 11, 2016
I created pull request #6215 to resolve this issue. Please feel free to close this issue after merging the pull request.