Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Error running fetch-test.sh: "Unknown cipher in list" #5628

Closed

Description

@jeremyn

I'm getting Fetch error messages such as you see below when I run fetch-test.sh against any ruleset I've tried so far. The example I've given below is run with commit 45083c6. This is using Vagrant with a Windows 10 host/Fedora 24 Cloud guest (see #5584). bash test.sh seems to finish without error.

What's causing this error and how can I fix it? Thanks.

[vagrant@localhost vagrant]$ ./fetch-test.sh src/chrome/content/rules/Amnesty_USA.org.xml
INFO Finished comparing http://amnestyusa.org/ -> https://amnestyusa.org/. Rulefile: src/chrome/content/rules/Amnesty_USA.org.xml.
INFO Finished comparing http://www.amnestyusa.org/ -> https://www.amnestyusa.org/. Rulefile: src/chrome/content/rules/Amnesty_USA.org.xml.
ERROR src/chrome/content/rules/Amnesty_USA.org.xml: Fetch error: http://amnestyusa.org/ => https://amnestyusa.org/: (59, 'Unknown cipher in list: RC4-MD5:RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-S')
ERROR src/chrome/content/rules/Amnesty_USA.org.xml: Fetch error: http://www.amnestyusa.org/ => https://www.amnestyusa.org/: (59, 'Unknown cipher in list: RC4-MD5:RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-S')
INFO Finished in 0.29 seconds. Loaded rulesets: 1, URL pairs: 2.
[vagrant@localhost vagrant]$

Activity

  1. fuglede commented on Jul 18, 2016

    @fuglede
    Contributor

    The tests use pycurl, so a first guess would be that pycurl or (more likely) one of its dependencies are outdated on your setup. Here's what I see on a system which normally runs the tests without problems:

    In [1]: import pycurl
    
    In [2]: pycurl.version
    Out[2]: 'PycURL/7.19.5.3 libcurl/7.38.0 OpenSSL/1.0.1t zlib/1.2.8 libidn/1.29 libssh2/1.4.3 librtmp/2.3'
    

    Can you check the corresponding versions on your system?

  2. jeremyn commented on Jul 18, 2016

    @jeremyn
    ContributorAuthor

    Sure. This is on Windows 10/Fedora 24 Cloud/Vagrant. I had to separately install pycurl because the default pycurl from pip did not work (see pull request #5584).

    Output:

    >>> import pycurl
    >>> pycurl.version
    'PycURL/7.43.0 libcurl/7.47.1 NSS/3.23 Basic ECC zlib/1.2.8 libidn/1.32 libpsl/0.13.0 (+libidn/1.32) libssh2/1.7.0 nghttp2/1.7.1'
    

    The main difference between our configurations seems to be my NSS vs your OpenSSL, which makes sense given the problem I documented in the pull request.

  3. jeremyn commented on Jul 18, 2016

    @jeremyn
    ContributorAuthor

    Also, I'm able to use pycurl to fetch https://amnestyusa.org/ in the Python interpreter using the Python 2 directions at http://pycurl.io/docs/latest/quickstart.html .

  4. fuglede commented on Jul 27, 2016

    @fuglede
    Contributor

    I see. We've had similar issues with gnutls; I can see how that's annoying but something you can always do would be to double-check the problematic hosts, and let Travis carry out its OpenSSL based tests afterwards.

    I'm not sure we can do much else here. Perhaps have the script output a specific error in case pycurl uses libraries that have issues. @Hainish?

  5. jeremyn commented on Jul 28, 2016

    @jeremyn
    ContributorAuthor

    I dug into this some more. Also, I'm checking this with Fedora 24 Workstation using a GUI in VirtualBox, instead of using Vagrant, just to simplify things.

    Hardcoded in test/rules/src/https_everywhere_checker/http_client.py is a list of default ciphers, currently:

    RC4-MD5:RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256
    

    These ciphers are ultimately passed to pycurl. The error Unknown cipher in list means that pycurl sees a cipher in this list that it doesn't recognize. The cipher list logged with the error (by this code) is a truncated version of the above list.

    The problem is that NSS and OpenSSL use different, possibly incompatible names for any given cipher, assuming they both support that cipher. I'm guessing a little here, but as an example, consider what OpenSSL labels RC4-MD5. If you look at this list in the Mozilla Servo repository, it looks like RC4-MD5 matches TLS_RSA_WITH_RC4_128_MD5 among others. (I have no idea how reliable this Servo list is, by the way.) curl documentation for CURLOPT_SSL_CIPHER_LIST here leads us to mod_nss documentation here. (The content type for the mod_nss page is text/plain so you might need to save it to your desktop and then open the file with your browser to see rendered HTML.) In the mod_nss documentation we see that TLS_RSA_WITH_RC4_128_MD5 is labeled rsa_rc4_128_md5.

    So if we're using NSS instead of OpenSSL, we need to tell pycurl to use rsa_rc4_128_md5 instead of RC4-MD5. Lucky for us, we can override the cipherList by adding the following line to the [http] section of test/rules/manual.checker.config:

    cipherList = rsa_rc4_128_md5
    

    The code that enables this override is here. With this option set, we no longer get Unknown cipher in list when running ./fetch-test.sh src/chrome/content/rules/Amnesty_USA.org.xml. Of course, I guess this only works because https://amnestyusa.org/ happens to support rsa_rc4_128_md5.

    So a possible fix for this issue is to provide an alternate default cipherList for NSS, corresponding to the existing default list for OpenSSL, and to somehow switch to the new cipherList when using NSS instead of OpenSSL. If you agree this is an appropriate fix, please let me know and I can work on it.

  6. fuglede commented on Jul 28, 2016

    @fuglede
    Contributor

    Thanks a lot for the digging! Ultimately what is appropriate and what is not is not for me to decide, but that certainly sounds like a reasonable approach to me.

  7. Hainish commented on Jul 28, 2016

    @Hainish
    Member

    I'm inclined to just standardize on OpenSSL for testing. When we were testing in travis' build environment rather than a full travis container, we had to specify pycurl to build with OpenSSL support. It seems to me that this is the most appropriate solution, and the one that's likely to give us the most standardized results across all dev environments.

    @jeremyn I suggest for development & testing to use docker and follow the standard steps for testing specified in .travis.yml. Barring that, you can build pycurl with OpenSSL in virtualenv using the steps here: http://pycurl.io/docs/latest/install.html

  8. jeremyn commented on Jul 28, 2016

    @jeremyn
    ContributorAuthor

    @Hainish pycurl is an interface to libcurl, and both need to use the same SSL backend. On Red Hat systems, libcurl uses the NSS SSL backend. libcurl is also a dependency of dnf and rpm, so it can't be removed or modified as far as I know. So requiring OpenSSL means that you don't support development on Red Hat systems. Is that right?

  9. jeremyn commented on Jul 28, 2016

    @jeremyn
    ContributorAuthor

    Also, for what it's worth, this seems to be a maximal cipherList based on the mod_nss documentation linked above, pulled from the table under NSSCipherSuite and removing some ciphers that were not recognized on Fedora 24:

    cipherList = rsa_3des_sha,rsa_des_sha,rsa_null_md5,rsa_null_sha,rsa_rc2_40_md5,rsa_rc4_128_md5,rsa_rc4_128_sha,rsa_rc4_40_md5,fips_des_sha,fips_3des_sha,rsa_des_56_sha,rsa_rc4_56_sha,rsa_aes_128_sha,rsa_aes_256_sha,rsa_aes_128_gcm_sha_256,dhe_rsa_aes_128_gcm_sha_256,ecdh_ecdsa_null_sha,ecdh_ecdsa_rc4_128_sha,ecdh_ecdsa_3des_sha,ecdh_ecdsa_aes_128_sha,ecdh_ecdsa_aes_256_sha,ecdhe_ecdsa_null_sha,ecdhe_ecdsa_rc4_128_sha,ecdhe_ecdsa_3des_sha,ecdhe_ecdsa_aes_128_sha,ecdhe_ecdsa_aes_256_sha,ecdh_rsa_null_sha,ecdh_rsa_128_sha,ecdh_rsa_3des_sha,ecdh_rsa_aes_128_sha,ecdh_rsa_aes_256_sha,echde_rsa_null,ecdhe_rsa_rc4_128_sha,ecdhe_rsa_3des_sha,ecdhe_rsa_aes_128_sha,ecdhe_rsa_aes_256_sha,ecdhe_ecdsa_aes_128_gcm_sha_256,ecdhe_rsa_aes_128_gcm_sha_256
    
  10. Hainish commented on Jul 29, 2016

    @Hainish
    Member

    I'm not interested in putting effort into supporting Red Hat, but if you want to open a PR to add support I'd be happy to review / merge, just @ me.

  11. jeremyn commented on Jul 29, 2016

    @jeremyn
    ContributorAuthor

    @Hainish The install-dev-dependencies.sh script optionally uses dnf to install packages (see here) which suggests that development is supported on Fedora. If Red Hat/Fedora is not officially supported, that should be noted somewhere that's easy to find.

    Also, I submitted PR #5584 in June with some commits and documentation to improve Fedora support. Please feel free to review and comment there.

  12. Hainish commented on Jul 29, 2016

    @Hainish
    Member

    I'll get a chance to look at #5584 next week

  13. Hainish commented on Aug 10, 2016

    @Hainish
    Member

    @jeremyn now that #5584 is merged, your comment in #5628 (comment) seems like the appropriate fix - if you want to start working on providing a default ciphersuite for NSS that sgtm.

  14. jeremyn commented on Aug 11, 2016

    @jeremyn
    ContributorAuthor

    I created pull request #6215 to resolve this issue. Please feel free to close this issue after merging the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions