Feature request: Pass certificates and keys without writing them to disk #2114

Activity

1. whiteowl3

   changed the title [-]Feature request: Pass certificates without writing to disk[/-] [+]Feature request: Pass certificates and keys without writing them to disk[/+] on Mar 7, 2022
2. whiteowl3 commented on Mar 7, 2022

   

   whiteowl3

   on Mar 7, 2022

   Author

   More actions

   the urllib3 monkey patch:
   https://github.com/urllib3/urllib3/blob/9f4d05c11eb2f80682c1b7a2671b0636292ef932/src/urllib3/contrib/pyopenssl.py#L149
3. lovelydinosaur commented on Apr 6, 2022

   

   lovelydinosaur

   on Apr 6, 2022

   Member

   More actions

   Here's where I got to with attempting to use pyopenssl directly with httpcore...

   import httpcore
   from OpenSSL import SSL
   
   
   class SSLContext(SSL.Context):
       def set_alpn_protocols(self, protos):
           # The stdlib uses `set_alpn_protocols`, which is named slightly
           # different from the `pyopenssl` version. Also the `pyopenssl`
           # expects the arguments a list of bytes, not a list of str.
           super().set_alpn_protos([p.encode("ascii") for p in protos])
   
       def wrap_socket(self, sock, server_hostname=None):
           # The `pyopenssl` Context doesn't provide a `wrap_socket` method.
           #
           # Note: I'm unclear how to set the `server_hostname` here.
           conn = SSL.Connection(self, sock)
           conn.set_connect_state()
           conn.do_handshake()
           return conn
   
   
   pool = httpcore.ConnectionPool(ssl_context=SSLContext(SSL.SSLv23_METHOD))
   r = pool.request("GET", "https://www.example.com")
   print(r)

   This is a slightly neater starting point, since httpcore.ConnectionPool accepts an ssl_context argument directly.

   (By contrast, httpx accepts verify, cert, and trust_env arguments to setup the ssl_context argument, and there's a little extra hoop-jumping you'd need to do to pass a custom SSLContext all the way through.)

   A good starting point for this issue would be for someone to check if the example I've provided above is complete, and works as expected or not.
4. stale commented on Oct 15, 2022

   

   staleBot

   on Oct 15, 2022

   More actions

   This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
5. stale

   added

   wontfix

   on Oct 15, 2022
6. whiteowl3 commented on Oct 25, 2022

   

   whiteowl3

   on Oct 25, 2022

   Author

   More actions

   Not stale.
7. Kludex

   removed

   wontfix

   on Oct 26, 2022
8. deepsurfacesec commented on Mar 27, 2023

   

   deepsurfacesec

   on Mar 27, 2023

   More actions

   Another reason to allow usage of pyopenssl is that the ssl module doesn't have any way to control the verification callback. Specifically, I'm referring to this method:
   https://www.pyopenssl.org/en/latest/api/ssl.html#OpenSSL.SSL.Context.set_verify

   If you have a need, as I do, to do more in-depth inspection of server certificates during a handshake, this allows you to define a callback function that can perform those checks. The built-in ssl module has no way to do this, which is super disappointing.

   With requests and urllib3, they used to have the ability to use a pyOpenSSL backend. Then with a simple adapter you could set the verification callback. But now they're deprecating pyOpenSSL and it is really hard to find any HTTP library whose authors care about pyOpenSSL and any advanced certificate management. :-(

   👍React with 👍1Reacted by 双草酸酯
9. deepsurfacesec commented on Mar 28, 2023

   

   deepsurfacesec

   on Mar 28, 2023

   More actions

   Here's what I was able to get working, based on @tomchristie's post a year ago:

   ctx = certificates.TOFUContext(verifier, 'any_tls', logger=logger)
   
   pool = httpcore.ConnectionPool(ssl_context=ctx)
   transport = httpx.HTTPTransport()
   transport._pool = pool
   client = httpx.Client(transport=transport, timeout=None)
   r = client.request("GET", f"https://{hostname}")
   print(r.text)
   

   The TOFUContext is a custom context very similar to what Tom posted, but is tied in to some other logic that leverages the set_verify function and callback that I mentioned earlier.

   The two parts that make this a little hacky are:

   • The need to override the _pool attribute after initializing the HTTPTransport (subclassing wasn't convenient given how complex the constructor is)
   • The need to set timeout=None. This was needed because the SyncStream.read() method was blowing up for me when the pyOpenSSL-based socket was set to non-blocking. It raises WantReadError when there's no data available to read... I guess? I don't fully understand that bit yet, but I've run into it before. I'm currently on pyopenssl 19.0.0
10. post-j-ma commented on Apr 13, 2023

    

    post-j-ma

    on Apr 13, 2023

    More actions

    Nice to see a solution that doesn't depend on urllib3's support for pyOpenSSL but goes directly (since as already noted above, that's ending, urllib3/urllib3#2680).
11. deepsurfacesec commented on Apr 13, 2023

    

    deepsurfacesec

    on Apr 13, 2023

    More actions

    Unfortunately my solution above still has race conditions. Even though I'm setting timeout=None, somewhere along the line the socket is switched to nonblocking and that results in race conditions during the SSL/TLS handshake. I'm hoping to get to the bottom of that soon.

    👍React with 👍1Reacted by Kim Christie
12. lovelydinosaur commented on Jul 31, 2023

    

    lovelydinosaur

    on Jul 31, 2023

    Last edited by lovelydinosaur

    Member

    More actions

    I'm going to close this off now since it's not something that we're going to handle.

    Python's SSLContext doesn't support this... see python/cpython#60691 and python/cpython#2449

    There's enough information here on the pyopenssl workaround for someone to be able to work with. If anyone digs into that feel free to update this thread for the sake of others, or consider contributing to the third party packages docs. Will happily help work together with folks to help make any workarounds there more widely available but we're not currently able to address this directly.

    Also discussed here... #2037
13. lovelydinosaur

    closed this as completedon Jul 31, 2023