How can I use a custom SSLContext / PyOpenSSLContext when creating a Client? #924

Activity

1. sethmlarson commented on May 2, 2020

   

   sethmlarson

   on May 2, 2020

   Last edited by sethmlarson

   Contributor

   More actions

   Thanks for bringing this use-case to my attention. Got me thinking how this use case can be reconciled with my other thoughts on safe high level TLS APIs.

   👍React with 👍4Reacted by Matt Kafonek, Florimond Manca, Ryan and Kim Christie
2. lovelydinosaur commented on May 4, 2020

   

   lovelydinosaur

   on May 4, 2020

   Member

   More actions

   Heya,

   How can I use a custom SSLContext / PyOpenSSLContext when creating a Client?

   Yup, the verify argument optionally takes an SSLContext instance. So let's thrash out if there's an issue here or not.

   One thing that I did notice prompted by this ticket, is that even though that's a supported usage, and is type annotated as bool|str|SSLContext, the SSLContext usage isn't actually currently noted in the docs. https://www.python-httpx.org/advanced/#ssl-certificates

   That's something we should clearly improve. We ought to include a very minimal possible example, such as...

   >>> import httpx
   >>> import ssl
   >>> import certifi
   >>> context = ssl.create_default_context()
   >>> context.load_verify_locations(cafile=certifi.where())
   >>> httpx.get('https://www.example.com', verify=context)
   <Response [200 OK]>

   I got TypeError: expected str, bytes or os.PathLike object, not PyOpenSSLContext on Client init.

   Could you include a traceback with that?

   We don't have any instances of PathLike in the httpx codebase, and the example above works as expected, so it's not immediately obvious how to replicate this?
3. kafonek commented on May 4, 2020

   

   kafonek

   on May 4, 2020

   Last edited by kafonek

   Author

   More actions

   Thanks for the reply @tomchristie. I went back and did some more tests on httpx 0.12.1 and 0.13.0.dev. It looks like the verify= syntax does work for both httpx.get(url, verify=context) and httpx.Client(verify=context).get(url) for ssl.SSLContext objects generated by pypki2. I should have tested that initially.

   For requests_pkcs12 generated urllib3.contrib.pyopenssl.PyOpenSSLContext objects, I get the same general error. Here's the full traceback from httpx 0.12.1:

   import requests_pkcs12
   content = open('/home/jovyan/.devpki/mrkafon.p12', 'rb').read()
   context = requests_pkcs12.create_ssl_context(content, b"changeme")
   context.load_verify_locations(cafile='/home/jovyan/.devpki/vast-ca.pem')
   context
   >>> <urllib3.contrib.pyopenssl.PyOpenSSLContext at 0x7fe95baafa90>
   
   url = "https://internal-site.com"
   resp = httpx.get(url, verify=context)
   resp
   >>>
   
   TypeError                                 Traceback (most recent call last)
   <ipython-input-3-2fa41aa34910> in <module>
         1 url = "https://internal-site.com"
   ----> 2 resp = httpx.get(url, verify=context)
         3 resp
   
   /opt/conda/lib/python3.7/site-packages/httpx/_api.py in get(url, params, headers, cookies, auth, allow_redirects, cert, verify, timeout, trust_env)
       166         verify=verify,
       167         timeout=timeout,
   --> 168         trust_env=trust_env,
       169     )
       170 
   
   /opt/conda/lib/python3.7/site-packages/httpx/_api.py in request(method, url, params, data, files, json, headers, cookies, auth, timeout, allow_redirects, verify, cert, trust_env)
        80     """
        81     with Client(
   ---> 82         cert=cert, verify=verify, timeout=timeout, trust_env=trust_env,
        83     ) as client:
        84         return client.request(
   
   /opt/conda/lib/python3.7/site-packages/httpx/_client.py in __init__(self, auth, params, headers, cookies, verify, cert, proxies, timeout, pool_limits, max_redirects, base_url, dispatch, app, trust_env)
       470             dispatch=dispatch,
       471             app=app,
   --> 472             trust_env=trust_env,
       473         )
       474         self.proxies: typing.Dict[str, SyncDispatcher] = {
   
   /opt/conda/lib/python3.7/site-packages/httpx/_client.py in init_dispatch(self, verify, cert, pool_limits, dispatch, app, trust_env)
       499 
       500         return URLLib3Dispatcher(
   --> 501             verify=verify, cert=cert, pool_limits=pool_limits, trust_env=trust_env,
       502         )
       503 
   
   /opt/conda/lib/python3.7/site-packages/httpx/_dispatch/urllib3.py in __init__(self, proxy, verify, cert, trust_env, pool_limits)
        33     ):
        34         ssl_config = SSLConfig(
   ---> 35             verify=verify, cert=cert, trust_env=trust_env, http2=False
        36         )
        37         hard_limit = pool_limits.hard_limit
   
   /opt/conda/lib/python3.7/site-packages/httpx/_config.py in __init__(self, cert, verify, trust_env, http2)
        69         self.trust_env = trust_env
        70         self.http2 = http2
   ---> 71         self.ssl_context = self.load_ssl_context()
        72 
        73     def __eq__(self, other: typing.Any) -> bool:
   
   /opt/conda/lib/python3.7/site-packages/httpx/_config.py in load_ssl_context(self)
        92 
        93         if self.verify:
   ---> 94             return self.load_ssl_context_verify()
        95         return self.load_ssl_context_no_verify()
        96 
   
   /opt/conda/lib/python3.7/site-packages/httpx/_config.py in load_ssl_context_verify(self)
       121         elif isinstance(self.verify, bool):
       122             ca_bundle_path = self.DEFAULT_CA_BUNDLE_PATH
   --> 123         elif Path(self.verify).exists():
       124             ca_bundle_path = Path(self.verify)
       125         else:
   
   /opt/conda/lib/python3.7/pathlib.py in __new__(cls, *args, **kwargs)
      1020         if cls is Path:
      1021             cls = WindowsPath if os.name == 'nt' else PosixPath
   -> 1022         self = cls._from_parts(args, init=False)
      1023         if not self._flavour.is_supported:
      1024             raise NotImplementedError("cannot instantiate %r on your system"
   
   /opt/conda/lib/python3.7/pathlib.py in _from_parts(cls, args, init)
       667         # right flavour.
       668         self = object.__new__(cls)
   --> 669         drv, root, parts = self._parse_args(args)
       670         self._drv = drv
       671         self._root = root
   
   /opt/conda/lib/python3.7/pathlib.py in _parse_args(cls, args)
       651                 parts += a._parts
       652             else:
   --> 653                 a = os.fspath(a)
       654                 if isinstance(a, str):
       655                     # Force-cast str subclasses to str (issue #21127)
   
   TypeError: expected str, bytes or os.PathLike object, not PyOpenSSLContext
   

   If it helps, I'm using my own internal CA to sign the .p12 certs, and that CA is used by the internal server I'm connecting to. I can get to the internal site using the same .p12 and CA with requests_pkcs12. It's still possible I'm messing up the cafile loading though?

   Thanks.
4. lovelydinosaur commented on May 4, 2020

   

   lovelydinosaur

   on May 4, 2020

   Member

   More actions

   If you want custom SSL context support, we do have that, but you need to be using an SSLContext instance, from the stdlib.

   We don't have support for PyOpenSSLContext, and the SSL configuration is (correctly) falling through the if isinstance(self.verify, ssl.SSLContext) check, and raising an error at the point that it attempts to treat the context as a string.

   httpx/httpx/_config.py

   Lines 107 to 115 in d34c89a

   	if isinstance(self.verify, ssl.SSLContext):
   	# Allow passing in our own SSLContext object that's pre-configured.
   	context = self.verify
   	self._load_client_certs(context)
   	return context
   	elif isinstance(self.verify, bool):
   	ca_bundle_path = self.DEFAULT_CA_BUNDLE_PATH
   	elif Path(self.verify).exists():
   	ca_bundle_path = Path(self.verify)

   I'd suggest you start by looking into setting up a custom SSLContext instance, and passing that to verify=....

   If there's some capabilities exposed by the third party pyopenssl package, that aren't also present in Python 3's ssl implementation, then it's feasible that there's a valid feature request in here that we could work towards, tho it looks like pyopenssl is more aimed at improving SSL capabilities for Python 2.7-3.5.

   👍React with 👍4Reacted by Matt Kafonek, Ben Eshed, whiteowl3 and Pavel Nikonorov
5. lovelydinosaur

   closed this as completedon May 4, 2020
6. beneshed commented on Nov 11, 2020

   

   beneshed

   on Nov 11, 2020

   More actions

   Seconding this, pyopenssl allows the use of loading certificates from memory instead of from disk

   👍React with 👍2Reacted by Alec Vivian and 双草酸酯
7. whiteowl3 commented on Feb 26, 2022

   

   whiteowl3

   on Feb 26, 2022

   More actions

   If there's some capabilities exposed by the third party pyopenssl package, that aren't also present in Python 3's ssl implementation, then it's feasible that there's a valid feature request in here that we could work towards, tho it looks like pyopenssl is more aimed at improving SSL capabilities for Python 2.7-3.5.

   There absolutely is such functionality.

   from urllib3.contrib.pyopenssl import inject_into_urllib3
   from urllib3 import PoolManager
   from OpenSSL import SSL
   
   #cert, key = generate with pyopenssl, or convert from cryptography, generate with certauth or other tools
   #cert is a SSL.X509, key is a SSL.PKey
   
   ctx = SSL.Context()
   ctx.use_cert(cert)
   ctx.use_privatekey(key)
   
   inject_into_urllib3()
   
   with PoolManager(ssl_context=ctx) as pool:
   	pool.request(...)
   

   This demonstrates the generation and use of an ssl context which never writes its certs or keys to disk, something that, in all of pythonland, is only possible using pyopenssl, because only pyopenssl uses the cffi bindings to the openssl functionality. important peps were proposed and withdrawn, and to date, we appear to be no closer to loading keys from memory in std python than we were ten years ago. httpx solving for this use case, especially async, would be a big deal, as people usually move to other languages to get this done.

   👍React with 👍1Reacted by Alex Shpilkin