Application data after close notify #1397

Activity

1. lovelydinosaur commented on Nov 23, 2020

   

   lovelydinosaur

   on Nov 23, 2020

   Member

   More actions

   I assume it means that the server has terminated the SSL connection, but then sent additional data.

   Is it something you're able to replicate?
2. florimondmanca

   added

   tls+pkiIssues and PRs related to TLS and PKIIssues and PRs related to TLS and PKI

   on Nov 23, 2020
3. florimondmanca commented on Nov 23, 2020

   

   florimondmanca

   on Nov 23, 2020

   Last edited by florimondmanca

   Contributor

   More actions

   @flvndh Which version of HTTPX / HTTPCore are you using? Unfortunately in the traceback we can see that map_exceptions shuts down the rest of the previous exceptions (raise ... from None) and so you don't see any additional output that might help figure out what's going on, at least where it's coming from.

   Also agree, some information on replicating this would be helpful. Do you hit it consistently? Is it when requesting a particular host? Do you have custom certs involved? Etc… Feel free to look back and fill in the issue template — we have several helpful questions there that you can fill in so anyone looking at this issue can have better context what's going on.

   👍React with 👍1Reacted by Flavi
4. flvndh commented on Nov 24, 2020

   

   flvndh

   on Nov 24, 2020

   Last edited by flvndh

   Author

   More actions

   Thank you guys for the quick response. Here is some additional information:

   Environment

   • OS: Linux
   • Python version: 3.8.6
   • httpx version: 0.14.1
   • httpcore version: 0.10.2
   • OpenSSL version: 1.1.1d
   • Async environment: asyncio
   • Event loop: uvloop
   • HTTP proxy: no
   • Custom certificates: no

   Additional context

   The app is deployed as an Azure Function and is a GraphQL endpoint wrapping a REST API. The function receives a list of entity IDs and GraphQL resolves these entities by calling a bunch of REST endpoints behind the scene. Each entity resolution is scheduled as a separate task on the event loop.

   As the list of entity IDs grow, I see more and more ConnectionTimeout appearing. I implemented a simple backoff strategy to retry the call after some period of time after such an error but ultimately I get the ReadError above.

   I'll try to collect more information about the issue and will keep you posted.
5. florimondmanca commented on Nov 25, 2020

   

   florimondmanca

   on Nov 25, 2020

   Last edited by florimondmanca

   Contributor

   More actions

   @flvndh Couod you try Horta sing to HTTPX at least 0.14.2? This version shipped #1199 which made it so that the underlying tracebacks from HTTPCore aren't cut off. You'll be able to have more info on where the exception comes from (eg during a read or a write, etc).

   Also consider leaving the application running with HTTPX_LOG_LEVEL=trace. That's going to generate lots of logs (a dozen or more per request) but it can also help see more clearly at what stage and in what state the error happens. https://www.python-httpx.org/environment_variables/#httpx_log_level
6. flvndh commented on Nov 25, 2020

   

   flvndh

   on Nov 25, 2020

   Author

   More actions

   @florimondmanca I think I know what happens, please correct me if I'm wrong.

   The entry point of the function roughly looks like this:

   def resolve_entities(entity_ids):
       async with HTTPClient() as http:
           graph = GraphService(http)
   
           tasks = [
               asyncio.create_task(resolve_entity(graph, entity_id))
               for entity_id in entity_ids
           ]
   
           return await asyncio.gather(*tasks)

   and the HTTPClient like this:

   class HTTPClient:
   
       def __init__(self):
           auth = AzureIdentityAuth()
           self._client = AsyncClient(auth=auth)
   
       async def __aenter__(self):
           return self
   
       async def __aexit__(self, exc_type, exc_value, traceback):
           await self._client.aclose()
   
       async def get(self, *args, **kwargs):
           max_trials = 3
           for i in range(max_trials):
               try:
                   response = await self._client.get(*args, **kwargs)
                   response.raise_for_status()
               except TimeoutException:
                   if i == max_trials - 1:
                       raise
                   await asyncio.sleep((i + 1)**2 + random.random())
               else:
                   return response

   After 3 timeouts, the exception propagates and the context manager closes the client connection. The problem is that the other tasks are still active, maybe waiting on data to come.

   I partially resolved the error by slightly increasing the timeout value but an other option might be to limit the amount of concurrent calls active at the same time.
7. florimondmanca commented on Nov 25, 2020

   

   florimondmanca

   on Nov 25, 2020

   Last edited by florimondmanca

   Contributor

   More actions

   The problem is that the other tasks are still active, maybe waiting on data to come.

   Yes, sounds about right to me. Cleanly cleaning up tasks in asyncio is a bit of a nightmare, to be honest. :-) But it's feasible. Basically you'd need to wrap the gather call around a try/finally; on finally run done, pending = asyncio.wait_for(tasks, timeout=), then cancel everything that's in pending. Then by the exit manager exits all tasks will be properly cancelled already.

   (Or you could reach out to things like anyio that provide a clean and sturdy TaskGroup implementation that takes care of this for you. Usually my go-to; it's so easy to mess it up.)
8. florimondmanca

   added

   concurrency/asyncioIssues related to concurrency (asyncio-only)Issues related to concurrency (asyncio-only)

   questionFurther information is requestedFurther information is requested

   and removed

   tls+pkiIssues and PRs related to TLS and PKIIssues and PRs related to TLS and PKI

   on Nov 25, 2020
9. florimondmanca commented on Nov 25, 2020

   

   florimondmanca

   on Nov 25, 2020

   Contributor

   More actions

   @flvndh Ill close for now since I expect you'll be able to sort this out and this isn't really an HTTPX / HTTPCore issue, if that sounds alright!
10. florimondmanca

    closed this as completedon Nov 25, 2020