Write error on HTTP/2: "AssertionError: assert waiter is None or waiter.cancelled()" #551

Activity

1. yeraydiazdiaz commented on Dec 1, 2019

   

   yeraydiazdiaz

   on Dec 1, 2019

   Contributor

   More actions

   Hi @PrimozGodec, I cannot reproduce this issue with a stripped down version of your code.

   import asyncio
   from pathlib import Path
   
   import httpx
   
   
   async def send_requests(path: Path) -> list:
       requests = []
       async with httpx.Client(
           timeout=httpx.TimeoutConfig(timeout=60), base_url="http://localhost:5000/"
       ) as client:
           for _ in range(1000):
               requests.append(_send_to_server(client, path))
   
           results = await asyncio.gather(*requests)
   
       return results
   
   
   async def _send_to_server(client: httpx.Client, path: Path) -> httpx.Response:
       # this should probably be done elsewhere
       with open(path, "rb") as fd:
           payload = fd.read()
           size = len(payload)
       headers = {"Content-Type": "application/octet-stream", "Content-Length": str(size)}
       response = await client.post("/", headers=headers, data=payload)
       print(response.json())
       return response
   
   
   if __name__ == "__main__":
       path = Path(__file__).parent / 'image.pkl.zip'
       asyncio.run(send_requests(path))

   Hitting a very simple Quart server.

   from quart import Quart, request
   
   app = Quart(__name__)
   
   
   @app.route("/", methods=["POST"])
   async def post() -> dict:
       data = await request.get_data()
       return {"size": len(data)}
   
   
   if __name__ == "__main__":
       app.run(host="0.0.0.0", port=5000)

   I'm using the same image.pkl.zip file that you posted and encountered no errors on master. I did encounter the PoolTimeout error in 0.8.0 described in #557 and fixed via #563.

   Maybe you could try again using the master branch?
2. lovelydinosaur commented on Dec 2, 2019

   

   lovelydinosaur

   on Dec 2, 2019

   Member

   More actions

   For housekeeping I'll close this, pending confirmation.
3. lovelydinosaur

   closed this as completedon Dec 2, 2019
4. PrimozGodec commented on Dec 3, 2019

   

   PrimozGodec

   on Dec 3, 2019

   ContributorAuthor

   More actions

   I can confirm that this issue is fixed with the master branch. Probably it is the consequence of one of the changes.

   👍React with 👍1Reacted by Kim Christie
5. PrimozGodec commented on Dec 3, 2019

   

   PrimozGodec

   on Dec 3, 2019

   ContributorAuthor

   More actions

   I said it too fast. The problem was that I didn't notice that HTTPX switched to HTTP1.1 as default. :( The issue still persists.

   Since I had problems to set up HTTP2 local server I changed @yeraydiazdiaz example such that it communicates with our server (which is HTTP2 compatible).

   import asyncio
   import pickle
   from pathlib import Path
   
   import httpx
   
   
   MAX_R = 0
   
   async def send_requests(path: Path) -> list:
       requests = []
       async with httpx.Client(
           timeout=httpx.TimeoutConfig(timeout=60), base_url="https://api.garaza.io/",
           http2=True
       ) as client:
           for _ in range(1000):
               requests.append(_send_to_server(client, path))
   
           results = await asyncio.gather(*requests)
   
       return results
   
   async def _send_to_server(client: httpx.Client, path: Path) -> httpx.Response:
       # this should probably be done elsewhere
       with open(path, "rb") as f:
           payload = pickle.load(f)
           size = len(payload)
       headers = {"Content-Type": 'image/jpeg', "Content-Length": str(size)}
       response = await client.post(
           "/image/inception-v3?machine=1&session=1&retry=0", headers=headers, data=payload)
       print(response.http_version)
       print(response.json())
       return response
   
   
   if __name__ == "__main__":
       path = Path(__file__).parent / 'image.pkl'
       asyncio.run(send_requests(path))
   

   I changed:

   • url adresses
   • upickling the image
   • adding http2=True to the client

   After those changes code still works correctly. The issue appears when I add my custom waiting strategy:

   import asyncio
   import pickle
   from pathlib import Path
   
   import httpx
   
   
   MAX_R = 0
   
   async def send_requests(path: Path) -> list:
       requests = []
       async with httpx.Client(
           timeout=httpx.TimeoutConfig(timeout=60), base_url="https://api.garaza.io/",
           http2=True
       ) as client:
           for _ in range(1000):
               requests.append(_send_to_server(client, path))
   
           results = await asyncio.gather(*requests)
   
       return results
   
   async def __wait_until_released():
       while MAX_R >= 100:
           await asyncio.sleep(0.1)
   
   async def _send_to_server(client: httpx.Client, path: Path) -> httpx.Response:
       await __wait_until_released()
       global MAX_R
       MAX_R += 1
       with open(path, "rb") as f:
           payload = pickle.load(f)
           size = len(payload)
       headers = {"Content-Type": 'image/jpeg', "Content-Length": str(size)}
       response = await client.post(
           "/image/inception-v3?machine=1&session=1&retry=0", headers=headers, data=payload)
       print(response.http_version)
       print(response.json())
       MAX_R -= 1
       return response
   
   
   if __name__ == "__main__":
       path = Path(__file__).parent / 'image.pkl'
       asyncio.run(send_requests(path))
   

   This functionality makes sure that the client never sends more than 100 requests to the server. I know that it is not necessary for HTTPX but I need that functionality to prevent the client from loading too many images in the memory at once. The loading functionality is omitted here since it is not required to reproduce the error.

   It is also possible that waiting functionality is not good. Do you have any idea how to change it or what is the problem here?

   The error stops appearing when I add the lock around write as it is described in the issue description.
6. lovelydinosaur

   reopened this on Dec 3, 2019
7. lovelydinosaur

   added

   http/2Issues and PRs related to HTTP/2Issues and PRs related to HTTP/2

   on Dec 3, 2019
8. lovelydinosaur

   changed the title [-]asyncio: write error[/-] [+]write error on HTTP/2[/+] on Dec 3, 2019
9. lovelydinosaur commented on Dec 8, 2019

   

   lovelydinosaur

   on Dec 8, 2019

   Member

   More actions

   Expect this to have been closed in master via #612, but happy to reopen if I've made an incorrect assessment there.
10. lovelydinosaur

    closed this as completedon Dec 8, 2019
11. PrimozGodec commented on Dec 8, 2019

    

    PrimozGodec

    on Dec 8, 2019

    ContributorAuthor

    More actions

    Sorry, but it is not fixed with #612 the issue still persists. The example from the above still fails with the same error as before. I can not reopen the issue since it was not closed by me.
12. lovelydinosaur

    reopened this on Dec 8, 2019
13. florimondmanca

    changed the title [-]write error on HTTP/2[/-] [+]Write error on HTTP/2: "AssertionError: assert waiter is None or waiter.cancelled()"[/+] on Dec 9, 2019
14. lovelydinosaur

    added

    bugSomething isn't workingSomething isn't working

    on Dec 12, 2019
15. lovelydinosaur

    closed this as completedin #699on Dec 31, 2019