Repository navigation
HTTPX and Celery \ GEvent Issue #3218
Description
I have a gevent based celery worker that will not start if httpx is installed. My code does not use HTTPX, it was installed as a dependancy of openai.
If I pip uninstall httpx my worker runs again.
I am using Python 3.11 and the latest celery.
I also posted the issue here: celery/celery#9032
The main issue is I can't find any indication of the conflict as the celery worker just dies with nothing in the logs or system journal.
I am assuming some other package is dynamically using httpx when it is available but leads to an issue under gevent.
Is there a way to debug the package loading process to see what is trying to use httpx?
Activity
pg1671 commentedon Jun 9, 2024
This looks to be caused by the httpx dependancy on trio.
If I run the code:
from gevent.monkey import patch_all; patch_all()
import gevent
import httpx
def worker(n):
print(n)
if name == 'main':
jobs = [gevent.spawn(worker, job_no) for job_no in range(10)]
gevent.joinall(jobs)
I get:
File "/tests/gevent_test.py", line 3, in
import httpx
File "/work/env3.11/lib/python3.11/site-packages/httpx/init.py", line 2, in
from ._api import delete, get, head, options, patch, post, put, request, stream
File "/work/env3.11/lib/python3.11/site-packages/httpx/_api.py", line 6, in
from ._client import Client
File "/work/env3.11/lib/python3.11/site-packages/httpx/_client.py", line 32, in
from ._transports.default import AsyncHTTPTransport, HTTPTransport
File "/work/env3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 32, in
import httpcore
File "/work/env3.11/lib/python3.11/site-packages/httpcore/init.py", line 1, in
from ._api import request, stream
File "/work/env3.11/lib/python3.11/site-packages/httpcore/_api.py", line 5, in
from ._sync.connection_pool import ConnectionPool
File "/work/env3.11/lib/python3.11/site-packages/httpcore/_sync/init.py", line 1, in
from .connection import HTTPConnection
File "/work/env3.11/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 12, in
from .._synchronization import Lock
File "/work/env3.11/lib/python3.11/site-packages/httpcore/_synchronization.py", line 11, in
import trio
File "/work/env3.11/lib/python3.11/site-packages/trio/init.py", line 23, in
from ._core import TASK_STATUS_IGNORED as TASK_STATUS_IGNORED # isort: split
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/env3.11/lib/python3.11/site-packages/trio/_core/init.py", line 21, in
from ._local import RunVar, RunVarToken
File "/work/env3.11/lib/python3.11/site-packages/trio/_core/_local.py", line 9, in
from . import _run
File "/work/env3.11/lib/python3.11/site-packages/trio/_core/_run.py", line 2840, in
raise NotImplementedError("unsupported platform")
NotImplementedError: unsupported platform
lovelydinosaur commentedon Jun 10, 2024
To clarify... we don't have a dependency on
trio, although it is optional and will be in imported if installed.Could you migrate this issue to the
httpcorerepo, and we'll deal with it at the source?pg1671 commentedon Jun 10, 2024
Thanks tom .. The issue I am experiencing only happens when httpx is installed. If I removed httpx and left httpcore everything was fine.
I did further testing yesterday and the underlying cause was trio.
python-trio/trio#3013
My application does not use either httpx or trio. Httpx is a dependancy of the openai > 1.0.0 package. My code that runs under gevent (via celery gevent worker) does not call any functions that use httpx or trio but having them installed causes the problem where the worker will not start.
celery/celery#9032
I was just pinning openai to the versions prior to using httpx for now. But found the best option was just to uninstall trio which was the root cause.
I am hoping to use httpx under gevent in the future so I can make http 2.0 calls and to call the AI libraries that use HTTPX under gevent.
Not sure If I will run into other issues then.
Let me know if you still want me to move this to httpcore.
lovelydinosaur commentedon Aug 27, 2024
Okay, here's the issue in
httpcore... encode/httpcore#946 ...there's an importtry ... catchthat can be fixed up to resolve this. You're welcome to tackle a PR for it if you'd like.seansmoove27 commentedon Oct 2, 2024
Has this issue been resolved?
lovelydinosaur commentedon Oct 2, 2024
Thanks @see-ann, yes. Resolved in httpcore 1.0.6. encode/httpcore#955