Repository navigation
Configuration of windows event loop for libraries #81554
Description
Activity
bdarnell commented
on Jun 22, 2019 bdarnellmannequinMannequinAuthorOn Windows there are two event loop implementions with different interfaces: The proactor event loop is missing the file descriptor family of methods (add_reader()), while the selector event loop has other limitations including missing support for pipes and subprocesses and generally lower scalability. (The default has changed from selector to proactor in Python 3.8).
If an application requires the selector event loop, it can set the global event loop policy when it starts up. But what if a library requires the selector event loop? It wouldn't be appropriate for a library to set the event loop policy. The best I can do is document that "This library requires an event loop which supports the add_reader() method; to use this library on Windows (directly or indirectly) you must install the WindowsSelectorEventLoopPolicy."
This places a burden on application developers that target Windows to examine all their transitive dependencies to see if any require selectorevent loops (and if any have conflicting requirements for proactor event loops, which are even less likely to be documented since this is now the default). Concretely, this is a concern for Tornado (which requires add_reader()) and applications in the scientific python community (including Jupyter) which depend on it.
I know it's probably too late to do anything about this for 3.8, but it would be great if there were some better process to negotiate the right event loop. Some ideas (none of which are very satisfying):
- A declarative marker in setup.cfg or similar indicating the required feature set for an event loop
- A dummy package that could be depended on to indicate the same thing (and maybe this package could even use .pth hacks to change the default simply by being installed, but that seems like a hack too far)
- Some sort of runtime registry: at import time, Tornado could call
asyncio.add_required_features(asyncio.ADD_READER)and this would influence the default policy - Flags to
get_event_loop()indicating the required features (passed through to the policy).
Victor, Andrew,
I'm not an expert in IOCP, but is it possible to implement add_reader/writer in ProactorEventLoop? If there's no native API for that, I guess we can spawn a thread with a 'select()' call to emulate this API?
Lukasz,
Another question: if we fix this, would you allow this to go in beta2/3? Strictly speaking it's going to be a new functionality.
BigStone commented
on Jun 22, 2019 BigStonemannequinMannequinWindows users would certainly prefer to have an emulated non-performant emulation of the old API, rather than a breakage of their beloved Jupyter/Tornado ecosystem.
Is it possible to implement add_reader/writer in ProactorEventLoop?
Sorry, no. There is no way IIUC.
Regarding a request for selecting event loop by a library.
- You can call asyncio.set_event_loop_policy() already.
- From my understanding, there is no issue for Tornado itself. If Jupiter Notebook needs Tornado, Tornado needs selector event loop on Windows -- Jupiter can install the proper loop.
bdarnell commented
on Jun 22, 2019 bdarnellmannequinMannequinAuthorFrom my understanding, there is no issue for Tornado itself. If Jupiter Notebook needs Tornado, Tornado needs selector event loop on Windows -- Jupiter can install the proper loop.
Right. I'm just advocating for something that would make the transition smoother than each project independently stumbling across this issue and adding their own patch (and then fielding support issues from users who have ended up with a combination of versions that doesn't quite work). This of course depends on how many affected projects there are; I know Jupyter is the biggest but they're not the only one.
There's no more direct way, but a thread that does select() should work. In fact, ProactorEventLoop could simply spin up a SelectorEventLoop in another thread when one of the FD methods is called, and proxy the callbacks back and forth.
def add_reader(self, fd, callback, *args): if not self.selector_started: self.start_selector() self.selector.call_soon(self.selector.add_reader, fd, lambda: self.call_soon(callback, *args)) def start_selector(self): self.selector = SelectorEventLoop() def target(): asyncio.set_event_loop(self.selector) self.selector.run_forever() thread = threading.Thread(target=target) thread.start() # Clean shutdown is left as an exercise for the reader.
Unifying the two interfaces like this would be preferable to adding more complexity for configuration IMHO.
Andrew,
Sorry, no. There is no way IIUC.
Well, in this case we should do that via a thread+select approach as I and Ben suggested.
I can write some PoC code; will you have some time to polish it and commit it? (I'm OOO and don't have a windows machine/VM at the moment).
I doubt if we should split a proactor loop into two objects in two threads.
I see no strong objections but have a feeling that we'll get other problems in this way. asyncio is just not designed for this mode.The proposal looks like a very dirty hack and smells as it is.
There is no very strong reason to apply it.
Jupiter Notebook can fix the problem on its own side in two-three lines of clear code.bdarnell commented
on Jun 23, 2019 bdarnellmannequinMannequinAuthorYeah, it's definitely a hack. The argument for it, at best, is "practicality beats purity". The solution is two simple lines, but those lines need to be repeated in every project that depends on Tornado and cares about Windows, now or in the future. How many projects have to be affected to justify a hack like this? I'm not sure, but without this hack the add_reader methods, and by extension Tornado, will remain a compatibility trap.
+1 to what Ben said.
Andrew,
The proposal looks like a very dirty hack and smells as it is.
I wonder why are you feeling like that about this idea. I don't think this is a hack at all. There's no native API, so we're forced to use another approach.
There is no very strong reason to apply it.
I think there's such a reason. asyncio has a well defined API that people use and create programs dependent on it. And then people learn that the API isn't really supported on Windows with one loop, or when it is supported, other APIs aren't. They are then forced to either not support Windows or to rewrute their programs. The situation further complicates if they are using third-party code.
This is a very unfortunate and weird position we put our users into. So this indeed is a perfect example where "practicality beats purity" principle should be invoked.
I don't like to have not required parts like thread pool, time handlers, and the exception handler executed in the auxiliary thread.
What's about an alternative proposal: embed into ProactorEventLoop not entire SelectorEventLoop but selectors.SelectSelector only? We will have a little more code but the code is very clear and self-contained.
Ben:
Concretely, this is a concern for Tornado (which requires add_reader()) and applications in the scientific python community (including Jupyter) which depend on it.
If you need add_reader/add_writer in Python 3.8, you can switch the default event loop to selector at the beginning of your application:
if sys.platform == 'win32': asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
Yury:
If there's no native API for that, I guess we can spawn a thread with a 'select()' call to emulate this API?
There is no native API because IOCP design is to run an asynchronous read/write and then wait for its completion. Unix select() has the opposite design: check if a file descriptor is read for read/write.
Reimplementing add_reader/add_writer using a single select() call in a thread sounds like a good idea, but we have to make sure that it can be stopped whenever using a dedicated "self-pipe" (to awake the blocked select(), so loop.close() can stop this thread.
See attached proof-of-concept: selector_thread.py: run a selector in a separated thread which pass pack events to the loop using call_soon().
I would prefer to use a single selector to better scale with the number of FD.
Note: On Windows, select() only supports sockets.
Note: select.select() may be extended to support more than 512 sockets on Windows, see bpo-28708 :-)
Yury:
Another question: if we fix this, would you allow this to go in beta2/3? Strictly speaking it's going to be a new functionality.
It's a new feature, so it can wait for Python 3.9 :-)
I don't see any regression here, as soon as you can opt-in for SelectorEventLoop.
It's also possible to implement 'select' on top of IOCP using undocumented private APIs. That's what libuv does (and how 'select' works internally). There's a bunch of information on this collected here: python-trio/trio#52
4 remaining items
bdarnell commented
on Feb 22, 2020 bdarnellmannequinMannequinAuthorI have an implementation of the selector-in-another-thread solution in tornadoweb/tornado#2815. Is something like this worth considering for Python 3.9, or was Tornado the only project experiencing this pain and a tornado-specific solution is enough?
From my understanding, the issue is still desirable but we have a lack of resources.
Regarding the patch for tornado, using selectors.SelectSelector for implementing add_reader()/remove_reader() and add_writer()/remove_writer() in ProactorEventLoop looks is more robust solution IMHO.was Tornado the only project experiencing this pain
At least twisted and anyio are still broken on 3.8+Windows because of this change:
https://twistedmatrix.com/trac/ticket/9766
agronholm/anyio#77bdarnell commented
on Feb 25, 2020 bdarnellmannequinMannequinAuthorI considered using the
selectorsmodule directly, but it's not as simple as it sounds. Using the low-level interface means you need to also use a self-waker-pipe (or socket on windows) and manage a queue analogous to that used bycall_soon_threadsafe. We already have two implementations of this pattern in asyncio with subtle differences between them (such as this one i just found: https://bugs.python.org/issue39651). In the end you'd have to duplicate a non-trivial portion of SelectorEventLoop.While there might be some efficiency gains to be had by working directly with the lower-level interface (and avoiding some redundancies between the two threads' event loops), I think the most robust/safest option is to use the well-tested SelectorEventLoop so that the only new code is what's needed to pass things back and forth between threads.
I'd be +1 to bringing uvloop into the stdlib, it would solve many things while introducing an acceptable dependency in the form of libuv.
However, uvloop itself is written in Cython which makes it impossible for us to directly merge it. So that option is pretty much off the table as rewriting the library is not something we have resources for right now.
Ben, Nathaniel is onto something though. Would it be acceptable for you to *require* use of uvloop when Tornado is used with AsyncIO? That indeed solves the Windows problems, and more. I use uvloop in all asyncio applications I maintain (*except for Black which is tested to work on Windows with the Proactor loop).
For Python 3.9, it is a bit late, but not *too late* yet, to make the Proactor loop support AFD_POLL. Maybe counter-intuitively I would feel better about *that* rather than have a background thread with a SelectorEventLoop.
bdarnell commented
on Apr 19, 2020 bdarnellmannequinMannequinAuthorWould it be acceptable for you to *require* use of uvloop when Tornado is used with AsyncIO?
How, exactly? Adding the dependency is no problem, but AFAIK I'd still be stuck with an import-time side effect to set the event loop policy (or a .pth file hack, I guess). Maybe an import-time side effect that chooses uvloop is better since it's a functional superset of the two default windows event loops, but setting the policy at import time still has its problems (What if another module has initialized the event loop before tornado is imported? What if someone also wants to set a custom policy to work around asyncio's strict "only on the main thread" defaults?)
That's why I started this thread not asking for a proactor+selector hybrid event loop, but a better way to *configure* the event loop because policies aren't really doing the job in this case.
I considered using the
selectorsmodule directly, but it's not as simple as it sounds.FWIW, I've reconsidered this. Treating SelectorEventLoop as a black box means you don't have enough control over synchronization and it's hard to avoid spurious wakeups and busy loops on the selector thread. I have a (broken) prototype using SelectorEventLoop that I plan to rewrite to call select.select directly.
minrk commented
on May 10, 2021 minrkmannequinMannequinA hiccup to using uvloop is that it doesn't support Windows yet (MagicStack/uvloop#14), so it can't be used in the affected environment.
I'm exploring this again for pyzmq / Jupyter, and currently investigating relying on tornado's AddThread loop functionality. It's even slightly easier for tornado, which can reasonably set the proactor-wrapper policy at IOLoop start time, which means
asyncio.get_event_loop()returns a loop with add_reader. But pyzmq doesn't get invoked until an event loop is already running. That means the selector thread needs to work not as a wrapper of the loop itself, as in tornado's AddThreadSelector, but attached after-the-fact. Using tornado's AddThread seems to work for this, but I'm not sure that should be assumed.bdarnell commented
on May 15, 2021 bdarnellmannequinMannequinAuthorIt's even slightly easier for tornado, which can reasonably set the proactor-wrapper policy at IOLoop start time, which means
asyncio.get_event_loop()returns a loop with add_reader. But pyzmq doesn't get invoked until an event loop is already running.That's not what I'm doing in Tornado; I don't change the policy or the result of get_event_loop. Instead, I call get_event_loop (only once) and wrap its result in AddThreadSelectorEventLoop. This works even while the event loop is already running (which is not an uncommon case; there is no expectation that you use tornado.ioloop.IOLoop.start instead of asyncio.EventLoop.run_forever).
This relies on the fact that I already have my own thread-local lookup function to retrieve the wrapped event loop; an application that used the more typical asyncio patterns and relied on get_event_loop would indeed have difficulty with this pattern.
Reacted by Min RK- added a commit that references this issue
on Apr 21, 2022 Just sharing a user perspective here. I just came from a committee meeting which considered rejecting a paper due to nonreplicability of a simulation model, which apparently throws this error. (RuntimeWarning: Proactor event loop does not implement add_reader family of methods required for zmq. Registering an additional selector thread for add_reader support via tornado. Use
asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())to avoid this warning.)I think notebooks are a great way to share models, but if they're fragile, they aren't very useful. Specifically, faculty reviewers aren't going to sift through github and stackoverflow threads to get something working, they're just going to reject it or review it by inspection only, which defeats the purpose of runnable artifacts.
This needs to get fixed, or at least have a workaround documented somewhere.
Reacted by Oleg Iarygin and Dave TapleyReacted by Dave Tapleyanyio now supports
anyio.wait_readableandanyio.wait_writableon the windows ProactorEventLoop using the selectors module and a thread.It might make sense to land this code in cpython directly, and in the future do what trio on windows does to support it
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs