Repository navigation
Add support for mypy daemon on Windows #5019
Description
There is a major limitation for mypy daemon and fine-grained incremental mode: mypy daemon doesn't work on Windows, see http://mypy.readthedocs.io/en/latest/mypy_daemon.html#limitations
In particular, dmypy_server.daemonize uses some functionalities not available on Windows (such as os.fork). It would be great to remove this limitation (and update Appveyor to test the support).
@ethanhs It looks like you have most experience on Windows among all of us, would you like to take this task?
Activity
emmatyping commentedon May 10, 2018
@ilevkivskyi I have a few more things to follow up on for PEP 561, but after that I can take this.
emmatyping commentedon May 16, 2018
Mostly for myself, but also those who are interested:
After looking at the implementation a bit and talking with Steve Dower (who was very helpful!) I think the best plan is to use named pipes (which act esesentially like Unix sockets on Windows) for communication and a detached process (using creation flags of Popen). This should be enough to approximate the current UNIX API used by the mypy daemon. We will end up using
_winapi.emmatyping commentedon Jun 8, 2018
As an update, I have finally figured out the Windows API incantations needed to get a server/client setup working over named pipes, and can spawn a detached process which together essentially emulate much of the fork/Unix pipe aspects of the classic Unix daemon. I will start working on the actually mypy daemon now I know how I will implement it :)
ilevkivskyi commentedon Aug 9, 2018
@ethanhs Sorry for disturbing you, just wanted to check if there are any updates on this. This problem just appeared again in other context.
Thanks for helping with this!
emmatyping commentedon Aug 9, 2018
@ilevkivskyi no problem! I've been a bit side-tracked with some PEP 561 stuff, so nothing new to report sadly. I will get back to this soon hopefully.
emmatyping commentedon Sep 20, 2018
Unfortunately, I do not think I will have time to work on this any time soon (at least until December most likely). I am going to un-assign myself so someone else can pick this up if they are interested, but if not, I will come back to this in December.
JukkaL commentedon Oct 24, 2018
@ethanhs Do you happen to have any prototype/work-in-progress code you can share? It's okay if it's not very polished.
emmatyping commentedon Oct 24, 2018
@JukkaL I have some experiments with named pipes and detached processes I could upload. Nothing really mypy specific, mostly experiments to understand the APIs and how a daemon would work.
JukkaL commentedon Oct 24, 2018
Those could be quite helpful, and I'd appreciate if you can share them! Our plan is to add Windows support in the next few weeks, as part of making mypy daemon a more fully supported feature.
emmatyping commentedon Oct 24, 2018
@JukkaL That is good to know. If it is such high priority I would be happy to help how I can. I originally put this off because I felt other things were higher priority (with many things "high-priority" it sometimes can be hard to tell how to prioritize 😄 ).
Anyway, here is the experimental code I had: https://github.com/ethanhs/windows-daemon-python
It doesn't exactly work correctly, but it covers a rough sketch of how the named pipes code is supposed to be used. For the DETACHED_PROCESS flag, you basically need to just add
creationflags=0x00000008to your subprocess call (0x00000008 being the DETACHED_PROCESS flag).emmatyping commentedon Oct 26, 2018
Okay, well I got somewhat nerd-sniped (perhaps I self-sniped) :) I have a branch with some work I just did that daemonizes the server with a detached process, passing arguments via a temporary pickle'd file. I'm still working on getting named pipes to work for IPC, but hopefully I can get something working today or tomorrow.
The branch is available here: https://github.com/ethanhs/mypy/tree/windmypy
JukkaL commentedon Oct 29, 2018
@ethanhs Thanks! It looks like you are most of the way there already.
emmatyping commentedon Oct 29, 2018
@JukkaL yes, if you want, I can probably get a prototype working this week.
JukkaL commentedon Oct 29, 2018
@ethanhs That would be great! But I can also finish it up if you are busy.
emmatyping commentedon Oct 30, 2018
@JukkaL I will have time to work on it this week. If I get stuck I will let you know.
Add support for dmypy on Windows (#5859)