Skip to content

typeshed not being found on windows user install #3988

Closed
#4005
#3991
Closed
typeshed not being found on windows user install#3988
#4005
#3991
Labels
priority-0-hightopic-usability

Description

@Summertime
Summertime
opened on Sep 23, 2017
PS D:\testing> git clone https://github.com/python/mypy.git
# no errors

PS D:\testing> cd .\mypy\

PS D:\testing\mypy> git submodule update --init typeshed
# worked, checked out '55dbb967ad006d5835590c53ab023774dc5a1341'

PS D:\testing\mypy> pip install --user --upgrade .
Processing D:\testing\mypy
Requirement already up-to-date: typed-ast<1.2.0,>=1.1.0 in c:\users\[redacted!]\appdata\roaming\python\python36\site-packages (from mypy===0.530-dev-2bd11f60f0a0332b4c9537983a6ac7da4416e9ca)
Installing collected packages: mypy
  Running setup.py install for mypy ... done
Successfully installed mypy-0.530-dev-2bd11f60f0a0332b4c9537983a6ac7da4416e9ca

PS D:\testing\mypy> mypy setup.py  # to test
Could not resolve typeshed subdirectories. If you are using MyPyfrom source, you need to run "git submodule --init update".Otherwise your MyPy install is broken.

The data directory (after searching for typeshed) appears to be %AppData%\Python\lib\mypy

full path: C:\Users\[redacted!]\AppData\Roaming\Python\lib\mypy

PS D:\testing\mypy> ls $Env:AppData\Python\lib\mypy

typeshed
xml

using this directory leads to more expected results

PS D:\testing\mypy> mypy --custom-typeshed-dir $Env:AppData\Python\lib\mypy\typeshed setup.py
setup.py:15: error: No library stub file for module 'setuptools'
setup.py:15: note: (Stub files are from https://github.com/python/typeshed)
setup.py:16: error: No library stub file for module 'setuptools.command.build_py'
setup.py:74: error: Need type annotation for variable

Activity

  1. gvanrossum commented on Sep 23, 2017

    @gvanrossum
    gvanrossum
    on Sep 23, 2017
    Member

    It would seem that mypy's search for typeshed needs to be augmented with a case for this type of configuration. Mypy itself is at $HOME\appdata\roaming\python\python36\site-packages\mypy and typeshed is at $HOME\AppData\Python\lib\mypy\typeshed. Could you take a debugger and step through the default_data_dir() function (or insert some print statements) to find out where it goes wrong? I'm guessing the reason might be that it's looking for a component of the path named lib, but in your case there isn't one.

    How did your Python install get in that location? I don't think that's where the Windows installers downloaded from python.org put it (but I might be wrong).

  2. emmatyping commented on Sep 23, 2017

    @emmatyping
    emmatyping
    on Sep 23, 2017
    Member

    I can debug this.

    How did your Python install get in that location? I don't think that's where the Windows installers downloaded from python.org put it (but I might be wrong).

    For me the default 3.x install directory for the newer installers (3.5+) point to %USERPROFILE%\AppData\Local\Programs\Python\PythonXY instead of C:\PythonXY.

  3. ilevkivskyi
    closed this as completedin #3991on Sep 24, 2017
  4. ilevkivskyi
    added a commit that references this issue on Sep 24, 2017

    Fix finding typeshed user install on Windows (#3991)

    6c409b4

  5. Summertime commented on Sep 24, 2017

    @Summertime
    Summertime
    on Sep 24, 2017
    Author

    sorry for a slightly-too-late response, the fix doesn't seem to work for me :(

    https://gist.github.com/Summertime/403ce90d067253669796326212becce0 probably-overkill install logs are here for the exact paths of everything (installing from the latest merge)

    I can give a shot at trying to work out the issue in a couple days, however the problem might be obvious from the install logs already, I'm not too sure.

  6. emmatyping commented on Sep 24, 2017

    @emmatyping
    emmatyping
    on Sep 24, 2017
    Member

    Hm okay. Based on the output of the log it seems that you are using a custom compiled Python. Is that true? Sorry that fix didn't work for you.

  7. Summertime commented on Sep 24, 2017

    @Summertime
    Summertime
    on Sep 24, 2017
    Author

    no custom compiling! I reinstalled python just to make sure, clicked the main 3.x.x download link on python.org, did the following, then ran pip install . --user in the same folder as before, same result!

  8. emmatyping
    reopened this on Sep 24, 2017

  9. emmatyping commented on Sep 24, 2017

    @emmatyping
    emmatyping
    on Sep 24, 2017
    Member

    Could you tell me what where mypy gives you now that you re-installed Python? Thanks!

  10. gvanrossum commented on Sep 24, 2017

    @gvanrossum
    gvanrossum
    on Sep 24, 2017 via email
    Member

  11. emmatyping commented on Sep 25, 2017

    @emmatyping
    emmatyping
    on Sep 25, 2017
    Member

    Could you point out the comparison that you are thinking of? The part of the code in question is independent of the OS path separator to my knowledge. I'm thinking of here in particular

    mypy/mypy/build.py

    Line 231 in 6c409b4

    if os.path.basename(lib) in ('lib', 'lib32', 'lib64') \

  12. gvanrossum commented on Sep 25, 2017

    @gvanrossum
    gvanrossum
    on Sep 25, 2017
    Member

    I wasn't thinking of os.sep but of different cases for lib or python (on the next line). In particular the OP has Python (%AppData%\Python\lib\mypy and C:\Users\[redacted!]\AppData\Roaming\Python\lib\mypy).

  13. emmatyping commented on Sep 25, 2017

    @emmatyping
    emmatyping
    on Sep 25, 2017
    Member

    Confusingly, %APPDATA% expands to %USERPROFILE%\AppData\Roaming or C:\Users\<username>\AppData\Roaming, so essentially the two are the same.

  14. gvanrossum commented on Sep 25, 2017

    @gvanrossum
    gvanrossum
    on Sep 25, 2017
    Member

    I'm working on the hypothesis that ...\appdata\roaming\python\python36\site-packages (as quoted by OP) might be using capitalization everywhere, i.e. ...\AppData\Roaming\Python\Python36\site-packages. The only time this path is quoted by the OP it's clearly already lowercased by pip, so we have no hard evidence against this hypothesis. And if that's true, the algorithm (which seems to be searching for the python36 segment of the path) would fail since it's doing a case-sensitive comparison.

    I have no idea what installer was used, so I don't want to exclude this hypothesis yet -- it seems to be using capitalization in earlier segments of the path (at least ...\AppData\Roaming\Python are quoted by the OP). Also, I found this about AppData\Roaming: https://askleo.com/whats-the-appdata-roaming-folder/

  15. Summertime commented on Sep 25, 2017

    @Summertime
    Summertime
    on Sep 25, 2017
    Author

    path of mypy after reinstall

    PS D:\[username]\testing\mypy> where.exe mypy
C:\Users\[username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

    I did make sure to wipe all site-package/script directories before reinstalling too, so not an artifact of the previous install

    paths and capitalization

    As reported by powershell/windows (and as a result, probably the "official" capitalization that would show up:

    PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path .\mypy.exe | %{$_.Path}
C:\Users\[Username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path .\mypy.exe | %{$_.Path}
C:\Users\[Username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path ..\site-packages\mypy | %{$_.Path}
C:\Users\[Username]\AppData\Roaming\Python\Python36\site-packages\mypy

PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path ..\..\lib\mypy\typeshed | %{$_.Path}
C:\Users\[Username]\AppData\Roaming\Python\lib\mypy\typeshed

    custom Installer

    I'm working on seeing if this reproduces in a fresh windows 10 VM using only
    https://www.python.org/ftp/python/3.6.2/python-3.6.2.exe and
    https://github.com/git-for-windows/git/releases/download/v2.14.1.windows.1/Git-2.14.1-64-bit.exe

  16. emmatyping commented on Sep 25, 2017

    @emmatyping
    emmatyping
    on Sep 25, 2017
    Member

    Aha, I see an issue. In the screenshots of your installer, you are using Python 3.6 32 bits. It is being installed in %USERPROFILE%\AppData\Local\.... It seems that the mypy.exe is left over (it is installed in %USERPROFILE%\AppData\Roaming).

    However, I agree that the check for prefixes with python should probably be case insensitive. So I will make a PR about that.

  17. ilevkivskyi commented on Sep 25, 2017

    @ilevkivskyi
    ilevkivskyi
    on Sep 25, 2017
    Member

    By the way #3991 broke installing mypy on Linux Mint. So if #3991 doesn't really solve this problem then I propose to remove the or os.path.basename(lib).startswith('python') line.

  18. emmatyping commented on Sep 25, 2017

    @emmatyping
    emmatyping
    on Sep 25, 2017
    Member

    Hm, that is concerning. Could you describe how it is broken?

  19. ilevkivskyi commented on Sep 25, 2017

    @ilevkivskyi
    ilevkivskyi
    on Sep 25, 2017
    Member

    ...or make this check dependent on sys.platform == 'linux'.

  20. ilevkivskyi commented on Sep 25, 2017

    @ilevkivskyi
    ilevkivskyi
    on Sep 25, 2017
    Member

    I see this when I install from git master:

    tst22.py: error: Name '__builtins__' is not defined
tst22.py:1: error: No library stub file for standard library module 'typing'
tst22.py:1: note: (Stub files are from https://github.com/python/typeshed)

  21. ilevkivskyi commented on Sep 25, 2017

    @ilevkivskyi
    ilevkivskyi
    on Sep 25, 2017
    Member

    FWIW, removing the line I mentioned fixes the problem. But I have no idea why.

  22. Summertime commented on Sep 25, 2017

    @Summertime
    Summertime
    on Sep 25, 2017
    Author

    it goes into appdata/roaming when done with --user, if I did it without --user it would end up in the appdata/local spot

    I did say I wiped the folder!

    And it reproduces in a VM too!

    # after installing python like I did in the screenshots before:
# Also using absolute paths since too lazy to update my path for pip's user install location
PS [blah]\mypy> C:\Users\[differentusername!]\AppData\Roaming\Python\Python36\Scripts\mypy.exe setup.py
Could not resolve typeshed subdirectories. If you are using mypy
from source, you need to run "git submodule --init update".
Otherwise your mypy install is broken.
Python executable is located at c:\users\[differentusername!]\appdata\local\programs\python\python36-32\python.exe.
Mypy located at C:\Users\[differentusername!]\AppData\Roaming\Python\Python36\site-packages

    However I will note, running the module directly works:

    PS [blah]\mypy> python -m mypy .\setup.py
setup.py:15: error: No library stub file for module 'setuptools'
setup.py:15: note: (Stub files are from https://github.com/python/typeshed)
setup.py:16: error: No library stub file for module 'setuptools.command.build_py'
setup.py:74: error: Need type annotation for variable

    (but for some reason my main install doesn't mirror that, dunno!) (I guess I'll consider my main install damaged for now since it repros differently)

  23. ilevkivskyi commented on Sep 25, 2017

    @ilevkivskyi
    ilevkivskyi
    on Sep 25, 2017
    Member

    It looks like installation on Mac is also broken now. I see:

    tst22.py: error: Name '__builtins__' is not defined
tst22.py:1: error: No library stub file for standard library module 'typing'
tst22.py:1: note: (Stub files are from https://github.com/python/typeshed)
  24. ilevkivskyi
    added
    priority-0-high
    topic-usability
    on Sep 25, 2017

  25. gvanrossum commented on Sep 25, 2017

    @gvanrossum
    gvanrossum
    on Sep 25, 2017 via email
    Member

  26. emmatyping commented on Sep 25, 2017

    @emmatyping
    emmatyping
    on Sep 25, 2017
    Member

    I made a PR which should fix everything in #4005.

  27. ilevkivskyi
    closed this as completedin #4005on Sep 26, 2017
  28. ilevkivskyi
    added a commit that references this issue on Sep 26, 2017

    Support user install on Windows. (#4005)

    9615b9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority-0-hightopic-usability

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions