Repository navigation
On Windows scandir handles paths different from other filesystem APIs #150880
Description
Activity
What does
os.listdir()return for such path?That throws
FileNotFoundError, the same asscandir.I don't understand well which paths are causing troubles. Can you provide an example? Maybe a script reproducing the issue?
I don't understand well which paths are causing troubles
Normally Windows will ignore trailing spaces on a path, so e.g.
open("file.txt ")will succeed, because the OS ignores the space.But it doesn't ignore trailing spaces within the path, so if you do
os.scandir("C:\dir ")then you might think the trailing space will be ignored, but internally we do+ "\*"(becauseFindFirstFileWrequires a wildcard) which then means the space isn't trailing anymore. It doesn't get stripped, and"dir "doesn't actually exist (even though"dir"does), so it fails.I would say that
os.pathshouldn't mess with trailing spaces, because those are manipulation functions, and we want to let the user control it as much as possible. (Concretely, I don't wantos.path.jointo start stripping its arguments.)Equally, anything that we pass through to the OS (like most of the
osfunctions) should pass through directly (excepting cases that are API mismatches, such as embedded nulls in strings).The grey area is where we modify the user's value before passing it to the OS. On one hand, we can reasonably assume that since
"C:\dir "won't ever exist, we should trim the space before we join with*. On the other hand, there are at least a few ways for the caller to say that they meant for the space to be there (the most prominent being"\\?\C:\dir ") and in that case we shouldn't trim the space. I don't really like having to make choices at that level of "we know exactly how the OS is going to interpret this", so the question is should we try?I do think
GetFullPathNameWis the correct API to ask the OS to trim the path for us if that's the route we go down. Paths starting with\\?\are the only special case Python would need to handle itself (by not doing anything special with them).Okay, yeah, having now looked at the implementation of
GetFullPathNameWand in particular, the comments around this behaviour, we should be using that API to resolve any paths before we manipulate and then use them.In terms of a PR, I think that means we should probably be updating our
joinpathimplementation infileutils.c, and then making sure that posixmodule.c is using that rather than directly manipulating strings. I'm going from memory on those, so not totally sure how appropriate that is and it may turn out to be the wrong idea, but we should already have fairly central native path manipulation that is separate fromntpath.py/os.path, and sincescandir(andlistdir) are doing path manipulation they should be using it.Hi everyone,
I investigated this on the current
mainbranch and was able to reproduce the behavior on Windows.From my testing:
open(),os.stat(), andos.path.exists()all allow Windows to normalize a trailing space in the final path component.os.listdir(),os.scandir(), and consequentlypathlib.Path.iterdir()do not. They append a wildcard before callingFindFirstFileW(), so the trailing space becomes an internal path component and Windows no longer trims it.
I also verified that the same behavior occurs for UNC paths.
While investigating, I confirmed that calling
GetFullPathNameW()before appending the wildcard normalizes ordinary paths as expected, whereas\\?\extended paths intentionally preserve trailing spaces. That seems consistent with the discussion above.My current understanding is that the inconsistency comes from the wildcard being appended before normalization rather than from
FindFirstFileW()itself.One thing I wanted to check before preparing a PR:
Would the preferred direction be to normalize only non-extended paths before constructing the enumeration pattern (while preserving
\\?\semantics), or is there another implementation approach you would prefer?I'm happy to prepare a PR once the intended direction is confirmed.
I put the direction in my last message - extend
_Py_add_relfileor_Py_join_relfileto useGetFullPathNameWon the base directory before appending, and make sure we're using that everywhere.Thanks for the clarification!
That makes sense. I'll follow that approach by updating
_Py_add_relfile/_Py_join_relfileto normalize the base directory withGetFullPathNameW()before appending the wildcard, while preserving the existing\\?\path semantics, and then update the affected callers accordingly.I'll prepare a PR based on that direction. Thanks!
- added a commit that references this issue
on Jul 3, 2026 - added 7 commits that reference this issue
on Jul 3, 2026
Python has to append
\\*to the path before callingFindFirstFileW. This slightly changes how paths are parsed by the OS.Originally posted by @zooba in #115104
To fix this,
GetFullPathNameWcould be used before appending\\*to the path and callingFindFirstFileW. This will create a normalized absolute path without changing the meaning of the path (as far as the OS is concerned). So it'll be consistent with other filesystem APIs.Linked PRs