Skip to content

pathlib.Path.glob does not follow symlinks #77609

Closed
Closed
pathlib.Path.glob does not follow symlinks#77609
Labels
OS-windowsstdlibStandard Library Python modules in the Lib/ directorytopic-pathlibtype-bugAn unexpected behavior, bug, or error

Description

@BrianMSheldon
BrianMSheldon
mannequin
opened on May 5, 2018
BPO 33428
Nosy @pfmoore, @pitrou, @tjguk, @jreese, @zware, @zooba, @emilyemorehouse, @BrianMSheldon

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-05-05.06:23:00.558>
labels = ['type-bug', 'library', 'OS-windows']
title = 'pathlib.Path.glob does not follow symlinks'
updated_at = <Date 2020-04-29.11:12:49.344>
user = 'https://github.com/BrianMSheldon'

bugs.python.org fields:

activity = <Date 2020-04-29.11:12:49.344>
actor = 'Danya.Alexeyevsky'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2018-05-05.06:23:00.558>
creator = 'brianmsheldon'
dependencies = []
files = []
hgrepos = []
issue_num = 33428
keywords = []
message_count = 4.0
messages = ['316197', '316724', '316880', '367640']
nosy_count = 9.0
nosy_names = ['paul.moore', 'pitrou', 'tim.golden', 'jreese', 'Danya.Alexeyevsky', 'zach.ware', 'steve.dower', 'emilyemorehouse', 'brianmsheldon']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33428'
versions = ['Python 3.6']

Linked PRs

Activity

  1. BrianMSheldon commented on May 5, 2018

    @BrianMSheldon
    BrianMSheldonmannequin
    on May 5, 2018
    MannequinAuthor

    Given a pathlib.Path that contains symlinked subfolders, Path.glob (and .rglob) do not follow symlinks. This is not consistent with glob.glob which does.

    For example given the following:
    C:\Folder
    C:\Folder\Subfolder -> D:\Subfolder
    D:\Subfolder\File.txt

    pathlib.Path('C:/Folder').glob('**/*') yields the following paths:
    WindowsPath('C:/Folder/Subfolder')

    glob.glob('C:/Folder/**/*') yields the following paths:
    'C:/Folder\Subfolder'
    'C:/Folder\Subfolder\File.txt'

    Notice how the contents of Subfolder are present in the glob.glob results but not for Path.glob.

    I would expect Path.glob to be consistent with glob.glob. This is not the only inconsistency (e.g. bpo-22276, bpo-31202) and perhaps Path.glob should be re-implemented using glob.glob.

  2. BrianMSheldon
    added
    stdlibStandard Library Python modules in the Lib/ directory
    OS-windows
    type-bugAn unexpected behavior, bug, or error
    on May 5, 2018

  3. jreese commented on May 15, 2018

    @jreese
    jreesemannequin
    on May 15, 2018
    Mannequin

    This looks like an issue specific to Windows? I can't replicate on Mac, and given Windows' method of implementing "symlinks" as junctions.

  4. BrianMSheldon commented on May 17, 2018

    @BrianMSheldon
    BrianMSheldonmannequin
    on May 17, 2018
    MannequinAuthor

    Windows does not implement symlinks as junctions. Windows has hardlinks, symlinks and junctions which are all distinctly different in behaviour.

    I don't doubt that this is a Windows-specific issue, although I have not tested other platforms. Path.glob and .rglob does work for junctions and hardlinks but glob.glob works consistently for all three.

  5. DanyaAlexeyevsky commented on Apr 29, 2020

    @DanyaAlexeyevsky
    DanyaAlexeyevskymannequin
    on Apr 29, 2020
    Mannequin

    I can reproduce the bug with Linux and python 3.7.5:

    Python 3.7.5 (default, Apr 19 2020, 20:18:17) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import Path
>>> Path('a/b').mkdir(parents=True)
>>> Path('c/d').mkdir(parents=True)
>>> Path('a/c').symlink_to('../c')
>>> Path('e').symlink_to('c')
>>> list(Path('.').rglob('*'))
[PosixPath('e'), PosixPath('c'), PosixPath('a'), PosixPath('c/d'), PosixPath('a/c'), PosixPath('a/b')]

    Expected result:

    [PosixPath('e'), PosixPath('e/d'), PosixPath('c'), PosixPath('a'), PosixPath('c/d'), PosixPath('a/c'), PosixPath('a/c/d'), PosixPath('a/b')]
  6. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  7. serhiy-storchaka
    added this to Fnmatch and glob issueson May 19, 2022
  8. brettcannon
    added
    topic-pathlib
    on Jun 21, 2022

  9. dlukes commented on Nov 24, 2022

    @dlukes
    dlukes
    on Nov 24, 2022

    Following symlinks was disabled on purpose as a fix for #70200:

    cpython/Lib/pathlib.py

    Line 319 in 69f6cc7

    if entry_is_dir and not entry.is_symlink():

    To re-enable it, we'd have to come up with a different mitigation for the symlink loops problem.

    Or possibly, if hidden behind a follow_symlinks arg which would be False by default, it might be enough to just document it as a limitation when follow_symlinks=True? I don't know.

  11. barneygale commented on Jan 28, 2023

    @barneygale
    barneygale
    on Jan 28, 2023
    Contributor

    Or possibly, if hidden behind a follow_symlinks arg which would be False by default, it might be enough to just document it as a limitation when follow_symlinks=True? I don't know.

    glob() still follows symlinks when matching segments like foo/, */ and so forth; it only refuses to follow symlinks when it encounters a ** wildcard. So the current behaviour is roughly follow_symlinks=Sometimes.

    I realised this because I've been trying to implement an iterative version of rglob(), building upon walk() and filtering paths with a compiled re.Pattern object. walk() has its own follow_symlinks argument that accepts True and False, but neither option aligns with the existing behaviour, so it seems like a dead end unless this "bug" is "fixed".

  12. barneygale commented on Jan 29, 2023

    @barneygale
    barneygale
    on Jan 29, 2023
    Contributor

    zsh has a *** wildcard that recurses into symlinks to directories, see Recursive Globbing here: https://linux.die.net/man/1/zshexpn

  13. barneygale commented on Feb 6, 2023

    @barneygale
    barneygale
    on Feb 6, 2023
    Contributor

    Having thought about this some more, I'd like to propose that we add follow_symlinks arguments to glob() and rglob(), where:

    • follow_symlinks=False treats symlinks as files (default)
    • follow_symlinks=True follows symlinks to directories

    Note that the default behaviour would therefore change: a pattern like foo/bar or */bar would no longer match foo/bar if foo is a symlink. This is consistent with how **/bar works today.

    With that in place, we could implement glob() iteratively atop walk(), which would address #89727 and substantially improve performance!

    Thoughts?

  14. 24 remaining items

  15. barneygale
    added 3 commits that reference this issue on May 10, 2023

    Merge branch 'main' into pythongh-77609-glob-follow-symlinks

    d3a3396

    Merge branch 'main' into pythongh-77609-glob-follow-symlinks

    6c61822

    Merge branch 'main' into pythongh-77609-glob-follow-symlinks

    3722b64
  16. barneygale
    added 2 commits that reference this issue on May 23, 2023

    Merge branch 'main' into pythongh-77609-glob-follow-symlinks

    9700dd1

    GH-77609: Add follow_symlinks argument to `pathlib.Path.glob()` (GH-1…

    ace676e

  17. barneygale commented on May 29, 2023

    @barneygale
    barneygale
    on May 29, 2023
    Contributor

    I've added a keyword-only follow_symlinks argument, which will be available in Python 3.13. See 9700dd1 / #102616

    Thank you everyone, and especially @zooba, for the help in getting this through. Resolving!

  18. barneygale
    closed this as completedon May 29, 2023
  19. github-project-automation
    moved this to Done in Fnmatch and glob issueson May 29, 2023

  20. gpshead commented on Mar 23, 2024

    @gpshead
    gpshead
    on Mar 23, 2024
    Member

    Lets change the new follow_symlinks=None to follow_symlinks=NotRecursive (or some other explicitly named sentinel value defined in the pathlib module).

    People reading code that explicitly specifies follow_symlinks=None as would be needed if we ever decide to change the default would not be able to reason about what that does and why =None isn't the same behavior as =False or =0. Seeing a named constant makes it much more clear to those not intimately familiar with the API.

    This is good even if we never decide to work towards a default change.

  21. gpshead
    reopened this on Mar 23, 2024
  22. barneygale
    added a commit that references this issue on Mar 28, 2024

    pythonGH-77609: Add recurse_symlinks argument to `pathlib.Path.glob()`

    9b0646f
  23. barneygale
    added a commit that references this issue on Apr 5, 2024

    GH-77609: Add recurse_symlinks argument to `pathlib.Path.glob()` (#11…

    6150bb2

  24. barneygale commented on Apr 7, 2024

    @barneygale
    barneygale
    on Apr 7, 2024
    Contributor

    Re-resolving - the argument is now recurse_symlinks and accepts a boolean.

  25. barneygale
    closed this as completedon Apr 7, 2024
  26. diegorusso
    added a commit that references this issue on Apr 17, 2024

    pythonGH-77609: Add recurse_symlinks argument to `pathlib.Path.glob()` (

    a4eb3f6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    OS-windowsstdlibStandard Library Python modules in the Lib/ directorytopic-pathlibtype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions