Skip to content

Add fast path to os.[l]stat() that returns incomplete information #99726

Closed
Closed
Add fast path to os.[l]stat() that returns incomplete information#99726
Labels
3.12only security fixesOS-windowsperformancePerformance or resource usage

Description

@zooba
zooba
opened on Nov 23, 2022

A future update to Windows is bringing a new filesystem API for getting stat(-like) information more efficiently from a filename. Currently, we have to open the file, which is quite a slow operation. Being able to simply request metadata based on the path is a real improvement. My testing shows os.stat() and os.lstat() (in the case where no traversal is needed) taking less than 1/4 of their current time when using the new API. I'll link the change in a PR below.

However, the new API does not include the volume serial number, which is how we fill in the st_dev field. Adding an additional call to get the VSN takes all the time we were taking before, so there's no performance benefit.1

So I'd like to propose adding a fast=False argument to os.stat and os.lstat. When left as False, you get the current behaviour. If you pass True, we only guarantee a smaller set of data, and warn that other fields may be absent on some platforms.

Looking through the fields, I have proposed that the file type bits of st_mode (not permissions), the st_size and st_mtime[_ns] fields are the only ones that are important to guarantee.2 All the rest can stay as they are, but we then have the option to drop them from the fast path in the future.3 It's no accident that these are the APIs we already offer as other os.path functions (apart from samestat, which will have to stay on the slow path and probably needs an even slower check in order to be x-plat reliable...)

I'm not sure who cares most about this, so I'm going to leave this open for a while.

Linked PRs

Footnotes

  1. There is still discussion about changing this API before it releases. If that happens, the rest of this proposal is moot, unless we like the idea anyway. ↩

  2. On Windows, we can further guarantee st_file_attributes and st_reparse_tag, as these are the raw values used to calculate the file type bits of st_mode. ↩

  3. stat is already very fast on POSIX-ish filesystems, so it's unlikely to be an issue there, but if we wanted to specialise for network FS or similar then we'd be able to. ↩

Activity

  1. zooba
    added
    performancePerformance or resource usage
    OS-windows
    3.12only security fixes
    on Nov 23, 2022
  2. zooba
    added a commit that references this issue on Nov 23, 2022

    pythongh-99726: Add 'fast' argument to os.[l]stat for faster calculation

    9871983

  3. zooba commented on Nov 23, 2022

    @zooba
    zooba
    on Nov 23, 2022
    MemberAuthor

    Updating importlib has given a ~5-10% improvement for python -c "import asyncio, typing, urllib.request" (with all .pyc files already cached) on Windows builds with the new API.

  4. eryksun commented on Nov 24, 2022

    @eryksun
    eryksun
    on Nov 24, 2022
    Contributor

    I'm familiar with NTAPI NtQueryInformationByName(), which currently supports FileStatInformation and FileCaseSensitiveInformation. Apparently, the Windows API will expose them as FileStatByNameInfo and FileCaseSensitiveByNameInfo, as well as a new FileStatBasicByNameInfo class.

    If you can, please ask the them to also expose FileStatInfo and FileStatBasicInfo for use with GetFileInformationByHandleEx(). Even if the basic stat info lacks the volume serial number, I'd rather combine FileStatBasicInfo with GetVolumeInformationByHandleW() -- than combine FileAttributeTagInfo with GetFileInformationByHandle().

    the new API does not include the volume serial number, which is how we fill in the st_dev field

    The filesystem has nearly zero-cost access to the volume serial number. In the NT API, it's available in FileFsVolumeInformation as VolumeSerialNumber. This info class also includes the volume name (label) and volume creation time, but those aren't important for a stat result.

    Another value that could be useful is FileSystemAttributes (e.g. FILE_READ_ONLY_VOLUME, FILE_PERSISTENT_ACLS, FILE_SUPPORTS_REPARSE_POINTS, FILE_SUPPORTS_HARD_LINKS, etc) from FileFsAttributeInformation. This info class also includes the filesystem name (e.g. "Ntfs") and maximum component length (e.g. 255 characters), but those aren't broadly useful.

  5. eryksun commented on Nov 24, 2022

    @eryksun
    eryksun
    on Nov 24, 2022
    Contributor

    The basic stat info doesn't include the EffectiveAccess field that FileStatInformation includes. Instead it has DeviceType and DeviceCharacteristics. These are mostly likely based on NTAPI FileFsDeviceInformation, the info class that's used by WinAPI GetFileType() and GetDriveTypeW().

    Based on your code, they're applying a file-type categorization to the DeviceType field. NT has about 90 device types (e.g. FILE_DEVICE_DISK, FILE_DEVICE_CD_ROM, FILE_DEVICE_CONSOLE, FILE_DEVICE_NULL). GetFileType() categorizes the NT device type of an open file as one of 4 file types: FILE_TYPE_DISK, FILE_TYPE_CHAR, FILE_TYPE_PIPE, and FILE_TYPE_UNKNOWN.

    There are several useful device characteristics, including FILE_REMOVABLE_MEDIA, FILE_READ_ONLY_DEVICE, FILE_REMOTE_DEVICE, FILE_DEVICE_IS_MOUNTED, FILE_PORTABLE_DEVICE, and FILE_CHARACTERISTIC_WEBDAV_DEVICE1.

    WinAPI GetDriveTypeW() returns DRIVE_REMOTE2 if the device has the FILE_REMOTE_DEVICE characteristic. It returns DRIVE_REMOVABLE if a regular disk device (i.e. not a CD-ROM or RAM disk) has the FILE_REMOVABLE_MEDIA characteristic. Otherwise it returns one of DRIVE_FIXED, DRIVE_CDROM, DRIVE_RAMDISK, or DRIVE_UNKNOWN based on the NT device type.

    It would be nice to expose the device characteristics as st_device_characteristics.

    Footnotes

    1. WebDAV remote devices (e.g. accessing "\\live.sysinternals.com\tools" on Windows) are a pain because the WebDAV driver doesn't implement a volume serial number (st_dev) or file IDs (st_ino). ↩

    2. This is useful to know, for example, because the target path of a junction in a remote path is just a local path on the remote system (e.g. "C:\spam\eggs" on the server's "C:" drive). It cannot be followed manually, such as by realpath() in non-strict mode. ↩

  6. eryksun commented on Nov 24, 2022

    @eryksun
    eryksun
    on Nov 24, 2022
    Contributor

    Regarding the ChangeTime field in FileStatByNameInfo and FileStatBasicByNameInfo, Python's st_ctime on Windows has always been defined incorrectly according to POSIX. It's supposed to be the change time, not the creation time. Some Unix systems return the creation time as st_birthtime or st_btime.

    Currently, the change time could be obtained in stat() via GetFileInformationByHandleEx(): FileBasicInfo. In os.scandir(), the change time could be obtained by listing the directory via GetFileInformationByHandleEx() with FileFullDirectoryInfo (the fastest directory listing that returns basic stat info) or FileIdBothDirectoryInfo.

  7. eryksun commented on Nov 24, 2022

    @eryksun
    eryksun
    on Nov 24, 2022
    Contributor

    Related idea:

    Given FileStatInfo is available and the non-basic FileStatByNameInfo has an EffectiveAccess field, we could use them to implement os.access(path, mode, effective_ids, follow_symlinks). If effective_ids is false and the thread is impersonating, temporarily clear the thread's impersonation token (i.e. revert to self).

  8. clintonroy commented on Nov 24, 2022

    @clintonroy
    clintonroy
    on Nov 24, 2022

    The statx call has a reasonably nice api, in that you pass along the list of attributes you're interested in, maybe we could mirror that, and the implementation could work out what system call to use?

  9. zooba commented on Nov 24, 2022

    @zooba
    zooba
    on Nov 24, 2022
    MemberAuthor

    The statx call has a reasonably nice api

    Now that's an idea. https://man7.org/linux/man-pages/man2/statx.2.html for reference. I've got two days of work with nobody else around (US Thanksgiving), so maybe I'll have a go at adding this API instead.

  10. zooba
    added 4 commits that reference this issue on Nov 24, 2022

    pythongh-99726: Adds os.statx function and associated constants

    6f7e9fa

    Merge remote-tracking branch 'cpython/main' into pythongh-99726

    64111f4

    Merge remote-tracking branch 'cpython/main' into pythongh-99726

    d53c013

    Merge remote-tracking branch 'cpython/main' into pythongh-99726

    768ba42

  11. 2 remaining items

  12. zooba
    added 6 commits that reference this issue on Feb 28, 2023

    Merge remote-tracking branch 'cpython/main' into pythongh-99726-2

    0314273

    Merge remote-tracking branch 'cpython/main' into pythongh-99726-2

    652dd19

    Merge remote-tracking branch 'cpython/main' into pythongh-99726-2

    64cc626

    Merge remote-tracking branch 'cpython/main' into pythongh-99726-2

    ce037f3

    Merge branch 'main' into pythongh-99726-2

    a7c6394

    gh-99726: Improves correctness of stat results for Windows, and uses …

    0f17576

  13. zooba commented on Mar 16, 2023

    @zooba
    zooba
    on Mar 16, 2023
    MemberAuthor

    Implemented, without returning incomplete information! The Windows team updated their API to include everything we needed, so it's just a transparent perf improvement now (as well as a minor correctness improvement).

  14. zooba
    closed this as completedon Mar 16, 2023
  15. carljm
    added a commit that references this issue on Mar 17, 2023

    Merge branch 'main' into inlinecomp2

    1a8f4a0
  16. zooba
    added 2 commits that reference this issue on Mar 23, 2023

    pythongh-99726: Fix order of recently added fields for FILE_STAT_BASI…

    b1d4950

    gh-99726: Fix order of recently added fields for FILE_STAT_BASIC_INFO…

    f1e3eee
  17. Fidget-Spinner
    added 2 commits that reference this issue on Mar 27, 2023

    pythongh-99726: Improves correctness of stat results for Windows, and…

    a10c1f2

    pythongh-99726: Fix order of recently added fields for FILE_STAT_BASI…

    6f57d0a
  18. warsaw
    added 2 commits that reference this issue on Apr 11, 2023

    pythongh-99726: Improves correctness of stat results for Windows, and…

    ea7d6fe

    pythongh-99726: Fix order of recently added fields for FILE_STAT_BASI…

    0d92bc8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixesOS-windowsperformancePerformance or resource usage

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions