os.path.realpath raises a ValueError when path contains a null byte

Bug report

Bug description:

import os
os.path.realpath("\x00")  # raises unexpected ValueError

The ValueError is initially raised by os.lstat.

This behavior is inconsistent with other path operations catching the ValueError raised by os.lstat:
os.path.lexsists: https://github.com/python/cpython/blob/main/Lib/genericpath.py#L30
os.path.exists: https://github.com/python/cpython/blob/main/Lib/genericpath.py#L20
os.path.islink: https://github.com/python/cpython/blob/main/Lib/genericpath.py#L64
os.path.ismount: https://github.com/python/cpython/blob/main/Lib/posixpath.py#L197
os.path.ismount: https://github.com/python/cpython/blob/main/Lib/posixpath.py#L213

CPython versions tested on:

3.10, CPython main branch

Operating systems tested on:

Linux

Linked PRs

I already created a PR #117573 for this issue

I do not think there is a bug. Functions like os.path.exists() answer the question "is there a node (of the specified type) with such path?" Since the valid path cannot contain the null character, the answer is "no". You get the same answer for a path containing non-encodable characters. os.path.realpath() answers the different question -- "what is the real path of the specified node?"

os.path.realpath() ignores some OSError in non-strict mode. This allows it to process paths to non-existent nodes. These errors can be fixed by making a change to the file system. But no matter what you do with the file system, an invalid path (a path containing the null characters or non-encodable characters) will remain invalid.

In case of os.path.exists I can agree.

I guess my selection of examples was not very good.
os.path.basename, os.path.dirname, os.fspath, os.path.abspath, os.path.relpath ... (list is not complete) also do not raise a ValueError when given a path with a null byte.
All the methods do not raise a ValueError when given a path with a null byte, they just accept it as part of the path.

I would except consistency across all path operations, but I found only os.path.realpath with another behaviour. So I assumed this is the one with the bug.

Python 3.13.0a5+ (heads/fix-realpath-value-error-dirty:8da4ebb996, Apr  7 2024, 17:05:12) [GCC 13.2.1 20230918 (Red Hat 13.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.fspath("\x00")
'\x00'
>>> os.path.abspath("\x00")
'/var/home/stefan/Development/repos/cpython/\x00'
>>> os.path.basename("\x00")
'\x00'
>>> os.path.dirname("\x00")
''
>>> os.path.relpath("\x00")
'\x00'
>>> os.path.normpath("\x00")
'\x00'
>>>

os.path.dirname and os.path.relpath do not work with filesystem. They could check for the null character etc, but nobody is interesting in making them intentionally slower. In functions which interact with OS, it happens as the part of conversion to OS-specific representation.

Serhiy's model sounds good to me. To summarize:

exists and is* functions should return False if the path contains NUL
functions that deal with the OS should raise ValueError (before handing the path to an OS call)
functions that manipulate the path should either raise ValueError (if they can detect it easily), or treat NUL as a normal character (the same way as an invalid character like | is handled in ntpath.dirname)

But, it looks like that's not what we do on Windows.

pythongh-117596: Add more tests for os.path with invalid paths

I added some tests for realpath() with invalid paths in PR #134147. Then I decided that this issue was a better fit and separated the tests into a separate PR #134189. And added more tests.

Now, whatever we decide, there are already tests that will show changes in behavior.

gh-117596: Add more tests for os.path with invalid paths (GH-134189)

pythongh-117596: Add more tests for os.path with invalid paths (pytho…

[3.13] gh-117596: Add more tests for os.path with invalid paths (GH-1…

[3.14] gh-117596: Add more tests for os.path with invalid paths (GH-1…

pythongh-117596: Add more tests for os.path with invalid paths (pytho…

I am hitting a similar issue related this one in my linux.

In Lib/test/test_posixpath.py, most tests with FileNotFoundError hits something like this:

ERROR: test_realpath_invalid_paths (__main__.PosixPathTest.test_realpath_invalid_paths)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_posixpath.py", line 510, in test_realpath_invalid_paths
    self.assertRaises(FileNotFoundError, realpath, path, strict=ALL_BUT_LAST)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/unittest/case.py", line 811, in assertRaises
    return context.handle('assertRaises', args, kwargs)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lamdef/cpython/Lib/unittest/case.py", line 245, in handle
    callable_obj(*args, **kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/posixpath.py", line 460, in realpath
    st_mode = lstat(newpath).st_mode
              ~~~~~^^^^^^^^^
ValueError: lstat: embedded null character in path

Those seems all related to st_mode = lstat(newpath).st_mode in posixpath.py#L460.

L591 hits FileNotFoundError correctly.
L604 and L605 hits PermissionError differently.
The rest cases failed by ValueError instead of FileNotFoundError as above log shows.

Further verification:

>>> import os
>>> os.lstat(b'/\xff')
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    os.lstat(b'/\xff')
    ~~~~~~~~^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: b'/\xff'

>>> os.lstat('/nonexistent/x\x00')
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    os.lstat('/nonexistent/x\x00')
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
ValueError: lstat: embedded null character in path

>>> os.lstat(b'/nonexistent/\xff')
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    os.lstat(b'/nonexistent/\xff')
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: b'/nonexistent/\xff'

When I tried on some random online IDE (e.g. https://www.online-python.com/) and macOS, I do see the behavior changed, b'/\xff' and '/nonexistent/x\x00' get the same outcome as above, but b'/nonexistent/\xff' gets the expected FileNotFoundError.

I sent a very customized PR that handles the specific exception my environment raises.

Merge branch 'main' into pythongh-117596

Do you have directory /nonexistent?

[Updated] I somehow found my machine has the folder /nonexistent...

Let me address this in the PR. (I removed PermissionError for L604 and L605, the rest changes are kept.)

Merge branch 'main' into pythongh-117596

I suggest to close this issue as "not a bug".

Navigation Menu

Repository navigation

`os.path.realpath` raises a `ValueError` when path contains a null byte #117596

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

os.path.realpath raises a ValueError when path contains a null byte #117596

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

stefanhoelzl commented on Apr 7, 2024

serhiy-storchaka commented on Apr 7, 2024

stefanhoelzl commented on Apr 7, 2024

serhiy-storchaka commented on Apr 7, 2024

encukou commented on Apr 8, 2024

serhiy-storchaka commented on May 18, 2025

gkirchou commented on Dec 8, 2025

serhiy-storchaka commented on Dec 8, 2025

gkirchou commented on Dec 9, 2025

serhiy-storchaka commented on Jun 4, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`os.path.realpath` raises a `ValueError` when path contains a null byte #117596