Repository navigation
Multiple test failures with OSError: [Errno 84] Invalid or incomplete multibyte or wide character on ZFS with utf8only=on #81765
Description
| BPO | 37584 |
|---|---|
| Nosy | @gpshead, @vstinner, @benjaminp, @ezio-melotti, @serhiy-storchaka, @dimitern |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-07-13.10:13:56.997>
labels = ['type-bug', '3.9', '3.10', '3.11', 'tests', 'expert-unicode']
title = 'Multiple test failures with OSError: [Errno 84] Invalid or incomplete multibyte or wide character on ZFS with utf8only=on'
updated_at = <Date 2021-12-13.02:23:38.746>
user = 'https://github.com/dimitern'bugs.python.org fields:
activity = <Date 2021-12-13.02:23:38.746>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests', 'Unicode']
creation = <Date 2019-07-13.10:13:56.997>
creator = 'dimitern'
dependencies = []
files = ['48475']
hgrepos = []
issue_num = 37584
keywords = []
message_count = 5.0
messages = ['347794', '347801', '347998', '348006', '408420']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'vstinner', 'benjamin.peterson', 'ezio.melotti', 'serhiy.storchaka', 'dimitern']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37584'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
dimitern commentedon Jul 13, 2019
I'm running Ubuntu 19.04 on a ZFS mirrored pool, where my home partition is configured with 'utf8only=on' attribute. I've cloned cpython and after running the tests, as described in devguide.python.org, I have 11 test failures:
== Tests result: FAILURE ==
389 tests OK.
11 tests failed:
test_cmd_line_script test_httpservers test_imp test_import
test_ntpath test_os test_posixpath test_socket test_unicode_file
test_unicode_file_functions test_zipimport
I've been looking for similar or matching reported issues, but could not find one. I'm on the EuroPython 2019 CPython sprint and we'll be looking into this with the help of some of the core devs.
dimitern commentedon Jul 13, 2019
Here's some additional information I found for that specific attribute:
From the documentation at
http://dlc.sun.com/osol/docs/content/ZFSADMIN/gazss.html
(link is dead, but here's where I found the section below: https://zfs-discuss.opensolaris.narkive.com/3NqQVG0H/utf8only-and-normalization-properties#post1)
utf8only
Boolean
Off
This property indicates whether a file system should reject file names
that include characters that are not present in the UTF-8 character code
set. If this property is explicitly set to off, the normalization
property must either not be explicitly set or be set to none. The
default value for the utf8only property is off. This property cannot be
changed after the file system is created.
ezio-melotti commentedon Jul 16, 2019
I think Dimiter was able to fix most of the failures, except test_unicode_file_functions.
Yesterday during the sprints we were looking at it, and we did some tests using the following snippet:
On ext4 this creates 4 different files: ['\u03d4', '\u03d2\u0308', '\u03ab', '\u03a5\u0308']
On ZFS with utf8only=true (and I believe normalization=formD), only 2 files are created but each of the 4 filenames can be used to access either of the 2 files.
This is also the default behavior on Mac.
The test is already skipped on darwin (Lib/test/test_unicode_file_functions.py:120), and should be skipped for ZFS too (might depend on the exact flags used), however we weren't able to find a portable way to determine the filesystem and flags.
An alternative is to try creating the 4 files and skip the test if only 2 gets created and if all the names can be used to open these two files, however this might mask other failures. Unless someone can come up with a better way to do this, I think this is the only option.
In addition, different filesystems that don't exhibit this behavior can be used on Mac, so the test shouldn't be skipped in those cases.
vstinner commentedon Jul 16, 2019
"""
On ext4 this creates 4 different files: ['\u03d4', '\u03d2\u0308', '\u03ab', '\u03a5\u0308']
On ZFS with utf8only=true (and I believe normalization=formD), only 2 files are created but each of the 4 filenames can be used to access either of the 2 files.
This is also the default behavior on Mac.
The test is already skipped on darwin (Lib/test/test_unicode_file_functions.py:120), and should be skipped for ZFS too (might depend on the exact flags used), however we weren't able to find a portable way to determine the filesystem and flags.
"""
I suggest to create a temporary directory, create the 4 files and see how many files you can using os.listdir(). If you get 4, the FS doesn't normalize anything. If you get less, it's likely that the FS normalizes names.
gpshead commentedon Dec 13, 2021
Confirmed.
Repro: Do an ubuntu 20.04 install and choose "experimental zfs" support during install - https://ubuntu.com/blog/zfs-focus-on-ubuntu-20-04-lts-whats-new). On such a zfs filesystem, the following tests from a ./python -m test.regrtest run fail in 3.10:
11 tests failed:
test_cmd_line_script test_httpservers test_imp test_import
test_ntpath test_os test_posixpath test_socket test_unicode_file
test_unicode_file_functions test_zipimport
Move over to a tmpfs and all but test_httpservers now pass. test_httpservers tries to create such a path on /tmp
======================================================================
ERROR: test_undecodable_filename (test.test_httpservers.SimpleHTTPServerTestCase)
----------------------------------------------------------------------
I expect any filesystem mounted to reject non-UTF8 pathnames to cause similar failures. Our test suite needs to detect this environment and skip these tests there.
5 remaining items
erlend-aasland commentedon Oct 10, 2024
test_sqlite3has multiple "undecodable path" tests that fail on ZFS. Marking the following issues as duplicates of this issue:erlend-aasland commentedon Oct 10, 2024
ISTM we need a test support helper that can identify OS conditions such as this particular ZFS configuration. When that is in place, we can gate all "undecodable path" tests using this condition.
igoose1 commentedon Oct 30, 2024
Temporary workarounds if you need to build python on system with ZFS:
Build without optimizations.
OR build with optimizations in tempfs.
At the end, there's
./pythonbuilt with optimizations. This works because building is done in a different file system. Don't forget toumount buildpydirectory.Skip tests which interact with invalid UTF-8 files