Skip to content

unicodedata.ucd_3_2_0.is_normalized incorrectly rejects NFD with a character unassigned in Unicode 3.2 #158763

Description

@augusto-rehfeldt

Bug report

Bug description:

Documented behaviour: Python unicodedata documentation, unicodedata.is_normalized: "Return whether the Unicode string unistr is in the normal form form." The unicodedata.ucd_3_2_0 entry says: "This is an object that has the same methods as the entire module, but uses the Unicode database version 3.2 instead."

Expected: ucd_3_2_0.is_normalized('NFD', 'A\u0315\u1ab0') returns True.

Actual: Returns False despite the string being NFD under Unicode 3.2.

import unicodedata as ud

u = ud.ucd_3_2_0
m = '\u1ab0'
s = 'A\u0315' + m

if not (u.category(m) == 'Cn' and u.combining(m) == 0
        and ud.combining(m) > 0):
    print('REFUTATION REJECTED: input does not meet the stated premises')
else:
    # These characters are not Hangul syllables. NFD requires no canonical
    # decompositions and nondecreasing combining classes between class-zero boundaries.
    classes = [u.combining(c) for c in s]
    decomps = [u.decomposition(c) for c in s]
    expected = (
        all(not d or d.startswith('<') for d in decomps)
        and all(b == 0 or a <= b for a, b in zip(classes, classes[1:]))
    )
    actual = u.is_normalized('NFD', s)
    if actual != expected:
        print('REFUTATION CONFIRMED:', repr(s), actual, expected)
    else:
        print('REFUTATION REJECTED: actual agrees with independent Unicode 3.2 NFD check')

Output on Python 3.14.6 (Windows-11-10.0.26220-SP0), standard library unicodedata:

REFUTATION CONFIRMED: 'A᪰̕' False True

This report was found and written by an automated property-testing tool I run (bugforge). The reproducer above was executed and its output is pasted unedited; no person reviewed the report before it was filed. The search script is in https://github.com/augusto-rehfeldt/bugforge-results/tree/main/unicodedata-20261003-064954-c1

CPython versions tested on:

3.14

Operating systems tested on:

Windows

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions