Getting a buffer from a Unicode array uses invalid format

[vstinner]

BPO	13072
Nosy	@loewis, @birkenfeld, @mdickinson, @ncoghlan, @pitrou, @vstinner, @skrah, @meadori
Files	array_revert_pep393.patch array_revert_pep393-2.patch array_unicode_format.patch array_deprecate_u.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2012-08-24.18:22:37.401>
created_at = <Date 2011-09-30.00:09:50.065>
labels = ['type-bug', 'library', 'release-blocker']
title = 'Getting a buffer from a Unicode array uses invalid format'
updated_at = <Date 2012-08-24.18:22:37.400>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2012-08-24.18:22:37.400>
actor = 'skrah'
assignee = 'vstinner'
closed = True
closed_date = <Date 2012-08-24.18:22:37.401>
closer = 'skrah'
components = ['Library (Lib)']
creation = <Date 2011-09-30.00:09:50.065>
creator = 'vstinner'
dependencies = []
files = ['26646', '26647', '26704', '26892']
hgrepos = []
issue_num = 13072
keywords = ['patch']
message_count = 45.0
messages = ['144658', '144812', '144814', '144817', '144818', '158381', '158892', '167091', '167109', '167112', '167119', '167122', '167165', '167173', '167520', '167521', '167522', '167540', '167545', '167546', '167547', '167549', '167551', '167561', '167566', '167571', '167673', '167702', '167703', '167708', '167732', '167936', '167947', '167997', '168005', '168369', '168373', '168558', '168561', '168567', '168571', '168575', '169026', '169063', '169065']
nosy_count = 10.0
nosy_names = ['loewis', 'georg.brandl', 'mark.dickinson', 'ncoghlan', 'pitrou', 'vstinner', 'Arfrever', 'skrah', 'meador.inge', 'python-dev']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13072'
versions = ['Python 3.3']

Linked PRs

• gh-57281: Fix duplicate deprecation note #133113
• [3.13] gh-57281: Fix duplicate deprecation note: array 'u' format (GH-133113) #133147

[vstinner]

In Python 3.2, when you get a buffer from array.array('u'), "u" is used as buffer format. The format is supposed to be a format from the struct module, and "u" is an invalid struct format. "w" is used on wide mode.

I just upgraded the array module to use the new Unicode API (PEP-393). The array now uses a Py_UCS4 buffer. I used "I" or "L" format depending on the size of int and long C types.

It would be better to use a format for a Py_UCS4 string, but struct doesn't support such type.

For Python 2.7 and 3.2, I don't know if it is really a bug or not.

[skrah]

The automatic conversion of 'u' to 'I' or 'L' causes test_buffer
(PEP-3118 repo) to fail:

# Not implemented formats. Ugly, but inevitable. This is the same as
# issue python/cpython#46783: equality is also used for membership testing and must
# return a result.
a = array.array('u', 'xyz')
v = memoryview(a)
self.assertNotEqual(v, a)
self.assertNotEqual(a, v)

I don't have a better idea though what to do about 'u' except
officially implementing it for struct and memoryview as well.

[skrah]

It would be better to use a format for a Py_UCS4 string, but struct doesn't support such type.

PEP-3118 suggests for the extended struct syntax:

'c' -> ucs-1 (latin-1) encoding
'u' -> ucs-2
'w' -> ucs-4

[vstinner]

The automatic conversion of 'u' to 'I' or 'L' causes test_buffer
(PEP-3118 repo) to fail:

Not implemented formats. Ugly, but inevitable. This is the same as

issue bpo-2531: equality is also used for membership testing and must

return a result.

a = array.array('u', 'xyz')
v = memoryview(a)
self.assertNotEqual(v, a)
self.assertNotEqual(a, v)

I don't understand: a buffer format is a format for the struct module,
or for the array module?

[skrah]

STINNER Victor <report@bugs.python.org> wrote:

> # Not implemented formats. Ugly, but inevitable. This is the same as
> # issue bpo-2531: equality is also used for membership testing and must
> # return a result.
> a = array.array('u', 'xyz')
> v = memoryview(a)
> self.assertNotEqual(v, a)
> self.assertNotEqual(a, v)

I don't understand: a buffer format is a format for the struct module,
or for the array module?

It's like this: memoryview follows the current struct syntax, which
doesn't have 'u'. memory_richcompare() does not understand 'u', but
is required to return something for __eq__ and __ne__, so it returns
'not equal'.

This isn't so important, since I discovered (see my later post)
that 'u' and 'w' were scheduled for inclusion in the struct
module anyway.

So I think we should focus on whether the proposed 'c', 'u' and 'w'
format specifiers still make sense after the PEP-393 changes.

[vstinner]

@Stefan: What is the status of this issue?

[skrah]

I'm not sure what to do. Martin's opinion was that the change should
be reverted:

http://mail.python.org/pipermail/python-dev/2012-March/117390.html

[vstinner]

Should we do something before Python 3.3 final?

[skrah]

Is it possible without too much effort to keep the old behavior
('u' -> Py_UNICODE)? Then I'd say that should go into 3.3.

The problem with the current behavior is that it's neither backwards
compatible nor PEP-3118 compliant.

If it is too much work to restore the status quo, we could leave this
change with the excuse that 'u' is probably not used very often.

[vstinner]

Here is a patch reverting changes of the PEP-393, as suggested by Martin von Loewis. With the patch, array uses Py_UNICODE* type for the 'u' format. So array.array('u', '\u0010ffff')[0] should return '\uDBFF' on Windows.

[skrah]

The diff between b9558df8cc58 and default with array_revert_pep393.patch
applied is small, but I noticed that in some places you switched back to
Py_UNICODE typecode and in others not. For instance, in struct arraydescr
typecode is still char.

I'm not sure why typecode was originally Py_UNICODE though.

[vstinner]

The diff between b9558df8cc58 and default with array_revert_pep393.patch
applied is small, but I noticed that in some places you switched back to
Py_UNICODE typecode and in others not.

I just copied code from Python 3.2, I forgot to update typecode type
(Py_UNICODE => char). I attach a new patch which changes also the
documentation of the "u" format.

[skrah]

array_revert_pep393-2.patch looks good (checked against 7042a83f37e
and all following commits that should be kept).

[vstinner]

@georg: are you ok with this change? It reverts the behaviour of Python 3.2 and avoids to have to maintain an API that nobody wants to use ('u' format using Py_UCS4, 32 bits unsigned).