Repository navigation
implement PEP 3118 struct changes #47382
Description
Activity
MrJean1 commented
on Jun 21, 2008 MrJean1mannequinMannequinIf the struct changes are made, add also 2 formats for C types ssize_t and
size_t, perhaps 'z' resp. 'Z'. In particular since on platforms
sizeof(size_t) != sizeof(long).It's looking pessimistic that this is going to make it by beta 3. If
they can't get in by then, it's too late.Let's retarget it to 3.1 then. It's a new feature, not a behaviour
change or a deprecation, so adding it to 3.0 isn't a necessity.Actually, this may be a requirement of bpo-2394; PEP-3118 states that
memoryview.tolist would use the struct module to do the unpacking.:-(
However, we don't have any examples of the buffer API / memoryview
object working with something else than 1-dimensional contiguous char
arrays (e.g. bytearray). Therefore, I suggest that Python 3.0 provide
official support only for 1-dimensional contiguous char arrays. Then
tolist() will be easy to implement even without using the struct module
(just a list of integers, if I understand the functionality).teoliphant commented
on Aug 24, 2008 teoliphantmannequinMannequinThis can be re-targeted to 3.1 as described.
devdanzin commented
on May 16, 2009 devdanzinmannequinMannequinTravis,
Do you think you can contribute for this to actually land in 3.2? Having
a critical issue slipping from 3.0 to 3.3 would be bad...Does this supersede bpo-2395 or is this a subset of that one.?
Is anyone working on implementing these new struct modifiers? If not, then I would love to take a shot at it.
2010/2/12 Meador Inge <report@bugs.python.org>:
Meador Inge <meadori@gmail.com> added the comment:
Is anyone working on implementing these new struct modifiers? If not, then I would love to take a shot at it.
Not to my knowledge.
46 remaining items
Is this work something that might be suitable for the features/pep-3118 repo (http://hg.python.org/features/pep-3118/) ?
skrah commented
on Sep 5, 2011 skrahmannequinMannequinYes, definitely. I'm going to push a new memoryview implementation
(complete for all 1D/native format cases) in a couple of days.Once that is done, perhaps we could create a memoryview-struct
branch on top of that.Following up here after rejecting bpo-15622 as invalid
The "unicode" codes in PEP-3118 need to be seriously rethought before any related changes are made in the struct module.
-
The 'c' and 's' codes are currently used for raw bytes data (represented as bytes objects at the Python layer). This means the 'c' code cannot be used as described in PEP-3118 in a world with strict binary/text separation.
-
Any format codes for UCS1, UCS2 and UCS4 are more usefully modelled on 's' than they are on 'c' (so that repeat counts create longer strings rather than lists of strings that each contain a single code point)
-
Given some of the other proposals in PEP-3118, it seems more useful to define an embedded text format as "S{<encoding>}".
UCS1 would then be "S{latin-1}", UCS2 would be approximated as "S{utf-16}" and UCS4 would be "S{utf-32}" and arbitrary encodings would also be supported. struct packing would implicitly encode from text to bytes while unpacking would implicitly decode bytes to text. As with 's' a length mismatch in the encoded form would mean an error.
-
paulehoffman commented
on Apr 22, 2013 paulehoffmanmannequinMannequinFollowing up on http://mail.python.org/pipermail/python-ideas/2011-March/009656.html, I would like to request that struct also handle half-precision floats directly. It's a short change, and half-precision floats are becoming much more popular in applications.
Adding this to struct would also maybe need to change math.isinf and math.isnan, but maybe not.
Paul: there's already an open issue for adding float16 to the struct module: see bpo-11734.
paulehoffman commented
on Apr 22, 2013 paulehoffmanmannequinMannequinWhoops, never mind. Thanks for the pointer to 11734.
skrah commented
on Apr 13, 2016 skrahmannequinMannequinHere's a grammar that roughly describes the subset that NumPy supports.
As for implementing this in the struct module: There is a new data
description language on the horizon:http://datashape.readthedocs.org/en/latest/
It does not have all the low-level capabilities (e.g changing alignment
on the fly), but it is far more readable. Example:PEP-3118: "(2,3)10f0fZdT{10B:x:(2,3)d:y:Q:z:}B"
Datashape: "2 * 3 * (10 * float32, 0 * float32, complex128, {x: 10 * uint8, y: 2 * 3 * float64, z: int64}, uint8)"There are a lot of open questions still. Should "10f" be viewed as an
array[10] of float, i.e. equivalent to (10)f?In the context of PEP-3118, I think so.
It's been more than ten years now and most of the additions to
structproposed by PEP-3118 (https://peps.python.org/pep-3118/#additions-to-the-struct-string-syntax) have not been implemented. The?andccodes have been added, though.At this point, I would propose to close the issue. If there is interest in adding new codes, that should be discussed in a new feature with a more specific motivation. It doesn't make sense to implement them based on a PEP from more than a decade ago.
I'll note that Numpy does seem to support at least part of these additions:
>>> dt = np.dtype([('x', np.float64), ('y', np.float64)]) >>> a = np.array([(1, 2)], dtype=dt) >>> m = memoryview(a) >>> m.format 'T{d:x:d:y:}'
If there is interest in adding new codes, that should be discussed in a new feature with a more specific motivation. It doesn't make sense to implement them based on a PEP from more than a decade ago.
It would make sense to support at least what Numpy supports.
Also note that
ctypessupports?,g,O(...),T{...}, and:name:but chose:cforchar, notucs-1stringuforwchar_t, notucs-2stringC,E,F(notZd,Zf,Zg) for complex types (cc @skirpichev -- just FYI)
>>> import ctypes >>> class C(ctypes.Structure): ... _fields_ = [(f'f{num}', type) for num, type in enumerate([ ... ctypes.c_bool, ... ctypes.c_longdouble, ... ctypes.py_object, ... ctypes.c_double_complex, ... ctypes.c_wchar * 2, ... ])] ... >>> memoryview(C()).format 'T{<?:f0:15x<g:f1:<O:f2:<C:f3:(2)<u:f4:}'
There are formats in the wild that
structcan't parse. Makingstructparse them in an incompatible way could lead to data corruption.IMO, at this point, standardizing needs a new PEP.
Reacted by Alyssa Coghlan and Jelle ZijlstraC, E, F (not Zd, Zf, Zg) for complex types
That was discussed, see e.g. here. Unfortunately, the
fielddescstruct seems to be supporting only single-character names.Reacted by Petr Viktorin
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: