Repository navigation
Python does not provide PyLong_FromIntMax_t() or PyLong_FromUintMax_t() function #62070
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2013-04-29.20:57:04.995>
labels = ['type-feature']
title = 'Python does not provide PyLong_FromIntMax_t() or PyLong_FromUintMax_t() function'
updated_at = <Date 2017-06-15.14:08:50.685>
user = 'https://github.com/ssbr'bugs.python.org fields:
activity = <Date 2017-06-15.14:08:50.685>
actor = 'Devin Jeanpierre'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2013-04-29.20:57:04.995>
creator = 'Devin Jeanpierre'
dependencies = []
files = ['30082', '30085', '30240', '30745', '45641']
hgrepos = []
issue_num = 17870
keywords = ['patch']
message_count = 39.0
messages = ['188103', '188105', '188106', '188111', '188112', '188114', '188118', '188126', '188142', '188149', '188168', '188189', '188194', '188196', '188689', '188704', '189051', '189066', '189418', '190693', '190696', '192163', '192182', '229533', '229535', '281706', '281707', '281708', '290684', '296040', '296042', '296043', '296054', '296066', '296089', '296093', '296094', '296095', '296096']
nosy_count = 8.0
nosy_names = ['mark.dickinson', 'pitrou', 'vstinner', 'benjamin.peterson', 'Devin Jeanpierre', 'serhiy.storchaka', 'yselivanov', 'josh.r']
pr_nums = ['857', '1106']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17870'
versions = ['Python 3.5']
Activity
ssbr commentedon Apr 29, 2013
As far as I can tell, the only safe and correct way to convert a (for example) intptr_t to a python long is something akin to the following:
Does this not seem absurd?
PyLong_FromIntMax_t(myinteger) would be great. Or maybe even better would be PyLong_FromBytes(&myinteger, sizeof(myinteger)) ?
This is problematic because many things that can interact with the Python C-API do not use the C types like "long long" or "int". Instead they might use the equivalents of intptr_t and int32_t, which are more reliably sized.
vstinner commentedon Apr 29, 2013
Did you try PyLong_FromVoidPtr()?
ssbr commentedon Apr 29, 2013
PyLong_FromVoidPtr works for uintptr_t, but not intptr_t.
vstinner commentedon Apr 29, 2013
"PyLong_FromVoidPtr works for uintptr_t, but not intptr_t."
Ok correct. You should use something like:
The "if (sizeof(intptr_t) == sizeof(long))" should be optimized by your compiler.
I don't know if Python should provide such function. What is your use case for intptr_t?
It looks like intptr_t is only really used in the implementation of os.spawnv() and os.spawnve(). Extract:
[-]Hard to create python longs from arbitrary C integers[/-][+]Python does not provide a PyLong_FromIntptr_t() function[/+]ssbr commentedon Apr 29, 2013
Isn't it possible for a >64-bit architecture to have intptr_t be wider than long long?
As for my use-case, I am wrapping the C-API for Rust. Rust can call and be called by C (and therefore Python), but a Rust "int" is a C "intptr_t", and a Rust "uint" is a C "uintptr_t".
ssbr commentedon Apr 29, 2013
With regards to the title change, I would prefer a FromIntMax_t (and FromUintMax_t) to a FromIntPtr_t. The former covers every use case of the latter, and more.
[-]Python does not provide a PyLong_FromIntptr_t() function[/-][+]Python does not provide PyLong_FromIntMax_t() or PyLong_FromUintMax_t() function[/+]vstinner commentedon Apr 29, 2013
The problem is to support platforms not providing a intmax_t type.
I don't know if the following C code would be a good "approximation" of the maximum integral value (signed/unsigned).
#elif SIZEOF_VOID_P <= SIZEOF_LONG
typedef long Py_intmax_t;
typedef unsigned long Py_uintmax_t;
#else
# error "Python needs a typedef for Py_intmax_t in pyport.h."
#endif /* HAVE_INTMAX_T */
If it is not, conversion from/to other types like off_t, time_t, pid_t or uid_t would loose information.
It is the same question than yours: is there a platform with an integer type wider than a pointer (intptr_t/void*)?
serhiy-storchaka commentedon Apr 30, 2013
It's x86. sizeof(void*) == 4, sizeof(long long) == 8.
vstinner commentedon Apr 30, 2013
Ah yes. So "SIZEOF_VOID_P <=" is not a good test. File position
(off_t) size can be 64 bit on a 32-bit system (like Linux/i386 and
Win32), whereas pointers size is 32 bit.
I don't think that we can rely on the availability of PY_LONG_LONG, it
may depend on the compiler or even on the compiler options.
IMO, if we decide to add functions for intmax_t and uintmax_t types,
the safest option is to only define functions if the real C type
(intmax_t and uintmax_t types) is available. I mean, we should not
guess intmax_t ourself, or we may choose the wrong type (too small
type).
vstinner commentedon Apr 30, 2013
There is a private function:
Where PY_LITTLE_ENDIAN is 1 on little-endian platform, and is_signed should be 0 or 1.
This function is inefficient (slow). It would be better to have a functions for intmax_t and uintmax_t types. Such functions would be enough to support other "recent" C types: intptr_t, uintptr_t, ptrdiff_t, etc. We may also add support for these types in PyArg_Parse* (ex: with "j" and "J" types, intmax_t and uintmax_t).
vstinner commentedon Apr 30, 2013
Some Google searchs told me that no CPU support 128-bit integer and
using 64-bit integer as intmax_t is safe.
GCC has a __int128 type, but I didn't find on which platform it is
supported, nor if intmax_t is __int128 in this case.
Microsoft Visual Studio has stdint.h since its version 2010 (which is
the version required to compile Python 3.4 on Windows according to the
Python developer guide).
I propose a safer definition of Py_intmax_t:
#elif defined(HAVE_LONG_LONG) && SIZEOF_LONG_LONG >= 8
typedef unsigned PY_LONG_LONG Py_uintmax_t;
typedef PY_LONG_LONG Py_intmax_t;
#else
# error "Python needs a typedef for Py_uintmax_t in pyport.h."
#endif
I don't think that a fallback on the long type is required, testing
size_t should be enough.
At least, the compilation fails if the Py_intmax_t type cannot be defined.
Having generic PyLong_FromUintMax_t() and PyLong_AsUintMax_t()
functions (and signed version) would simplify the support of other OS
types with variable size: time_t, clock_t, pid_t, gid_t, uid_t, off_t,
etc.
vstinner commentedon Apr 30, 2013
Here is a first patch adding the following functions:
I used AC_TYPE_INTMAX_T and AC_TYPE_UINTMAX_T in configure.ac, so intmax_t and uintmax_t are defined if the platform does not provide them. (I guess that these types are defined in pyconfig.h.) On Windows, Visual Studio 2010 supports stdint.h, and so these types are available.
autoheader && autoconf must be called to regenerate configure script and pyconfig.h.in template.
21 remaining items
ssbr commentedon Jun 14, 2017
Hold on, nobody ever answered the question in the OP. How would you convert an intptr_t (e.g. Rust's int type) to a Python int?
You can't use FromVoidPtr because of signedness. You can use FromLongLong, but that's implementation-defined.
If what we should be using is FromLongLong for all "really big ints", why not just rename FromLongLong to FromIntMax and call it a day?
There is no standard relationship between long long and most other int types -- all we know is that it's at least 64 bits, but an int type can perfectly reasonably be e.g. 80 bits or 128 bits or similar. I think it *is* a worhtwhile goal to allow programmers to write C code that has as little implementation-defined or undefined behavior as possible.
If that isn't considered a worthwhile goal, maybe we should reconsider using such a dangerous and pointy language as C. :)
vstinner commentedon Jun 14, 2017
Write your own C extension to do that. Sorry, I don't know what is the best way to write such C extension.
Maybe look at https://github.com/PyO3/PyO3 ? cc Yury
ssbr commentedon Jun 15, 2017
If everyone who wants to convert intptr_t to a python int has to write their own function, then why not just include it in the C-API?
Having support for intmax_t means we never have to have this conversation ever again, because it should work for all int types.
Reopening since this use-case doesn't sound solved yet.
serhiy-storchaka commentedon Jun 15, 2017
It may be better to make _PyLong_FromByteArray() and _PyLong_AsByteArray() public.
vstinner commentedon Jun 15, 2017
Me:
Devin Jeanpierre:
Hum, who else needs such function except of you?
CPython is maintained by volunteers. The maintenance has a high cost, so we need a strong rationale before adding new features. In this case, it was decided by core developers that the added complexity (amount of code) is not worth it since the use case is too rare.
Devin:
Devin, I asked you for a strong rationale to add the feature. I don't see such rationale, so this issue will be closed again.
Serhiy:
That makes sense. I suggest to open a new issue for that.
ssbr commentedon Jun 15, 2017
I guess we have different definitions of "strong rationale". Clearer criteria would help.
This request was part of the original bug report, so why open a new issue?
vstinner commentedon Jun 15, 2017
Making two C functions public is very different from supporting intmax_t. I expect a change of a few lines, whereas my intmax_t patch modified a lot of code.
I wanted to simplify the C code of CPython, I didn't care of the public support for intmax_t (but I considered that it would be nice to have).
The thing is that portability is complex, CPython wants to support as much platforms as possible. We only moved to C99 with Python 3.6, before stdint.h was not required. There are always minor compatibility issues, so it's simpler to not try to support an uncommon C type (intmax_t), and rather support a very generic functions "here are bytes, give me a Python int object".
It would solve your use case as well ;-)
ssbr commentedon Jun 15, 2017
Oh, to be clear on this last point:
Right now there is no way to convert an int that might be > 64 bits, into a python long, except really bizarre shenanigans, unless we want to rely on implementation-defined behavior.
This would be fine if it were easy to implement, but it isn't -- as we've both agreed, there's no good way to do this, and it is significantly easier to add this to CPython than to implement this from outside of CPython. And I do think there is merit in writing code that doesn't rely on implementation-defined behavior.
I also think it's simpler -- imagine if we just didn't care about all these int types! Phew.
Ack that this isn't "strong rationale" per your standards, so do whatever is right for this bug.
ssbr commentedon Jun 15, 2017
I requested either a way to create from intmax_t, or from bytes. We have two existing functions (that I didn't know about) to do the latter, so it would fix this bug report to just make those public, from my POV.
encukou commentedon Jan 28, 2025
In 3.13, @zooba added PyLong_FromNativeBytes. You can use:
.. and similar for fixed-width types,
__int128, or if you play with the flags, even that foreign-endian buffer you got over the 'net.(It does assume 2's complement (but so does the C23 standard), whole bytes, and no padding bits. We can't really test on platforms with exotic number representations.)
I consider this issue solved.