Use 64-bit integers for bit counts

Some internal C functions represent the number of bits in the Python integer as size_t or Py_ssize_t. It is fine on 64-bit platforms, where you need exbibytes of memory to get an overflow error. But on 32-bit platform you can create an integer objects that has a size of just 0.5 GiB.

This problem can be solved if always use 64-bit integers (uint64_t or int64_t) for bit counts. We can even introduce a hard limit for the range of integers in CPython (to $2^{2^{64}-1}$ or $2^{2^{63}-1}$) and remove the possibility of overflow error in _PyLong_NumBits() and _PyLong_Frexp(). No existing 64-bit platform supports such large address space, and even if they support, it would take years to create a single integer object of such size (just to fill memory).

Linked PRs

gh-121485: Always use 64-bit integers for integers bits count #121486

pythongh-121485: Always use 64-bit integers for integers bits count

Currently math functions isqrt(), log(), log2() and log10() fail for integer argument 2**32 on 32-bit platforms, as shown in new tests #121495:

======================================================================
ERROR: test_isqrt_large (test.test_math.MathTests.test_isqrt_large)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Lib/test/test_math.py", line 1127, in test_isqrt_large
    w = math.isqrt(v)
OverflowError: int has too many bits to express in a platform size_t

======================================================================
ERROR: test_log_large_integer (test.test_math.MathTests.test_log_large_integer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Lib/test/test_math.py", line 1276, in test_log_large_integer
    self.assertAlmostEqual(math.log2(v), bits)
                           ~~~~~~~~~^^^
OverflowError: huge integer: number of bits overflows a Py_ssize_t

----------------------------------------------------------------------

@mdickinson

cc @mdickinson

PyLong_AsNativeBytes() also fails for such integer.

======================================================================
ERROR: test_long_asnativebytes_large (test.test_capi.test_long.LongTests.test_long_asnativebytes_large)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Lib/test/test_capi/test_long.py", line 461, in test_long_asnativebytes_large
    r = asnativebytes(v, buffer, 0, -1)
OverflowError: int has too many bits to express in a platform size_t

----------------------------------------------------------------------

gh-121485: Always use 64-bit integers for integers bits count (GH-121486

Navigation Menu

Repository navigation

Use 64-bit integers for bit counts #121485

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Use 64-bit integers for bit counts #121485

Description

Linked PRs

Activity

serhiy-storchaka commented on Jul 8, 2024

serhiy-storchaka commented on Jul 8, 2024

serhiy-storchaka commented on Jul 8, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions