Repository navigation
base64.b85encode uses significant amount of RAM #101178
Description
Bug report
On the same string:
- b64encode: RAM: 604MB, CPU: 79%, Time taken: 1.18sec and gave a result
- b85encode: RAM: 6.9GB, CPU: 64%, Time taken: 45sec and crashed due to insufficient RAM (most likely)
b85encode takes up my entire RAM and crashes
On IPython:
In [1]: import base64
In [2]: contents = b'a' * 250 * 1024 * 1024
In [3]: %time len(base64.b64encode(contents).decode("ascii")) / 1024 / 1024
CPU times: user 489 ms, sys: 414 ms, total: 904 ms
Wall time: 974 ms
Out[3]: 333.33333587646484
In [4]: %time len(base64.b85encode(contents).decode("ascii")) / 1024 / 1024
Killed
Here is the GNU time stats:
$ /usr/bin/time /opt/python3.9/bin/python -c "import base64; print(len(base64.b64encode(b'a' * 250 * 1024 * 1024)) / 1024 / 1024)"
333.33333587646484
0.55user 0.39system 0:01.18elapsed 79%CPU (0avgtext+0avgdata 604008maxresident)k
0inputs+0outputs (0major+151155minor)pagefaults 0swaps
$ /usr/bin/time /opt/python3.9/bin/python -c "import base64; print(len(base64.b85encode(b'a' * 250 * 1024 * 1024)) / 1024 / 1024)"
26.08user 3.39system 0:45.86elapsed 64%CPU (0avgtext+0avgdata 6966080maxresident)k
512inputs+0outputs (2major+1756895minor)pagefaults 0swaps
I have gotten same results in Python 3.6, 3.7, 3.8, 3.9
Your environment
- CPython versions tested on: Python 3.9 installed with miniconda
- Operating system and architecture: CentOS 7, AWS t3.large machine
Activity
sobolevn commentedon Jan 23, 2023
I can confirm, that this still happens on
main:Compare it to almost-instant
64version (it usesbinascii's C implementation):eendebakpt commentedon Feb 1, 2023
I looked into the implementation of
b85encode(in_85encode). The memory problems are caused by the fact that internally copies of the data are made. The copies consist of small chuncks, where each chuck has the memory overhead of a python object.words = struct.Struct('!%dI' % (len(b) // 4)).unpack(b). There is aiter_unpackmethod in theStructclass, but it does not work as expected.chunks = [ ... for word in words]. In the casepadding=Falsewe can replace this with a generator which would save memory. Forpadding=Truewe need to modify the last element of the list, which does not work for the generator.The poor performance (compared to b64encode) is mainly due to the part
(chars2[word // 614125] + chars2[word // 85 % 7225] + chars[word % 85]).To make any significant gains an implementation in C seems the most worthwhile.
Improve performance of Ascii85 and base85
Add Ascii85 and base85 support to binascii
Add Ascii85 and base85 support to binascii
Merge branch 'main' into pythongh-101178-rework-base85
pythongh-101178: refactor base64.b85encode to be memory friendly
pythongh-101178: refactor base64.b85encode to be memory friendly
pythongh-101178: refactor base64.b85encode to be memory friendly
Merge remote-tracking branch 'upstream/main' into pythongh-101178-rew…
Add Ascii85, base85, and Z85 support to binascii
kangtastic commentedon Apr 21, 2025
Pinging this issue in hopes of finding a reviewer for my PR gh-102753. It's been over two years 😅
1 remaining item
Merge branch 'main' into pythongh-101178-rework-base85
Merge branch 'main' into pythongh-101178-rework-base85
Merge branch 'main' into pythongh-101178-rework-base85
Merge branch 'main' into pythongh-101178-rework-base85
Merge branch 'main' into pythongh-101178-rework-base85
gh-101178: Add Ascii85, Base85, and Z85 support to binascii (GH-102753)
pythongh-101178: Fix possible integer overflow in Ascii85 encoder wit…
pythongh-101178: Add Ascii85, Base85, and Z85 support to binascii (py…
gh-101178: Fix possible integer overflow in Ascii85 encoder with wrap…
pythongh-101178: Fix possible integer overflow in Ascii85 encoder wit…
gh-101178: Fix UB in `binascii.b2a_ascii85` (#148166)
pythongh-101178: Add Ascii85, Base85, and Z85 support to binascii (py…
pythongh-101178: Fix possible integer overflow in Ascii85 encoder wit…
pythongh-101178: Fix UB in `binascii.b2a_ascii85` (python#148166)