Bug: binascii.a2b_uu incorrectly assumes padded bytes are always whitespace

Bug Description

I was decoding some UUEncoded data when I encountered a 'Trailing Garbage' error from the binascii.a2b_uu function. After digging into Linux's uu decode implementation(L248) and other resources (linked below) I'm decently certain the python implementation is bugged.

The following is what I tried:

from binascii import a2b_uu
s = '%-@     !'
decoded = a2b_uu(s)

The expected output is:

print(decoded)  # b'6\x00\x00\x00\x00'

The actual output is:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
binascii.Error: Trailing garbage

Notice there are 5 bytes in the expected output (b'6\x00\x00\x00\x00') because the % (first byte of input string, s) means 5 bytes of data follow (ascii code 37 - 32 = 5). UUEncoding requires output be divisible by 3 bytes so an extra padding character is added. In this case it's an !.

The python implementation assumes the padding is always whitespace. Different uuencoders will use different characters for padding though. I've seen three so far: , `, and !.

The following several lines of code are the issue

Proposed fix

Simply remove the following lines (279 - 296). Or if we really want the verification of padding we can include the '!' in the condition of valid padding chars. (The linked linux implementation does not verify padding, however.) And based on my research, there isn't a well defined padding character so we will be jumping to the same potentially false conclusion that we have here: believing we've accounted for all the padding characters that exist in the wild.

/*
** Finally, check that if there's anything left on the line
** that it's whitespace only.
*/
while( ascii_len-- > 0 ) {
    this_ch = *ascii_data++;
    /* Extra '`' may be written as padding in some cases */
    if ( this_ch != ' ' && this_ch != ' '+64 &&
         this_ch != '\n' && this_ch != '\r' ) {
        state = get_binascii_state(module);
        if (state == NULL) {
            return NULL;
        }
        PyErr_SetString(state->Error, "Trailing garbage");
        Py_DECREF(rv);
        return NULL;
    }
}

Problematically, this bug propagated up to the uu_codec decode implementation as well. See the following code

A comment indicates the caught exception and "workaround" are due to broken uuencoders. According to what I've read, it's the broken python binascii.a2b_uu that incorrectly assumes any padding bytes are or `.

Here are the sources for my understanding of uu encoding:
Examples of non whitespace padding
Wikipedia uuencoding
Busybox uudecode implementation

Following is an illustration that helped me find a sense of understanding:

[1] I couldn't find an RFC or other standards document so I looked for the earliest implementation I could find (1983 Linux implementation) along with the wikipedia entry.

In the meantime

If others encounter this issue I'm using the following workaround:

import binascii
from binascii import a2b_uu
from io import BytesIO

my_bytes = BytesIO()
line_bytes = b'%-@     !'
line = line_bytes.decode(encoding='ascii')
try:
    my_bytes.write(a2b_uu(line))
except binascii.Error as err:
    if 'trailing garbage' in str(err).lower():
        n_bytes = line_bytes[0] - 32
        assert n_bytes <= 45 and n_bytes <= len(line[1:])
        workaround_line = f'M{line[1:]}'  # replace first byte of UUEncoded line with max length specifier (M)
        data = a2b_uu(workaround_line)[:n_bytes]
        my_bytes.write(data)
    else:
        raise err

I guess the question is: are any of the padding bits allowed to be anything but zeros, or the special case (noted in the Wikipedia page) of using an accent grave?

What software did you see that used "!" as a padding character? I guess actually the padding is \x01, which gets converted to "!".

@ericvsmith

@ericvsmith I don't know which program generated the uuencoding for these, but public financial reports such as the following (scroll to the very bottom of the txt file) and thousands of other examples across SEC's EDGAR public database have UUEncoded data that uses the ! or as you've correctly stated, \x01, as its padding character.

I also found a StackOverflow post from over a decade ago raising the same issue here.

Maybe we could add a strict flag, with a default of True. But since the uu module and the uu_codec are scheduled for deprecation, I'm not sure it is worth the effort. See PEP 594.

That said, it looks like uu_codec will actually decode this currently:

>>> import codecs
>>> codecs.decode(b'begin\n%-@     !\nend\n', 'uu_codec')
b'6\x00\x00\x00\x00'

@ericvsmith

@ericvsmith I was noticing that as well, that the codecs module could decode the string. If you dig into it, it's using a similar workaround I produced. I just feel like the workaround shouldn't be necessary if the binascii module accepted other characters as padding like the linux and other implementations.

I agree though, a strict=True flag could maintain backwards compat while strict=False could be lenient on different encoders' idea of padding characters. Should we name it strict_padding for more clarity? Or is brevity preferred in this case?

Just FYI, the uu module is properly deprecated as of 3.11 (with removal scheduled for 3.13) per PEP 594 (PEP-594), though formal deprecation of the binascii functions and the uu codec in the codecs module is still pending in PR #92758 .

Given that there appear to be a couple niche legacy use cases, perhaps those would be better served by a dedicated PyPI module (perhaps based on the copy-pasted existing binascii and uu module code—#92758 is a pointer to that) developed and maintained by those, like yourself, with the most expertise with the format and who still have some real-world use for it. In the standard library, it has been effectively more or less unmaintained for many years, and you'd typically need to upgrade to a whole new Python feature version to gain new bug fixes, enhancements, etc—which I imagine is particularly undesirable for the legacy systems, applications and users still using this ancient and otherwise all-but-obsolete format.

You naturally get spaces (or backslashes) if you pad input by null bytes.

>>> binascii.b2a_uu(b'\x00\x00\x00')
b'#    \n'

You get ! at the end if you pad two bytes by \x01.

>>> binascii.b2a_uu(b'\x00\x00\x01')
b'#   !\n'

But if you pad one byte by two \x01 bytes, you get $! et the end:

>>> binascii.b2a_uu(b'\x00\x01\x01')
b'#  $!\n'

I do not know why one would use non-null bytes for padding -- the output depends on how many pad bytes are there. And the provided example only have one pad bytes -- what if they use different bytes to pad one input byte? Whatever they use and for what reason (maybe this is a kind of checksum or watermark), this is not a common uuencode, and we can only guess how they extend the uuencode format.

Python implementation is too strict on this issue. Other implementations do not check for "trailing garbage" if the length encoded in the first character is less than the line length, they simply ignore it. They can also produce different garbage if the encoded length is larger than the line length (Python silently fills it with null bytes). Python implementation also ignores newline characters in the middle of the line (other implementations can produce garbage). This is all beyond the specification.

We can add the strict_mode parameter, but
a) Python implementation is not strict in other ambiguous cases.
b) Different implementations can produce different output from the same non-canonical input. What should we imitate?

In 3.15 you could use a2b_base(line[1:].rstrip(b'\r\n'), strict_mode=True, alphabet=UU_ALPHABET, padded=False) and then truncate or pad the result depending on the length encoded in line[0]. This will be slower than using a2b_uu(), but much faster than doing decoding by bit manipulations in pure Python.

Navigation Menu

Repository navigation

Bug: `binascii.a2b_uu` incorrectly assumes padded bytes are always whitespace #100308

Bug Description

The following is what I tried:

The expected output is:

The actual output is:

Proposed fix

In the meantime

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Bug: binascii.a2b_uu incorrectly assumes padded bytes are always whitespace #100308

Description

Bug Description

The following is what I tried:

The expected output is:

The actual output is:

Proposed fix

In the meantime

Activity

ericvsmith commented on Dec 21, 2022

ajmedeio commented on Dec 21, 2022

ericvsmith commented on Dec 22, 2022

ajmedeio commented on Dec 22, 2022

CAM-Gerlach commented on Dec 22, 2022

serhiy-storchaka commented on Apr 4, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Bug: `binascii.a2b_uu` incorrectly assumes padded bytes are always whitespace #100308