Add utilities to "clean" surrogate code points from strings

@malemburg

BPO	18814
Nosy	@malemburg, @ncoghlan, @pitrou, @vstinner, @ezio-melotti, @stevendaprano, @bitdancer, @yaseppochi, @vadmium, @serhiy-storchaka
Dependencies	bpo-22286: Allow backslashreplace error handler to be used on input bpo-23676: Add support of UnicodeTranslateError in standard error handlers
Files	convert_surrogates.py codecs_convert_escapes.patch codecs_convert_escapes_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-08-23.04:02:31.968>
labels = ['3.8', 'type-feature', 'library']
title = 'Add utilities to "clean" surrogate code points from strings'
updated_at = <Date 2018-03-30.06:52:39.866>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2018-03-30.06:52:39.866>
actor = 'ncoghlan'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-08-23.04:02:31.968>
creator = 'ncoghlan'
dependencies = ['22286', '23676']
files = ['36700', '38506', '38520']
hgrepos = []
issue_num = 18814
keywords = ['patch']
message_count = 55.0
messages = ['195937', '195959', '195960', '196047', '196049', '196054', '196063', '225791', '225793', '225800', '225801', '225806', '225809', '225811', '225817', '225818', '225822', '225858', '225869', '225875', '225889', '225972', '227334', '227339', '227340', '227341', '227342', '227343', '227344', '227345', '227346', '227362', '227364', '227365', '227367', '227368', '227369', '238182', '238193', '238277', '238285', '238286', '238309', '242799', '242810', '242939', '251682', '251693', '251694', '251701', '251708', '251740', '251836', '251854', '314682']
nosy_count = 11.0
nosy_names = ['lemburg', 'ncoghlan', 'pitrou', 'vstinner', 'ezio.melotti', 'Arfrever', 'steven.daprano', 'r.david.murray', 'sjt', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18814'
versions = ['Python 3.8']

Linked PRs

gh-63014: Add utilites to transform surrogate codes and astral characters #121219

Prompted by bpo-18713 and http://lucumr.pocoo.org/2013/7/2/the-updated-guide-to-unicode/, here are some possible utilities we could add to the codecs module to help deal with/debug issues related to surrogate escaped strings:

    def has_escaped_bytes(s):
        """Returns true if string contains surrogate escaped bytes"""
        ...

    def replace_escaped_bytes(s):
        """Replaces each surrogate escaped byte with a valid code point"""
        ...

    def decode_escaped_bytes(s, nominal_encoding, actual_encoding):
        """Reinterprets incorrectly decoded text using a new encoding"""
        return s.encode(nominal_encoding, 'surrogateescape').decode(actual_encoding)

The email package needs has_escaped_bytes. Currently it tries to encode to ascii to find out if there are any, which we proved by microbenchmark is the fastest way to do it as things stand.

What does replace do? Replace it with the unknown character code point? Email also needs that.

These functions are trivial.

    def has_escaped_bytes(s):
        try:
            s.encode()
            return False
        except UnicodeEncodeError:
            return True

    def replace_escaped_bytes(s):
        return s.encode('utf-8', 'replace').decode()

Can you sum the use cases for these?
(don't want to read a blog post, sorry :-))

"The email package needs has_escaped_bytes. Currently it tries to encode to ascii to find out if there are any, which we proved by microbenchmark is the fastest way to do it as things stand."

In which function you need to check this? What do you do if there are surrogate characters?

The email package uses surrogateescape to store unknown bytes in unicode strings, just as with the handle-bad-data-from-os API surrogateescape was introduced for. (For the same reason: the source data may have improperly encoded bytes that we must nevertheless preserve). What action certain parts of the code takes differs depending on whether or not there are such encoded bytes in the string. So the code needs to check and branch based on that.

The utility function is email.utils._has_surrogates. You can grep the email source to see where it is used, if you like.

The use case is to take data from a surrogate escaped interface and either
filter it out entirely or convert it to a valid Unicode string at the point
of *input*, before letting it make its way into the rest of the
application. For example, this approach permits the Python equivalent of
the Linux console behaviour when it attempts to display the improperly
encoded file name.

This approach is likely to be easier to implement in cross-platform code
than dropping down to the bytes interfaces on POSIX systems and using the
Unicode APIs on Windows.

Based on the latest round of bytes handling discussions on python-dev, I came up with this updated proposal:

    # Constant in the string module (akin to string.ascii_letters et al)
    escaped_surrogates = bytes(range(128, 256)).decode('ascii', errors='surrogateescape')

    # Helper to ensure a string contains no escaped surrogates
    # This allows it to be safely encoded without surrogateescape
    _match_surrogates = re.compile('[{}]'.format(escaped_surrogates))
    def clean(s, repl='\ufffd'):
        return _match_surrogates.sub(repl, s)

    # Helper to redecode a string that was decoded incorrectly
    # For example, WSGI strings are passed from the server to the
    # framework as latin-1 by default and may need to be redecoded
    def redecode(s, encoding, errors='strict', old_encoding='latin-1', old_errors='strict'):
        return s.encode(old_encoding, old_errors).decode(encoding, errors)

In addition to the concrete use cases David describes, I think these will also serve a useful documentation purpose, in highlighting the two main mechanisms for "smuggling" raw binary data through text APIs (i.e. surrogate escapes and latin-1 decoding).

Note: I dropped the "has_escaped_bytes" idea, as "s != string.clean(s)" is functionally equivalent, albeit a bit more expensive if you don't actually want the cleaned string)

I think similar functions should be added in the unicodedata module rather than the string module or as str methods. If I'm not mistaken this was already proposed in another issue.
In C we already added macros like IS_{HIGH|LOW|}_SURROGATE and possibly others to help dealing with surrogates but AFAIK there's no Python equivalent yet.
As for the specific constants/functions/methods you propose, IMHO the name escaped_surrogates is not too clear. If it's a string of lone surrogates I would just call it unicodedata.surrogates (and .high_surrogates/.low_surrogates). These can also be used to build oneliner to check if a string contains surrogates and/or to remove them.
clean has a very generic name with no hints about surrogates, and its purpose is quite specific.
I'm also not a big fan of redecode. The equivalent calls to encode/decode are not much longer and more explicit. Also having to redecode often indicates that there's a bug before that should be fixed instead (if possible).

The purpose of these changes it to provide tools specifically for working with surrogate escaped data, not for working with arbitrary lone Unicode surrogates.

"escaped_surrogates" is not defined by the Unicode spec, it's defined by the behaviour of the surrogateescape error handler that lets us tunnel arbitrary bytes through str objects and reproduce them faithfully at the far end. On reflection, I think codecs would be a better home than string (as that's where the error handler is defined), but it doesn't belong in unicodedata.

I'd be OK with changing the name of the clean function to "clean_escaped_surrogates".

Needing redecode is not a bug: it's baked into the WSGI spec in PEP-3333. I would be OK with providing it in wsgiref rather than the codecs or string modules, but I think we should provide it somewhere.

I agree with Ezio in all points. escaped_surrogates is inefficient for any purposes and incomplete. _match_surrogates can be created in more efficient way. clean() has too general and misleading name. redecode() looks just ridiculous, this cumbersome 4-argument function just combine two methods.

Guys, you're Python 3 unicode experts, already thoroughly familiar with how surrogateescape works. These features are not for you.

The exact implementations don't matter. These need to exist, and they need to be documented with detailed explanations. People don't yet understand what the surrogateescape error handler is, and what it's for, or how to work with it.

People consider surrogate escaping this weird mysterious thing. It's not - it's a really simple mapping of the 128 bytes with the high order bit set to a different part of the Unicode code space. These functions make it obvious.

wsgiref.redecode is a natural consequence of the design of PEP-3333 - it makes sense to offer it there, as a hint that frameworks are likely to need it.

We have a serious, serious, learnability problem around binary data handling in Python 3. It's nowhere near as bad as the learnability problem for text handling in Python 2, but it still needs more scaffolding to help people grow in understanding.

That's why I think a function like redecode is a bad idea.
With Python 2 I've seen lot of people blindingly trying .decode when .encode failed (and the other way around) whenever they were getting an UnicodeError (and the fact that decoding Unicode results in an encode error didn't help).
I'm afraid that confused developers will try to (mis)use redecode as a workaround to attempt to fix something that shouldn't be broken in the first place, without actually understanding what the real problem is.
There's really no excuse for developers not to know Unicode nowadays (Joel Spolsky said the same thing more than 10 years ago).
IME people also tend to overestimate how difficult it is to understand Unicode. While it is true that you can go pretty deep down the rabbit hole, the basic concepts necessary for everyday use are pretty simple and straightforward.

[padding]

I think my suggested colours for the bikeshed would be handle_surrogates() and handle_surrogateescape(). “Rehandle” seems awkward and too assuming to me. And I agree with Serhiy that surrogates are a Unicode thing, not just related to the “surrogatepass” handler.

Adding them to “codecs” makes sense to me. The most important one, handle_surrogateescape() or equivalent, is closely related to the error handler of that module.

Having handle_surrogateescape or equivalent would probably be useful for bpo-25184 (displaying an arbitrary file path in a UTF-8 HTML file).

Hum, I suggest to put these functions in a package on PyPI, or recipes on a
website like stackoverfkow., and close the issue.

I'm still not convinced that these functions are useful . Usually we take a
function from an existing project used in applications to put it in the
stdlib. Here the use case still looks artifical. For example which
application requires to escape non-BMP character? How does it handle them
currently?

Threre are too many ways to handle surrogate characters. The common ways to
show undecodable bytes are not supported by functions proposed by Serhiy.
Example: %80 on Mac OS X. Gnome uses something else.

It was said that one reason to add new functions is performance. I'm not
convinced neither that such function is the bottleneck on any application.

I prefer to wait until users experiment with their own implementation and
see if a common function can be extracted from this to put it in the stdlib.

I think moving this forward mainly needs someone with the time and energy wrangle a python-ideas/dev discussion to get some additional feedback on the API design. As I see it, there are 2 main questions to be resolved:

Where to expose these functions

The default location would be the codecs module, as they're closely related to the error handlers in that module, and the main reasons for needing to clean data at all are handling dirty data produced by an interface that uses surrogatepass or surrogateescape when decoding (handle_surrogates, handle_surrogateescape), or encoding data for use in a context which doesn't correctly handle code points outside the basic multilingual plane (handle_astrals).

If added to the codecs module, they could be documented in new sections on "Postprocessing decoded text" and "Preprocessing text for encoding".

The main argument against that would be Stephen's one, which is that these aren't themselves encoding or decoding operations, but rather internal state manipulations on Python strings.

The exact function set to be provided.

The three potential data cleaning cases currently being considered:

process_surrogates: reprocessing all surrogates in the string, including lone surrogates and valid surrogate pairs. Such strings may be produced by using the "surrogatepass" handler when decoding, or by decomposing astral characters to surrogate pairs.
process_surrogateescape: reprocessing only lone surrogates in the U+DC80 to U+DCFF range, with other surrogate pairs or lone surrogates triggering UnicodeTranslateError. Such strings may be produced by using the "surrogateescape" error handler when decoding.
process_astrals: reprocessing all code points in the astral plane.

These seem to cover the essentials to me, and I changed the proposed prefix to "process_*" based on the idea of documentating them as preprocessing and postprocessing steps for encoding and decoding.

As far as the rationale for adding the functions at all goes, my main interest is still in having somewhere in the codecs module documentation to *define the problem*, and to my mind that entails also offering a simple way to do the relevant pre-/post-processing.

The nice aspect of building any related capabilities atop the standard error handlers is that it also means that third party modules can provide custom error handlers to support further escaping techniques, and those will also be available for use in decoding and encoding operations, rather than being specific to pre-/post-processing of the data.

However, it's also the case that we're generally going to be talking about the combination of encoding misconfiguration *and* processing data that gets potentially corrupted by the misconfiguration *and* doing something with it that isn't already handled by a surrogateescape round-trip, which is why I suspect in practice most applications are going to be able to get away with ignoring the problem entirely (especially with C.UTF-8 support coming to Fedora 24, so the Fedora/RHEL/CentOS ecosystem will be joining the Debian/Ubuntu ecosystem in offering that by default)

I also want "detect if there are any surrogates".

On Sun, Sep 27, 2015 at 04:17:45PM +0000, R. David Murray wrote:

I also want "detect if there are any surrogates".

I think that's useful enough it should be a str method. Here's a
pure-Python implementation:

def is_surrogate(s):
    return any(0xD800 <= ord(c) <= 0xDFFF for c in s)

The actual Flexible String Representation implementation can be even
more efficient. All-ASCII and all-Latin1 strings cannot possibly include
surrogates, so is_surrogate will be a constant-time operation.

(If we care about making this a constant-time operation for all
strings, the constructor could set a flag on the string object. Since
the constructor has to walk the string anyway, I don't think that will
cost much time-wise.)

I also want "detect if there are any surrogates".

Could you please open a separated issue for this function/method?

I believe that it's very different than other proposed functions/methods.

It was proposed before to add methods like "is_ascii()" but the request was rejected because other Python implementations don't implement Unicode using the PEP-393 and so the method would be less efficient than expected, depending on the implementation of Python.

Well, I don't know if we should add methods relying on the PEP-393 or not. It's probably better to discuss such "political" question on python-dev.

Done: bpo-25269.

With PEPs 538 and 540 implemented for 3.7, my thinking on this has evolved a bit.

A recent discussion on python-ideas [1] also introduced me to the third party library, "ftfy", which offers a wide range of tools for cleaning up improperly decoded data: https://ftfy.readthedocs.io/en/latest/

That includes a lone surrogate fixer: https://ftfy.readthedocs.io/en/latest/#ftfy.fixes.fix_surrogates

So a potential way to go here would be to a section on "Handling Improperly Decoded Text Data" to the codecs module documentation, and include ftfy as a See Also link in that new section.

If folks think that would be a reasonable way to go, then I think the clearest way to handle it would be to close this issue as "later" (which still implies "maybe never", but not as strongly as "rejected" does), and open a new issue for the suggested new section in the docs.

[1] https://mail.python.org/pipermail/python-ideas/2018-January/048583.html

pythongh-63014: Add utilites to transform surrogate codes and astral …

Navigation Menu

Repository navigation

Add utilities to "clean" surrogate code points from strings #63014

Linked PRs

37 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add utilities to "clean" surrogate code points from strings #63014

Description

Linked PRs

Activity

ncoghlan commented on Aug 23, 2013

bitdancer commented on Aug 23, 2013

serhiy-storchaka commented on Aug 23, 2013

pitrou commented on Aug 23, 2013

vstinner commented on Aug 23, 2013

bitdancer commented on Aug 23, 2013

ncoghlan commented on Aug 24, 2013

ncoghlan commented on Aug 24, 2014

ncoghlan commented on Aug 24, 2014

ezio-melotti commented on Aug 24, 2014

ncoghlan commented on Aug 24, 2014

serhiy-storchaka commented on Aug 24, 2014

ncoghlan commented on Aug 24, 2014

ezio-melotti commented on Aug 24, 2014

37 remaining items

vadmium commented on Sep 27, 2015

vstinner commented on Sep 27, 2015

ncoghlan commented on Sep 27, 2015

ncoghlan commented on Sep 27, 2015

bitdancer commented on Sep 27, 2015

stevendaprano commented on Sep 28, 2015

vstinner commented on Sep 29, 2015

bitdancer commented on Sep 29, 2015

ncoghlan commented on Mar 30, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions