Repository navigation
Try UnicodeDammit instead of using chardet library (or maybe combine the approaches). #42
Description
Activity
Can UnicodeDammit guess the incorrect encoding specified in the headers?
You mean guess the correct encoding if the wrong encoding or no encoding was specified in the HTTP headers? The problem with the encoding specification in the HTTP header is that people often forget to set it. The default according to the HTTP specification is ISO-8859-1, which is rarely the actual encoding especially when you’re dealing with non-English pages.
I’m using the requests library in just about all of my programs, which “correctly” assumes ISO-8859-1 in such cases. What my programs do to actually decode the page is that they confirm with the actual content-type header whether the charset was set explicitly or whether it was inferred. If it was inferred, they ignore it. Then they call
UnicodeDammitwith the encoded response as markup and the HTTP-level encoding inoverride_encodings. That library then tries countless sources of explicit encoding specifications in the (HTML or XML) content.Two problems remain, however. If none of the tried encodings could decode the response, I’m still in luck because the program actually knows that the decoding was unsuccessful; if, however, one of the incorrect encodings was coincidentally able to decode the page, no decoding errors are raised but the text is garbled. Barring comparisons of language-specific character n-gram models, I can’t think of any automatic method to detect such errors.
After all the sources of explicit encoding specifications have been tried,
UnicodeDammitwill try to use cchardet if it is available or chardet if it is not. If none is available, it’ll skip that step, which is what I’m currently doing for mostly German and English content. So far no one has alerted me of any decoding errors, and we’re fetching some 100k pages per day, so it must be working pretty okay.Here’s some similar code from a private project of mine: https://bitbucket.org/Telofy/resyndicator/src/b0fdce864919bbbf68561142442428e09fb26112/resyndicator/fetchers.py?at=master#cl-52.
The wrapper (only needed anymore to raise the exception): https://bitbucket.org/Telofy/utilofies/src/4a1852218b7fe59fbb28fb1bad5a7b26be2a5a46/utilofies/bslib.py?at=master#cl-14.Thanks, that’s very interesting! I should run a few more tests with cchardet and preprocessed HTML (with tags stripped); that’ll probably restore my confidence into that method. I’ve only been working with English and German sites so far—apart from some that may have found their way in by accident—and the only kind of “lie” that I often encountered in that area was when no HTTP encoding was specified.
There was another problem with my approach, which forced me to reimplement some small parts of UnicodeDammit: https://bitbucket.org/Telofy/utilofies/src/0d8cdc3ae5a0a08e7fb5906d96f0d8e2284751d1/utilofies/bslib.py?at=master#cl-15.
The encoding problem was reported to me, which usually means that it must’ve occurred on a number of pages, and I vaguely remember that I already ran into it and solved it for the old UnicodeDammit sometime in 2011.
When a page is declared, say, UTF-8 consistently everywhere (or anywhere) but contains a single illegal byte sequence—for example in an HTML comment like this one—then the “correct” encoding, UTF-8, is discarded. Moreover, Windows-1252 is somehow able to decode it, so that all umlauts and ligatures are mucked up.
When all declared encodings fail, I now immediately fall back on forcing the first one of them. Only if no encodings at all were declared anywhere do I fall back on UTF-8. I hope this will alleviate the problem.
I believe now this looks more like a package to deal only with the document encoding, based on document text, meta encoding and HTTP responses, that readability module could import it and reuse. What do you think?
Could you make such package?
