Bug report

Originally, the definition of the HTML format was not formally strict. It was similar to SGML and XML, but with a lot of looseness. HTMLParser tried its best to parse anything that looked like HTML. But after creation of HTML5, its specification defines the parsing rules for HTML documents, whether they are syntactically correct or not. It is important to follow these rules, for security reasons.

The current HTMLParser mainly follows the HTML5 specification, but there are a number of differences:

1. ✅ --!> should end the comment. gh-102555: Fix comment parsing in HTMLParser #135664
2. ✅ -- > should not end the comment. gh-102555: Fix comment parsing in HTMLParser #135664
3. ✅ <--> and <---> should be abnormally ended empty comments. gh-102555: Fix comment parsing in HTMLParser #135664
4. ✅ ] ]> and ]] > should not end the CDATA section. gh-135661: Fix CDATA section parsing in HTMLParser #135665
5. 🤕 CDATA handling should depend on the current node. This is important, because the ending condition are different for the CDATA section and the bogus comment (]]> and >).
6. ✅ Whitespaces should not be acceptable between </ and the tag name. E.g. </ script> should not end the script section. gh-135661: Fix parsing start and end tags in HTMLParser #135930
7. ✅ Vertical tabulation (\v) and non-ASCII whitespaces should not be recognized as whitespaces. The only whitespaces are \t\n\r\f . gh-135661: Fix parsing start and end tags in HTMLParser #135930
8. ✅ Null character (U+0000) should not end the tag name. gh-135661: Fix parsing start and end tags in HTMLParser #135930
9. Null character (U+0000), surrogate characters and many other special characters should be replaced by \xfffd. I think we can leave this, because it is easy to do in pre-processing or post-processing, and they usually do not cause issues in Python.
10. ✅ End tag can have attributes and slashes after tag name. It can not end after the first >. E.g. </script/foo=">"/>. gh-135661: Fix parsing start and end tags in HTMLParser #135930
11. ✅ Case-insensitive matching should only transform ASCII letters. E.g. </script> does not match </ſcript>, and LINK does not match LINK (the last letter is U+212A). gh-135661: Fix parsing start and end tags in HTMLParser #135930
12. ✅ There may be multiple slashes and whitespaces between the last attribute and closing > in both start and end tags. E.g. <a foo=bar/ //>. gh-135661: Fix parsing start and end tags in HTMLParser #135930
13. ✅ There should only be one = separator between attribute name and value. E.g. <a foo==bar> should have attribute "foo" with value "=bar". gh-135661: Fix parsing start and end tags in HTMLParser #135930
14. No whitespace should be acceptable between the = separator and attribute name and value. E.g. <a foo =bar> should have two attributes "foo" and "=bar", both with value None; <a foo= bar> should have two attributes: "foo" with value "" and "bar" with value None.

This can cause security issues for some programs. If the program uses HTMLParser to check the HTML input for dangerous code, it can miss some code. For example, "<!----!><script>...</script><!---->" is parsed by browsers as a script block surrounded by two comments, but the current HTMLParser parses it as a single comment.

Linked PRs

• gh-102555: Fix comment parsing in HTMLParser #135664
• gh-135661: Fix CDATA section parsing in HTMLParser #135665
• gh-135661: Fix parsing start and end tags in HTMLParser #135930
• [3.14] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) #136255
• [3.13] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) #136256
• [3.12] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) #136268
• [3.11] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) (GH-136268) #136291
• [3.10] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) (GH-136268) #136292
• [3.9] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) (GH-136268) #136293
• gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser #136908
• [3.13] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136918
• [3.12] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136919
• [3.11] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136920
• [3.10] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136921
• [3.9] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136922
• [3.14] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136927
• [3.14] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) #137772
• [3.13] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) #137773
• [3.12] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) #137774
• gh-135661: Fix parsing unterminated bogus comments in HTMLParser #137873
• [3.13] gh-135661: Fix parsing unterminated bogus comments in HTMLParser (GH-137873) #137875
• [3.11] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (GH-137774) #139659
• [3.10] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (GH-137774) #139660
• [3.9] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (GH-137774) #139661
• gh-135661: Fix abrupt closing of empty comment in HTMLParser #153007
• [3.15] gh-135661: Fix abrupt closing of empty comment in HTMLParser (GH-153007) #153024
• [3.14] gh-135661: Fix abrupt closing of empty comment in HTMLParser (GH-153007) #153025
• [3.13] gh-135661: Fix abrupt closing of empty comment in HTMLParser (GH-153007) #153026