Repository navigation
literal ampersands in HTML text create ill-formed XML #80
Description
Activity
Hey Francis,
Thank you for participating in improving SimpleBrowser. I've been thinking about how to reply to this issue for more than 12 hours. Honestly, I've been having quite an argument with myself. My final conclusion, which is still open for debate, is this:
I agree that the XML in the SimpleBrowser XDocument in this case is not well-formed and that, in an ideal world, it should be well formed. That said, this is a case of "garbage in, garbage out". The HTML going into SimpleBrowser's HTML parser is not well formed. It is not the role of SimpleBrowser, or any browser, to clean up the source HTML. At best, the browser does what it needs to render what it's given - no more, no less.
Then, if it were the role of the browser to clean up HTML, simply replacing the "&" with "&" in both cases above is still incorrect, as doing so for the first instance would create invalid JavaScript. Someone would eventually report that as an issue. Then there's the issues for "You're escaping ampersand. Why aren't you escaping the 'X'?" This is potentially the beginning of a slippery slope.
In my usage of SimpleBrowser, it is not uncommon to clean up and pre-process malformed HTML to appease the SimpleBrowser parser. It is shocking to me how many large, global companies have web sites that may look nice but are rendered from crap HTML apparently written by a chimpanzee and an Atari 400. It is unfortunate, but if you really need to use XPath, I suggest you preform similar preprocessing to clean up the malformed HTML coming into your application.
I'm going to leave this issue open for further discussion. My opinion, at least at this point, is that escaping ampersands to produce valid XML in the XDocument will not be implemented. I reserve the right to change my opinion at any time.
Kevin
Interesting question. When parsing HTML inside SimpleBrowser, we do accept
non-wellformed input and make a tidy XML tree out of it. We use this tree
internally to implement Find functionality using XPath. So it is possible
to expose the valid XDocument. However, if we return the current HTML, it
shouldn't be converted to XML in the meantime.You could also argue that when parsing text nodes, we might as well convert
the to CDATA nodes. That would leave the infomodel intact and the
javascript would still be valid.Teun
Op 5 apr. 2014 13:51 schreef "Kevin Yochum" notifications@github.com:Hey Francis,
Thank you for participating in improving SimpleBrowser. I've been thinking
about how to reply to this issue for more than 12 hours. Honestly, I've
been having quite an argument with myself. My final conclusion, which is
still open for debate, is this:I agree that the XML in the SimpleBrowser XDocument in this case is not
well-formed and that, in an ideal world, it should be well formed. That
said, this is a case of "garbage in, garbage out". The HTML going into
SimpleBrowser's HTML parser is not well formed. It is not the role of
SimpleBrowser, or any browser, to clean up the source HTML. At best, the
browser does what it needs to render what it's given - no more, no less.Then, if it were the role of the browser to clean up HTML, simply
replacing the "&" with "&" in both cases above is still incorrect, as doing
so for the first instance would create invalid JavaScript. Someone would
eventually report that as an issue. Then there's the issues for "You're
escaping ampersand. Why aren't you escaping the 'X'?" This is potentially
the beginning of a slippery slope.In my usage of SimpleBrowser, it is not uncommon to clean up and
pre-process malformed HTML to appease the SimpleBrowser parser. It is
shocking to me how many large, global companies have web sites that may
look nice but are rendered from crap HTML apparently written by a
chimpanzee and an Atari 400. It is unfortunate, but if you really need to
use XPath, I suggest you preform similar preprocessing to clean up the
malformed HTML coming into your application.I'm going to leave this issue open for further discussion. My opinion, at
least at this point, is that escaping ampersands to produce valid XML in
the XDocument will not be implemented. I reserve the right to change my
opinion at any time.Kevin
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/80#issuecomment-39635867
.Thanks Kevin - if only for reminding me that there was a time when I aspired to an Atari 400 - I ended up with a Jupiter Ace...
Thanks Teun - was I hasty in reporting a bug? Is the XPath-compatible XML view of malformed documents already exposed somewhere?
And to follow up on one angle, your excellent Find() method has done the trick for me, so this is no longer an issue for me personally, but if there's anything I can do to help, let me know.
Teun said:
... when parsing text nodes, we might as well convert the to CDATA nodes. That would leave the infomodel intact and the javascript would still be valid.
This suggestion is intriguing. If I understand your suggestion, this would not be escaping any inner text. Rather, it would wrap the inner text in a CDATA node for the sole purpose of creating a guaranteed valid XDocument. Is that correct?
I would recommend making this an optional SimpleBrowser feature, which is disabled by default. That would allow the XDocument to either be created strictly from the HTML source as it is now, or as XML for use with XPath.
In working on Issue #119, if this issue is implemented using a CDATA node to enclose text nodes, keep in mind that if an option element does not have a value attribute defined, the text is used as the value of the option element. That is, don't break options elements! :)
Hi guys,
Thanks for the excellent tool. I wanted to do some XPath so tried to use the browser.XDocument property but the XML I got back was not well formed because the HTML included literal ampersands, eg:
(Paste the above into http://www.xpathtester.com/xpath and hit format to illustrate the issue)
Probably not urgent for me - I'm going to try non-xpath methods like Find() - but it might be a good idea to escape any literal ampersands as part of your XML Parse method:
Thanks again for the great component!
Francis.