Skip to content

postprocess book v2 for entities in code blocks - #755

Merged
peff merged 1 commit into
git:masterfrom
jnavila:fix_entities
May 6, 2016
Merged

peff merged 1 commit into
git:masterfrom
jnavila:fix_entities

Conversation

@jnavila

@jnavila jnavila commented May 3, 2016

Copy link
Copy Markdown
Contributor

Right now Atlas seems to badly handle html entities in code block so we
fix it when importing it into the site. This hopefully fixes #753.

@peff

peff commented May 3, 2016

Copy link
Copy Markdown
Member

So this is moderately less hacky than the solution in #740, because at least it's happening on the backend rather than via javascript. But I have the same concerns as there that this is going to hit false positives.

I guess an interesting exercise would be to do the regex substitution on a downloaded copy of the ebook, render the results to text, and then diff the results. Eyeballing that diff should give us a sense of what's improving and what's regressing.

I suspect it will be a net gain, but I think it's worth trying to verify that.

@jnavila

jnavila commented May 4, 2016

Copy link
Copy Markdown
Contributor Author

Of course, this is a hack and I expect it to be reverted sometimes in the future.

I checked where the double escaping appears in the html output from Atlas. This patch does not cover all the cases because it only checks code tags, but the double escaping also appears in pre sections. So I think I'll have to reroll it. In fact, we can drop the condition and make it simpler.

Right now Atlas seems to badly handle html entities in code block so we
fix it when importing it into the site. This hopefully fixes git#753.
@jnavila

jnavila commented May 4, 2016

Copy link
Copy Markdown
Contributor Author

OK, the new simpler version of the patch covers all the cases I detected.

@peff

peff commented May 6, 2016

Copy link
Copy Markdown
Member

Thanks for working on this. I was thinking we'd match real > escapes, which is what worried me about false positives. But since it's the more-obviously-wrong >, that's a pretty good indication that it's bogus encoding, and not something somebody intended. I skimmed the grep output, and didn't see any false positives (and tons of improvements). So I think it's a net benefit.

I will still look forward to one day reverting it, though. :)

@jnavila

jnavila commented May 6, 2016

Copy link
Copy Markdown
Contributor Author

Yes, the number of new issues will decrease a lot. This is really an annoying part of the way the book is edited. Thank you very much for taking care.

@peff

peff commented May 6, 2016

Copy link
Copy Markdown
Member

I just rebuilt the copy of the book on the deployed site. Some cases are improved, but there are a few that aren't. E.g., look at https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git#TFS. The problem is that the incoming HTML looks like:

<code class="p">&amp;</code><code class="n">gt</code><code class="p">;</code>

There are several cases that look like that, so we can probably just match that whole nasty sequence.

@peff

peff commented May 6, 2016

Copy link
Copy Markdown
Member

Hmm, looks like there are more cases than that (e.g., on that same page are some where the middle code block has no class). I'll see if I can come up with a more inclusive regex.

@jnavila

jnavila commented May 6, 2016

Copy link
Copy Markdown
Contributor Author

Argh... I failed on this. The html generation is utterly broken.

@peff

peff commented May 6, 2016

Copy link
Copy Markdown
Member

I think I have something close to working with:

&amp;</code>(<code class=".">)?([gl]t)(</code>)?<code class="[^p]">;

as my regex.

@peff

peff commented May 6, 2016

Copy link
Copy Markdown
Member

I just pushed up 6994929 and bf95f7f, and they're currently deployed on the site (and the book rebuilt with them). The results look pretty good to me. Any comments?

@jnavila

jnavila commented May 6, 2016 •

Copy link
Copy Markdown
Contributor Author

a simple

> grep -E '&amp;<' *.html | wc -l
95

whereas

> grep -E '&amp;</code>(<code class=".">)?([gl]t)(</code>)?<code class="[^p]">;' *.html | wc -l
1

Something is missing in the regex.

We also have wrongly escaped '&', for instance in 1-git-server.html:

1-git-server.html:<code class="gp">$</code><code class="nb"> </code>mkdir .ssh <code class="p">&amp;</code>amp<code class="p">;&amp;</code>amp<code class="p">;</code> chmod <code class="m">700</code> .ssh

@peff

peff commented May 6, 2016

Copy link
Copy Markdown
Member

Oops, sorry, the regex I shared is where I was trying to debug the alternate classes. The final [^p] should be just . (all of the instances but one are p, and I wanted to see what the non-p one was).

I'll see if I can expand it to handle &amp.

@peff

peff commented May 6, 2016

Copy link
Copy Markdown
Member

OK, with 893725d this now looks good to me.

(That's all built on your commit here, and I'll merge it all in as one).

@jnavila

jnavila commented May 6, 2016

Copy link
Copy Markdown
Contributor Author

OK. Thanks.

@peff
peff merged commit 55f679f into git:master May 6, 2016
peff added a commit that referenced this pull request May 6, 2016
This pulls in #755, along with my fixups on top.

* fixup-pr-755:
  genbook2: match broken '&amp;amp;' escaping
  genbook2: find &amp;gt; inside split <code> blocks
  genbook2: simplify html fixup
  postprocess book v2 for entities in code blocks
@jnavila
jnavila deleted the fix_entities branch February 17, 2017 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Code blocks rendering html entities

2 participants