Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Use CodeMirror in ruleset debugger (#11794) - #13560

Merged
Hainish merged 5 commits into
EFForg:masterfrom
jeremyn:jeremyn-add-codemirror-to-ruleset-debugger
Nov 16, 2017
Merged

Hainish merged 5 commits into
EFForg:masterfrom
jeremyn:jeremyn-add-codemirror-to-ruleset-debugger

Conversation

@jeremyn

@jeremyn jeremyn commented Nov 11, 2017

Copy link
Copy Markdown
Contributor

@Hainish @cowlicks This should resolve issue #11794 to add CodeMirror to the ruleset debugger.

The commit message describes how I uglified (minified) the CodeMirror files. For security purposes I encourage someone at the EFF to verify these minifications.

Also, as the commit message says, this almost certainly requires another update to satisfy CodeMirror's MIT license. In #13062 (comment) I discuss some concerns I have about HTTPS Everywhere's approach to licensing, so for the time being I will let someone at the EFF figure out how to handle licensing here. Please feel free to push any needed changes directly to my branch.

@jeremyn

jeremyn commented Nov 11, 2017

Copy link
Copy Markdown
Contributor Author

It doesn't look like we use npm in the build process, for example we don't require it in install-dev-dependencies.sh. Could you be more specific?

@ghost

ghost commented Nov 11, 2017

Copy link
Copy Markdown

@jeremyn You used uglify-es, not uglifyjs, right?

@ghost

ghost commented Nov 11, 2017 •

Copy link
Copy Markdown

I cannot reproduce the codemirror-5.31.0.min.js file. My file is 114 bytes bigger.

@ghost

ghost commented Nov 11, 2017

Copy link
Copy Markdown

I can reproduce a CSS file but I had to delete an empty line at the end.

@jeremyn

jeremyn commented Nov 11, 2017

Copy link
Copy Markdown
Contributor Author

I used https://www.npmjs.com/package/uglifyjs as mentioned in the git commt message.

@ghost

ghost commented Nov 11, 2017

Copy link
Copy Markdown

@jeremyn

jeremyn commented Nov 11, 2017

Copy link
Copy Markdown
Contributor Author

I'm not sure what the relationship is between https://www.npmjs.com/package/uglifyjs and https://www.npmjs.com/package/uglify-js -- I see they share a collaborator -- but https://www.npmjs.com/package/uglify-js has many more downloads so I agree that's the one to use.

I've (force-)pushed an update that uses uglify-js@3.1.9 for the two JS files. I also removed an extra blank line from the middle of debugging-rulesets.js.

@jeremyn

jeremyn commented Nov 11, 2017

Copy link
Copy Markdown
Contributor Author

@koops76 I'm not clear from your comments whether you could reproduce the CSS minification. Try the following:

$ md5sum codemirror.css  # should be 4b107dce67663f59927a270fa63f3832
$ npm install uglifycss@0.0.27
$ node_modules/uglifycss/uglifycss codemirror.css > codemirror.min.css
$ md5sum codemirror.min.css  # should be 4169bd1daef71f34419197aeaa4b32b1

Do you get the same hashes?

@ghost

ghost commented Nov 11, 2017

Copy link
Copy Markdown

@jeremyn Use sha256sum.

@ghost

ghost commented Nov 11, 2017

Copy link
Copy Markdown

I reproduced everything.

@Hainish

Hainish commented Nov 13, 2017

Copy link
Copy Markdown
Member

I do think if we're adding dependencies (which I'm not necessarily against, as the sign-rulesets branch also adds dependencies and so there are multiple reasons why we're moving in that direction) we should include some kind of JS package management.

We already have npm as a requirement for many of the utilities we use. Although npm isn't exclusively tasked with front-end dependency graphs, it is nonetheless useful for this purpose. In combination with grunt or some other task-runner, we could include steps which minify the resources and place them properly in the extension path.

@koops76 I see you deleted comments that suggested similar. I'm curious why?

@jeremyn

jeremyn commented Nov 13, 2017

Copy link
Copy Markdown
Contributor Author

@Hainish I agree that listing a dependency in a package.json or equivalent has some advantages compared with checking the dependency into the repository. On the other hand, adding a new download source introduces another point of failure during builds, including test builds.

However assuming we prefer the package.json approach, is that available at all in the current build process? That was my question to @koops76. I don't think it is.

By the way, @koops76 recently opened an issue (#13512) to discuss remaking the build system.

@Hainish

Hainish commented Nov 13, 2017

Copy link
Copy Markdown
Member

@jeremyn my typical workflow would be to create a package.json with any dependencies, using that to do version tracking, and of course putting node_modules into .gitignore. I'd then create a grunt task to minify and copy the minified JavaScript or CSS to the extension paths, but actually check those files in. This way, we could periodically update those dependencies but wouldn't have to deal with update failures at build time. We also wouldn't need to add npm as a hard build dependency in that case, just as a dev dependency for those that are updating resources.

@jeremyn

jeremyn commented Nov 14, 2017

Copy link
Copy Markdown
Contributor Author

I think it's too early to create a project-wide installation architecture, just to make CodeMirror available to the ruleset debugger. It raises questions like, where should the package file go? How should we document the installation process? Also for security we should include checksums somewhere, and I don't think npm shrinkwrap supports that, so we'd need to use something like https://github.com/mozilla/npm-lockdown or maybe Yarn. We should wait for one or two more dependency/feature use cases before trying to figure these questions out.

Maybe you're concerned that the only documentation about how to create the minified CodeMirror files is in the git commit message, which isn't great. I can add a small README.md to chromium/codemirror/ if you want.

@ghost

ghost commented Nov 14, 2017

Copy link
Copy Markdown

@jeremyn NPM is better than Yarn security-wise. It creates a package-lock.json file with SHA512 sums instead of SHA1 sums Yarn currently uses.

@ghost

ghost commented Nov 14, 2017

Copy link
Copy Markdown

Maybe you're concerned that the only documentation about how to create the minified CodeMirror files is in the git commit message, which isn't great. I can add a small README.md to chromium/codemirror/ if you want.

👍

@jeremyn

jeremyn commented Nov 14, 2017

Copy link
Copy Markdown
Contributor Author

@koops76 According to https://docs.npmjs.com/files/package-lock.json#integrity : "For registry sources, this is the integrity that the registry provided, or if one wasn't provided the SHA1 in shasum." uglifycss has a SHA1 hash, for example you see this in package-lock.json after installing the package:

  "dependencies": {
    "uglifycss": {
      "version": "0.0.27",
      "resolved": "https://registry.npmjs.org/uglifycss/-/uglifycss-0.0.27.tgz",
      "integrity": "sha1-U1U7gRXeJtzle0K6CzTojdgNDN4="
    }
  }

If you change the sha1 value, for example from sha1-U1U7... to sha1-11U7... it will fail as expected. However if you change the integrity field to try to use a SHA512 hash, for example to sha512-1111... it will unexpectedly work. (The correct SHA512 hash for https://registry.npmjs.org/uglifycss/-/uglifycss-0.0.27.tgz is 0hA7g3cXvHdhetBkc2bGbA+Mu/C23jncs6kkmZLsTJCRloT72InV5XZs08t+m8AcFqBAsms+ZRK7cuY+eAXI6w== which you can find with openssl sha512 -binary uglifycss-0.0.27.tgz | base64.)

This is not all to say "you're wrong", but instead just that npm integrity checking looks complicated. I'm also concerned that it silently continues to work even with a bad hash. I don't know how Yarn would compare in similar tests.

@jeremyn

jeremyn commented Nov 14, 2017

Copy link
Copy Markdown
Contributor Author

And also, it would be nice if we could just merge this PR without having to first dig deep into comparing the integrity mechanisms of npm vs Yarn vs anything else.

@ghost

ghost commented Nov 14, 2017

Copy link
Copy Markdown

@jeremyn This is because uglifycss is too old and doesn't have a SHA512 hash yet.

@Hainish

Hainish commented Nov 14, 2017

Copy link
Copy Markdown
Member

@jeremyn I think it's sensible to merge in these dependencies without creating a dependency management system in general. I haven't had a chance to review this yet (I'm in meetings all day) but after just a cursory look, can I suggest we put any external deps in a separate path, say chromium/external/ and have codemirror included under that? Just so we're clear that this is an external library, and that way we could also perhaps add a licensing note under that path explaining how these libraries are covered under their respective licenses (or add that note, mentioning the path, under the top-level LICENSE.txt file).

@Hainish

Hainish commented Nov 14, 2017

Copy link
Copy Markdown
Member

I think it's sensible to merge in these dependencies without creating a dependency management system in general.

I mean this is sensible for now, given the relatively confined scope of the usage of this library. I do think it will be necessary to have a dependency management system in the future, as we build out the extension. I also think if we're not to go full-blown dependency system right now, having this under a chromium/external/ path (with the steps to recreate included) will ensure that when we do put a dependency management system in place, this won't fall through the cracks.

@ghost

ghost commented Nov 14, 2017

Copy link
Copy Markdown

@Hainish I think the directory should be called vendor.

EFF staff should review the CodeMirror license (MIT) at

https://raw.githubusercontent.com/codemirror/CodeMirror/5.31.0/LICENSE

and make any necessary updates to comply with the license.
@jeremyn

jeremyn commented Nov 14, 2017

Copy link
Copy Markdown
Contributor Author

I moved the minified files from codemirror/ to external/ and added an external/README.md. I also rebuilt the three minified files, because my original approach used redirection instead of the --output option to create the output files and apparently redirection introduces a newline at the end. I think the --output option makes a better build process.

I used a force push instead of a regular push. I'm sorry for the trouble. This should keep unnecessary versions of the minified files out of the repository.

I prefer external to vendor. external is less obscure, and it avoids odd naming situations in the future such as if HTTPS Everywhere uses a dependency from another EFF project.

@Hainish For licensing, as I describe in issue #13062 (comment) (still waiting for a response there), I have concerns about the overall licensing of this project. That being the case, you/the EFF can handle the CodeMirror licensing. In fact I don't want your licensing changes to be done under my name, so I've unchecked the "Allow edits from maintainers" option to make sure you don't push licensing changes and then accidentally squash-and-merge them. (I know you would not do that intentionally if I asked you not to, but you might forget if this issue drags on.) So that will need to be done in a separate PR.

@Hainish

Hainish commented Nov 15, 2017

Copy link
Copy Markdown
Member

@jeremyn thanks for those changes. As I mentioned I'm in meetings all day so I'll have to wait until at least tomorrow to review, but I definitely plan to have this merged by next release (which I'm thinking should be early next week.)

@Hainish Hainish left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. I've reproduced the codemirror js and css and get the same results. A few final changes and then I'm ready to merge.

## CodeMirror 5.31.0

```
$ npm install uglify-js@3.1.9

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is not a package dependency and is intended to be run as a command line app, let's add a -g flag for global install as the installation instructions specify: https://github.com/mishoo/UglifyJS2#install

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the drawback of defining all the dependencies in package.json in desired versions and define npm scripts that would execute proper commands?

Global packages should be avoided when possible. Someone may skip those steps as they've already performed them before and not notice they've since installed different versions since some other project required different ones. Using local packages gets rid of that problem.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, but the above command will cause uglify-js to be added to the dependencies hash, rather than the devDependencies hash. Thus we should preference running this with the -D flag, and the below reproducibility instructions should be modified to be:
../node_modules/uglify-js/bin/uglifyjs

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason to specify those packages directly instead of just having package.json checked into the repo and running npm install?

BTW, if you're referring to the binary by path, it's usually better to use ../node_modules/.bin/uglifyjs as that depends less on the internal package structure. node_modules/.bin contains links to binaries exported by top-level packages.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mgol creating a dependency management system is on the table, but I don't want to block merging until we have one in place. See #13560 (comment) and resulting discussion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These instructions are supposed to give someone who is knowledgeable about npm the key information needed to reproduce the files: package names, version numbers, options used. It's not supposed to completely walk a beginner through the process of making these files. The user can decide for themselves whether to install the minifier executables globally or not and how to get the files into whatever directories they want.


```
$ npm install uglify-js@3.1.9
$ npm install uglifycss@0.0.27

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread chromium/external/README.md Outdated

```bash
uglifycss \
--output codemirror-5.31.0.min.css

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a trailing \ here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread chromium/.eslintignore
@@ -0,0 +1 @@
external/*

@Hainish Hainish Nov 15, 2017 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to add chromium/.eslintignore to the top-level .build_exclusions as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread chromium/debugging-rulesets.html Outdated
<link href="debugging-rulesets.css" rel="stylesheet">
<script src="external/codemirror-5.31.0.min.js"></script>
<link href="external/codemirror-5.31.0.min.css" rel="stylesheet">
<script src="external/codemirror-5.31.0.xml.min.js"></script>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may have not been specific enough in my suggestion. Can we move each of these under external/codemirror/, just so each dependency moving forward is in its own neat path?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

--compress \
--mangle \
--comments '/copyright|license/' \
--output codemirror-5.31.0.min.js \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relating to the suggestion above of adding each external dependency to its own unique path, we'll want to change these output paths to external/codemirror/...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Hainish

Hainish commented Nov 15, 2017

Copy link
Copy Markdown
Member

Thinking it over, I don't really see the use for minification here. In web development, it's useful to minify in order to reduce the size of the downloaded resources. Here, we're already compressing the extension at build time, since xpi and crx files are effectively zips. And while there may be a memory overhead win, I can't imagine it would be very much.

If there isn't a definitive benefit to minification here, we could remove the necessity for uglify-js and uglifycss.

@ghost

ghost commented Nov 15, 2017

Copy link
Copy Markdown

@Hainish It's a common courtesy to minify any distributed files to save some bits, I think the extension's code should also be minified.

@Hainish

Hainish commented Nov 15, 2017

Copy link
Copy Markdown
Member

I do see some savings in download size between the minified

-rw-r--r-- 1 user user 1972105 Nov 15 14:51 pkg/https-everywhere-2017.10.30~pre-amo.xpi

and non-minified:

-rw-r--r-- 1 user user 2019772 Nov 15 14:54 pkg/https-everywhere-2017.10.30~pre-amo.xpi

so it appears that minification does save us some space.

@ghost

ghost commented Nov 15, 2017

Copy link
Copy Markdown

@jeremyn We will add a minification step later, you may completely remove minification for now.

@Hainish

Hainish commented Nov 15, 2017

Copy link
Copy Markdown
Member

@koops76 let's leave minification in if it's a win on download size.

@ghost

ghost commented Nov 15, 2017

Copy link
Copy Markdown

@Hainish I mean minification for the whole extension.

@jeremyn

jeremyn commented Nov 15, 2017

Copy link
Copy Markdown
Contributor Author

I kept the minified files. Someone who wants to see the originals can refer to the new README.md for instructions on where to get them. If they need runtime debugging with the originals, they can rebuild the extension for themselves with those files.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants