You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
I was wondering if it'd be possible to create a separate repository to contain only the rulesets?
With a tiny bit of modification to the setup scripts that already exist, it would be fairly trivial to include that repository as a sub-repository exactly where it already is in https-everywhere.
The reason we might want to do this is to make the rules more available to other applications that might want to use them. In fact, I'm hoping to do exactly that in a project I'm working on.
The problem with splitting out into a different repository is that we'd lose all ruleset history. Although maybe we could clone the whole repository history. A less invasive approach would be to just move the rules directory within HTTPS Everywhere to the top level, which makes logical sense since both Chrome and Firefox use the rules, and would save some typing.
I've considered the latter a bunch of times, and would like to do one or the other. But my experience is that seemingly small changes like this generate surprisingly disproportionate maintenance overhead, as unexpected things break. So I'd like to postpone either such move until after crunch time for Let's Encrypt is over and I can dedicate more time to fixing issues.
For the project you're working on, can you embed the https everywhere repo as a whole? The JS files shouldn't add too much size relative to the rules.
I ended up writing a simple bash script to clone https-everywhere, run the make script that compiles the rulesets into a single XML file and performs verification etc., and then just removes the unneeded parts of the rest of the codebase. It's far from elegant, but it works.
Your concerns are very valid, but I think I can suggest a method that might sit well with you and work out seamlessly.
Create a new repository, just call it rules
On your own filesystem, copy all of the contents of the https-everywhere directory to rules/, including the .git/ directory and related files/directories.
In rules/, directly modify the .git/[...] files to change the repository settings (e.g. name) appropriately
Push rules/ to origin master (github.com/EFForg/rules) - You now have the entire HTTPS Everywhere history copied
Move the contents of rules/src/chrome/content/rules to the top-level of rules/
git rm -rf everything but the ruleset files and commit + push. rules/ now contains only rulesets.
In https-everywhere, git rm -r src/chrome/content/rules
Make EFForg/rules a sub-repository of EFForg/https-everywhere in src/chrome/content
With all this, you should now retain the exact same structure in https-everywhere as before, and would only have to modify build scripts ever so slightly so that they perform the appropriate sub-repository management to handle the rules sub-repo.
A lot of this is speculative and I'm not sure it's guaranteed to work. I'll try the idea out in a proof-of-concept setting sometime and report back. It's certainly not completely effortless, but it should be fairly straightforward and achieve the desired effect. I understand not being able to do it immediately.
While I'm generally in favor of this one concern I have is that it makes the contributor experience worse, because you have to grapple with two repos instead of one. How is a ruleset contributor supposed to get work done? Clone the main repo and work in the submodule? How do they send PRs? (I know it's technically possible but it's beyond many people's git abilities since the mental model is too complex.) Clone just the rules repo? How do they test? Etc.
One potential compromise might be to provide a read-only (i.e. no PRs) repo that provided just the rules. It'd be built using a (modified version of) the filter-branch process on a cronjob. This would still provide a repo with just the rules, and even provide the benefit of ruleset issues being separate from codebase issues. But that in turn would cause some awkwardness with people being expected to send issues to one repo and PRs to another.
Or, third idea: maybe we could eliminate the requirement to have the source checked out in order to test rulesets? I.e. the ruleset contributor experience would be: install the extension from AMO, clone the ruleset repo, maybe run a build step (though hopefully we could eliminate this step too), and point the production extension at the local clone.
As I mentioned on #10320, splitting these two into separate repositories would be helpful in improving the speed at which the actual extension core is developed. It also fosters external implementations of the rewrite engine.
From what I've read, the main issue it the tight coupling between the core and the rulesets due to tests. Why is this?
Looking over this. It seems like the main priority would be to ensure the process is smooth for the ruleset maintainers.
So we should probably move the extension source to another repo, and leave the rulesets here. This way the github issue & PR history and git history is preserved and easily accessible for rs maintainers.
We should also provide a way for rulesets to be imported directly into the extension. This way rs maintainers would not need to mess with the development version of the HE extension.
Moving forward, I think we should start to develop an import function for maintainers to use. This could be built on top of @Hainish work in #12863
If that goes well, we could start asking rs maintainers to use the feature. If we are able ween them off of manually rebuilding the extension this way, then we can start to think about moving the extension source into another directory.
Activity
jsha commentedon Aug 27, 2015
The problem with splitting out into a different repository is that we'd lose all ruleset history. Although maybe we could clone the whole repository history. A less invasive approach would be to just move the rules directory within HTTPS Everywhere to the top level, which makes logical sense since both Chrome and Firefox use the rules, and would save some typing.
I've considered the latter a bunch of times, and would like to do one or the other. But my experience is that seemingly small changes like this generate surprisingly disproportionate maintenance overhead, as unexpected things break. So I'd like to postpone either such move until after crunch time for Let's Encrypt is over and I can dedicate more time to fixing issues.
For the project you're working on, can you embed the https everywhere repo as a whole? The JS files shouldn't add too much size relative to the rules.
Thanks,
Jacob
chaoticsmol commentedon Sep 4, 2015
I ended up writing a simple bash script to clone https-everywhere, run the make script that compiles the rulesets into a single XML file and performs verification etc., and then just removes the unneeded parts of the rest of the codebase. It's far from elegant, but it works.
Your concerns are very valid, but I think I can suggest a method that might sit well with you and work out seamlessly.
rulesrules/, including the.git/directory and related files/directories.rules/, directly modify the.git/[...]files to change the repository settings (e.g. name) appropriatelyrules/toorigin master(github.com/EFForg/rules) - You now have the entire HTTPS Everywhere history copiedrules/src/chrome/content/rulesto the top-level ofrules/git rm -rfeverything but the ruleset files and commit + push.rules/now contains only rulesets.git rm -r src/chrome/content/rulesEFForg/rulesa sub-repository ofEFForg/https-everywhereinsrc/chrome/contentWith all this, you should now retain the exact same structure in https-everywhere as before, and would only have to modify build scripts ever so slightly so that they perform the appropriate sub-repository management to handle the rules sub-repo.
A lot of this is speculative and I'm not sure it's guaranteed to work. I'll try the idea out in a proof-of-concept setting sometime and report back. It's certainly not completely effortless, but it should be fairly straightforward and achieve the desired effect. I understand not being able to do it immediately.
strugee commentedon May 22, 2017
@jsha just FYI it's pretty easy ("easy") to split the rulesets out to a different repository while still maintaining a clean history. Similar to what @zsck said, but with a bit less sketchiness: https://help.github.com/articles/splitting-a-subfolder-out-into-a-new-repository/
While I'm generally in favor of this one concern I have is that it makes the contributor experience worse, because you have to grapple with two repos instead of one. How is a ruleset contributor supposed to get work done? Clone the main repo and work in the submodule? How do they send PRs? (I know it's technically possible but it's beyond many people's git abilities since the mental model is too complex.) Clone just the rules repo? How do they test? Etc.
One potential compromise might be to provide a read-only (i.e. no PRs) repo that provided just the rules. It'd be built using a (modified version of) the filter-branch process on a cronjob. This would still provide a repo with just the rules, and even provide the benefit of ruleset issues being separate from codebase issues. But that in turn would cause some awkwardness with people being expected to send issues to one repo and PRs to another.
Or, third idea: maybe we could eliminate the requirement to have the source checked out in order to test rulesets? I.e. the ruleset contributor experience would be: install the extension from AMO, clone the ruleset repo, maybe run a build step (though hopefully we could eliminate this step too), and point the production extension at the local clone.
migueldemoura commentedon Jun 20, 2017
As I mentioned on #10320, splitting these two into separate repositories would be helpful in improving the speed at which the actual extension core is developed. It also fosters external implementations of the rewrite engine.
From what I've read, the main issue it the tight coupling between the core and the rulesets due to tests. Why is this?
cowlicks commentedon Oct 18, 2017
Looking over this. It seems like the main priority would be to ensure the process is smooth for the ruleset maintainers.
So we should probably move the extension source to another repo, and leave the rulesets here. This way the github issue & PR history and git history is preserved and easily accessible for rs maintainers.
We should also provide a way for rulesets to be imported directly into the extension. This way rs maintainers would not need to mess with the development version of the HE extension.
Moving forward, I think we should start to develop an import function for maintainers to use. This could be built on top of @Hainish work in #12863
If that goes well, we could start asking rs maintainers to use the feature. If we are able ween them off of manually rebuilding the extension this way, then we can start to think about moving the extension source into another directory.
pipboy96 commentedon Aug 10, 2019
This is planned for 2020.