You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
I agree! Do you mean storage in the repo, or the XML format, or the SQLite format, or ...? :-)
I was thinking recently that it's probably time to move the rulesets to a new repo that gets included into the main one. Submodules would be one way, but I would also like to have the property that if you check out latest https-everywhere and build, you get the latest rulesets, without needed to constantly update the submodule head in the code repo. If we do this, I think it makes sense to start the new repo as a fork of https-everywhere, deleting everything but the XML files, and moving the rulesets higher in the tree. That way we retain the commit history for the rulesets, which is important. We would also need to make sure that the ruleset tests get run in the ruleset repo in addition to the main repo.
It may also be time to split up the rulesets directory by first letter.
I disagree on this one. I've thought about it a number of times, but I think there's nothing fundamental about the XML format that makes it unsuitable, and programmatically reformatting all our XML into YAML or JSON would lose all history. I am not a huge fan of XML, but it's not bad enough to kill.
That said, I'm totally into improving our build-time transforms of the XML to improve stored size, in-memory size, and speed of access.
Develop a simpler way for people add "trivial rule" sites
Yes. Caveat: This will increase the total volume of submissions greatly. We need to do two things first: 1. Decide on a scope for ruleset inclusion in HTTPS Everywhere. Let's Encrypt has issued for ~450,000 unique domains. Even if we assume all those domains work perfectly, it's clearly not suitable to include them all in HTTPS Everywhere. It's just too much data. 2. Improve our fetch testing so that it runs at the time a PR is submitted, and checks for mixed content both on the root page of a domain and on a small set of pages crawled from the root.
Preserve a history of "broken" sites, rather than a complex git log
Closing for now, But tagging in #17268 to keep this in mind. And also, we now have a Rust library for fetching rulesets. Not quite the same topic of discussion, but it's related.
Activity
jsha commentedon Jan 27, 2016
I agree! Do you mean storage in the repo, or the XML format, or the SQLite format, or ...? :-)
I was thinking recently that it's probably time to move the rulesets to a new repo that gets included into the main one. Submodules would be one way, but I would also like to have the property that if you check out latest https-everywhere and build, you get the latest rulesets, without needed to constantly update the submodule head in the code repo. If we do this, I think it makes sense to start the new repo as a fork of https-everywhere, deleting everything but the XML files, and moving the rulesets higher in the tree. That way we retain the commit history for the rulesets, which is important. We would also need to make sure that the ruleset tests get run in the ruleset repo in addition to the main repo.
It may also be time to split up the rulesets directory by first letter.
semenko commentedon Jan 27, 2016
I was pondering ... all of the above! But primarily the repo storage / XML format.
I think we should:
I've been pondering some YAML style rules, or perhaps simple, empty files for non-complex rules.
What if someone could
touch somedomain.tld && git add somedomain.tldand have their rule added as a simple rule?[-]Overhaul rule storage[/-][+]Overhaul repo rule storage[/+]jsha commentedon Jan 28, 2016
I disagree on this one. I've thought about it a number of times, but I think there's nothing fundamental about the XML format that makes it unsuitable, and programmatically reformatting all our XML into YAML or JSON would lose all history. I am not a huge fan of XML, but it's not bad enough to kill.
That said, I'm totally into improving our build-time transforms of the XML to improve stored size, in-memory size, and speed of access.
Yes. Caveat: This will increase the total volume of submissions greatly. We need to do two things first: 1. Decide on a scope for ruleset inclusion in HTTPS Everywhere. Let's Encrypt has issued for ~450,000 unique domains. Even if we assume all those domains work perfectly, it's clearly not suitable to include them all in HTTPS Everywhere. It's just too much data. 2. Improve our fetch testing so that it runs at the time a PR is submitted, and checks for mixed content both on the root page of a domain and on a small set of pages crawled from the root.
I'm not sure what you mean here.
semenko commentedon Jan 28, 2016
e.g. #3983 -- we sometimes
git rmvery broken rules. Since rule .xml names aren't always equal to the domain, the history can be hard to find.DerekTurtleRoe commentedon Feb 24, 2016
@semenko @jsha Has any work been done on this? I was just curious. This seems cool.
jsha commentedon Mar 13, 2016
Only one small part of it: Now we ship the extension with a .json file containing XML rulesets, instead of a .sqlite file containing XML rulesets.
DerekTurtleRoe commentedon Mar 14, 2016
@jsha That is a step in the right direction, I would say. 😄
pipboy96 commentedon Jun 5, 2019
We already moved to JSON.
J0WI commentedon Jun 5, 2019
The issue is more about the format used in the repo where we still use XML.
pipboy96 commentedon Jun 5, 2019
@J0WI we discussed it in #12620. Seems that we decided to not adopt it for now.
J0WI commentedon Jun 7, 2019
This issue discusses more points than just JSON/YML formats: #3999 (comment)
I'd rather close #12620.
pipboy96 commentedon Jun 7, 2019
@J0WI Should we restart this discussion?
J0WI commentedon Jun 10, 2019
This could be mentioned in #18069 but I'm currently not aware of any new input to solve this.
pipboy96 commentedon Jun 11, 2019
@J0WI I also have not many ideas, but I am still thinking on how this may be done.
zoracon commentedon Jan 9, 2020
Closing for now, But tagging in #17268 to keep this in mind. And also, we now have a Rust library for fetching rulesets. Not quite the same topic of discussion, but it's related.