Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Overhaul repo rule storage #3999

Closed
Closed

Description

@semenko

Our ruleset storage is chaotic. Let's ponder overhauling it. (cc @fuglede)

Activity

  1. jsha commented on Jan 27, 2016

    @jsha
    Member

    I agree! Do you mean storage in the repo, or the XML format, or the SQLite format, or ...? :-)

    I was thinking recently that it's probably time to move the rulesets to a new repo that gets included into the main one. Submodules would be one way, but I would also like to have the property that if you check out latest https-everywhere and build, you get the latest rulesets, without needed to constantly update the submodule head in the code repo. If we do this, I think it makes sense to start the new repo as a fork of https-everywhere, deleting everything but the XML files, and moving the rulesets higher in the tree. That way we retain the commit history for the rulesets, which is important. We would also need to make sure that the ruleset tests get run in the ruleset repo in addition to the main repo.

    It may also be time to split up the rulesets directory by first letter.

  2. semenko commented on Jan 27, 2016

    @semenko
    ContributorAuthor

    I was pondering ... all of the above! But primarily the repo storage / XML format.

    I think we should:

    • Move away from XML
    • Preserve separate files per-rule (so git merges don't become insane)
    • Develop a simpler way for people add "trivial rule" sites
    • Preserve a history of "broken" sites, rather than a complex git log

    I've been pondering some YAML style rules, or perhaps simple, empty files for non-complex rules.

    What if someone could touch somedomain.tld && git add somedomain.tld and have their rule added as a simple rule?

  3. changed the title [-]Overhaul rule storage[/-] [+]Overhaul repo rule storage[/+] on Jan 27, 2016
  4. jsha commented on Jan 28, 2016

    @jsha
    Member

    Move away from XML

    I disagree on this one. I've thought about it a number of times, but I think there's nothing fundamental about the XML format that makes it unsuitable, and programmatically reformatting all our XML into YAML or JSON would lose all history. I am not a huge fan of XML, but it's not bad enough to kill.

    That said, I'm totally into improving our build-time transforms of the XML to improve stored size, in-memory size, and speed of access.

    Develop a simpler way for people add "trivial rule" sites

    Yes. Caveat: This will increase the total volume of submissions greatly. We need to do two things first: 1. Decide on a scope for ruleset inclusion in HTTPS Everywhere. Let's Encrypt has issued for ~450,000 unique domains. Even if we assume all those domains work perfectly, it's clearly not suitable to include them all in HTTPS Everywhere. It's just too much data. 2. Improve our fetch testing so that it runs at the time a PR is submitted, and checks for mixed content both on the root page of a domain and on a small set of pages crawled from the root.

    Preserve a history of "broken" sites, rather than a complex git log

    I'm not sure what you mean here.

  5. semenko commented on Jan 28, 2016

    @semenko
    ContributorAuthor

    Preserve a history of "broken" sites, rather than a complex git log

    I'm not sure what you mean here.

    e.g. #3983 -- we sometimes git rm very broken rules. Since rule .xml names aren't always equal to the domain, the history can be hard to find.

  6. DerekTurtleRoe commented on Feb 24, 2016

    @DerekTurtleRoe

    @semenko @jsha Has any work been done on this? I was just curious. This seems cool.

  7. jsha commented on Mar 13, 2016

    @jsha
    Member

    Only one small part of it: Now we ship the extension with a .json file containing XML rulesets, instead of a .sqlite file containing XML rulesets.

  8. DerekTurtleRoe commented on Mar 14, 2016

    @DerekTurtleRoe

    @jsha That is a step in the right direction, I would say. 😄

  9. pipboy96 commented on Jun 5, 2019

    @pipboy96
    Contributor

    We already moved to JSON.

  10. J0WI commented on Jun 5, 2019

    @J0WI
    Contributor

    The issue is more about the format used in the repo where we still use XML.

  11. reopened this on Jun 5, 2019
  12. pipboy96 commented on Jun 5, 2019

    @pipboy96
    Contributor

    @J0WI we discussed it in #12620. Seems that we decided to not adopt it for now.

  13. J0WI commented on Jun 7, 2019

    @J0WI
    Contributor

    This issue discusses more points than just JSON/YML formats: #3999 (comment)
    I'd rather close #12620.

  14. pipboy96 commented on Jun 7, 2019

    @pipboy96
    Contributor

    @J0WI Should we restart this discussion?

  15. J0WI commented on Jun 10, 2019

    @J0WI
    Contributor

    This could be mentioned in #18069 but I'm currently not aware of any new input to solve this.

  16. pipboy96 commented on Jun 11, 2019

    @pipboy96
    Contributor

    @J0WI I also have not many ideas, but I am still thinking on how this may be done.

  17. zoracon commented on Jan 9, 2020

    @zoracon
    Contributor

    Closing for now, But tagging in #17268 to keep this in mind. And also, we now have a Rust library for fetching rulesets. Not quite the same topic of discussion, but it's related.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions