Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Deliver incremental ruleset updates #15096

Closed

Description

@Hainish

Type: other

Now that signed-rulesets is merged, I've turned my focus on experimenting with incremental ruleset updates. jsondiffpatch seems like the most useful library here, but would require us to change the default.rulesets JSON rules from an array to a hash. This changes the rulesets from 1.3MB compressed to 1.4, roughly. The incremental changes would be absolutely tiny compared to the full delivery of rulesets every time, so I think it's worth it.

I'm still not sure how I would ensure everyone has rulesets applied in the correct order and is synced with the latest set, plus know where to find all the incremental updates without being referred to a large json manifest on every ping. Perhaps having a major and minor revision, each major revision is a full ruleset publication (that is applied every once in a while, or can be just derived from the rulesets delivered with the extension itself) and the minor revisions can be just a set of incremental files, like https://www.https-rulesets.org/v3/some-predictable-url-1, https://www.https-rulesets.org/v3/some-predictable-url-2, then the next release could be https://www.https-rulesets.org/v4/some-predictable-url-1, etc. Thoughts?

Activity

  1. jeremyn commented on Apr 7, 2018

    @jeremyn
    Contributor

    For the "large json manifest" approach, I argued at #14907 (comment) that the bandwidth increase would be trivial, on the order of maybe 20GB/month. This seems even more trivial to me with your comment at #15090 (comment) where you mention that shortly after launching the new feature, the CDN served 4TB in two days.

    You did not respond to my comment so I'm unsure if you disagree, didn't get around to responding, etc. Whether we use a JSON instruction set approach or not, it might help if you could quantify your concerns about bandwidth/file sizes.

  2. Hainish commented on Apr 11, 2018

    @Hainish
    MemberAuthor

    @jeremyn I agree the bandwidth overhead would be minimal for us on the server side. Over time, such a manifest may grow and become less than minimal for clients.

    Also keep in mind that if we try to minimize the amount of bandwidth required for pings, we can potentially increase the frequency of these pings. Though 10k may seem like a minimal bandwidth requirement when compared with the rulesets, it is quite large when compared to a 1k update.

    Additionally, I don't see any benefit in increasing the amount of bandwidth required to be above the absolute minimum needed to notify clients when they require an update. The timestamp serves this purpose, in addition to pointing directly to the files needed to download new rulesets currently. In the future, perhaps a manifest can be added as a follow-up request, after the client knows that some update has occurred. But for now, it seems unnecessary.

  3. self-assigned this
    on Mar 29, 2019
  4. pipboy96 commented on Mar 29, 2019

    @pipboy96
    Contributor

    @Hainish I don't see anyone working on this currenty (I checked your and @zoracon's fork). Can I start working on this in a fork?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions