You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Now that signed-rulesets is merged, I've turned my focus on experimenting with incremental ruleset updates. jsondiffpatch seems like the most useful library here, but would require us to change the default.rulesets JSON rules from an array to a hash. This changes the rulesets from 1.3MB compressed to 1.4, roughly. The incremental changes would be absolutely tiny compared to the full delivery of rulesets every time, so I think it's worth it.
I'm still not sure how I would ensure everyone has rulesets applied in the correct order and is synced with the latest set, plus know where to find all the incremental updates without being referred to a large json manifest on every ping. Perhaps having a major and minor revision, each major revision is a full ruleset publication (that is applied every once in a while, or can be just derived from the rulesets delivered with the extension itself) and the minor revisions can be just a set of incremental files, like https://www.https-rulesets.org/v3/some-predictable-url-1, https://www.https-rulesets.org/v3/some-predictable-url-2, then the next release could be https://www.https-rulesets.org/v4/some-predictable-url-1, etc. Thoughts?
For the "large json manifest" approach, I argued at #14907 (comment) that the bandwidth increase would be trivial, on the order of maybe 20GB/month. This seems even more trivial to me with your comment at #15090 (comment) where you mention that shortly after launching the new feature, the CDN served 4TB in two days.
You did not respond to my comment so I'm unsure if you disagree, didn't get around to responding, etc. Whether we use a JSON instruction set approach or not, it might help if you could quantify your concerns about bandwidth/file sizes.
@jeremyn I agree the bandwidth overhead would be minimal for us on the server side. Over time, such a manifest may grow and become less than minimal for clients.
Also keep in mind that if we try to minimize the amount of bandwidth required for pings, we can potentially increase the frequency of these pings. Though 10k may seem like a minimal bandwidth requirement when compared with the rulesets, it is quite large when compared to a 1k update.
Additionally, I don't see any benefit in increasing the amount of bandwidth required to be above the absolute minimum needed to notify clients when they require an update. The timestamp serves this purpose, in addition to pointing directly to the files needed to download new rulesets currently. In the future, perhaps a manifest can be added as a follow-up request, after the client knows that some update has occurred. But for now, it seems unnecessary.
Activity
jeremyn commentedon Apr 7, 2018
For the "large json manifest" approach, I argued at #14907 (comment) that the bandwidth increase would be trivial, on the order of maybe 20GB/month. This seems even more trivial to me with your comment at #15090 (comment) where you mention that shortly after launching the new feature, the CDN served 4TB in two days.
You did not respond to my comment so I'm unsure if you disagree, didn't get around to responding, etc. Whether we use a JSON instruction set approach or not, it might help if you could quantify your concerns about bandwidth/file sizes.
Hainish commentedon Apr 11, 2018
@jeremyn I agree the bandwidth overhead would be minimal for us on the server side. Over time, such a manifest may grow and become less than minimal for clients.
Also keep in mind that if we try to minimize the amount of bandwidth required for pings, we can potentially increase the frequency of these pings. Though 10k may seem like a minimal bandwidth requirement when compared with the rulesets, it is quite large when compared to a 1k update.
Additionally, I don't see any benefit in increasing the amount of bandwidth required to be above the absolute minimum needed to notify clients when they require an update. The timestamp serves this purpose, in addition to pointing directly to the files needed to download new rulesets currently. In the future, perhaps a manifest can be added as a follow-up request, after the client knows that some update has occurred. But for now, it seems unnecessary.
pipboy96 commentedon Mar 29, 2019
@Hainish I don't see anyone working on this currenty (I checked your and @zoracon's fork). Can I start working on this in a fork?