Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

rewriter should also rewrite hsts preload links #4396

Closed

Description

@Hainish

We've been removing rules from our rulesets for hosts that are included in the HSTS preload list. This is fine within the browser context, but it looks like the rewriter only sources from our rulesets and not the preload list. We should fix this.

Most users' browsers now include and use the preload list, so this is less important than rewriting for rules that aren't in the list. Nonetheless, it might be useful for users with older browsers or those that don't support the preload list.

Activity

  1. self-assigned this
    on Mar 17, 2016
  2. fuglede commented on Mar 20, 2016

    @fuglede
    Contributor

    Regarding the browser context, as you are probably already aware, it is worth noting that the various browsers currently have wildly different preload lists, so it may not be completely obvious where to draw the line: It takes a while before the entries from the chromium list makes it all the way to Firefox ESR (and currently, there seems to be a concensus that they are removed from HTTPS-E when they make it to ESR), but on the other hand, Mozilla is actively removing stale entries from the preload list, which Google does not do with chromium, so one is not a subset of the other. On the third hand, it can even take a while for submitted sites to make it to the preload list; currently, 2093 sites have been submitted but not included.

    Preloaded sites are submitted as HTTPS-E ruleset PRs every now and then; would it be worth it to decide on and document what can go into the ruleset database, just to have something to point new contributors to? (I know that I've been confused myself about the matter a couple of times already.)

  3. Hainish commented on Mar 25, 2016

    @Hainish
    MemberAuthor

    Thanks for the feedback, @fuglede. @jsha took the reins on this issue with #4462

  4. jeremyn commented on Aug 15, 2016

    @jeremyn
    Contributor

    In issue #3069, which is a list of specific sites that need rules, there was a brief discussion about whether sites that force HTTPS should be in that list. @rugk linked to this FAQ, "Why does HTTPS Everywhere include rules for sites like PayPal that already require HTTPS on all their pages?"

    In this issue it sounds like the policy is that we shouldn't make a ruleset for a site if that ruleset just duplicates what the HSTS preload list does. If so, should someone update this FAQ?

  5. rugk commented on Aug 15, 2016

    @rugk
    Contributor

    Well... the HSTS preload list is a special case. It is not mentioned in the FAQ.

  6. jeremyn commented on Aug 15, 2016

    @jeremyn
    Contributor

    The FAQ entry answers the question "Why do these rulesets exist?" If the HSTS preload list deprecates certain rules and rulesets, then the answer to the question in some cases is "Because we haven't gotten around to removing those rules yet." So, the FAQ entry should be clarified, especially if that entry is being used to justify creating rulesets that are possibly already deprecated by the HSTS preload list.

  7. rugk commented on Aug 15, 2016

    @rugk
    Contributor

    I am sure you can open a PR somewhere to improve the FAQ.

    However the HSTS preload list is really only an edge case. It is used by few websites and as you can see in this issue the sites may still be added to HTTPSE in certain cases. The FAQ just covers the "general case" where no HSTS preloading is involved.

  8. jeremyn commented on Aug 15, 2016

    @jeremyn
    Contributor

    I'm not entirely sure what the policy is or what this rewriter does so I'm not able to update the FAQ myself. If the EFF people agree a modification is needed and can explain the current policy to me in some more detail, I could come up with some wording if that would help.

    The HSTS preload list has well over ten thousand sites (see here, but warning: it might lag your browser), compared with HTTPS Everywhere's over twenty thousand rulesets, so I think it's more than an edge case. I agree it doesn't cover everything, in fact one big problem is that Let's Encrypt sites don't qualify because the certificates they provide aren't valid for long enough.

  9. rugk commented on Aug 15, 2016

    @rugk
    Contributor

    in fact one big problem is that Let's Encrypt sites don't qualify because the certificates they provide aren't valid for long enough.

    Ah, that's wrong. Certificate lifetime do not matter for HSTS preload inclusion. I speak from my own experience.

    The HSTS preload list has well over ten thousand sites (see here, but warning: it might lag your browser), compared with HTTPS Everywhere's over twenty thousand rulesets,

    Interesting numbers, but these are nothing compared to all other websites, or to the Alexa 1 Million. A recent scan by @ScottHelme showed that only circa 3% of the top 1 million sites use HSTS. That's 30.000, but keep in mind that this does not mean they are all preloaded. HSTS preloading just requires HSTS, so the number is way smaller.

  10. ScottHelme commented on Aug 15, 2016

    @ScottHelme

    Of the 29,908 sites that issue the HSTS header in the Alexa Top 1 Million, only 4,333 include the preload directive. Of those that do issue preload in their policy, the majority aren't even suitable for preloading and probably got the config as a copy/paste from the internet: https://scotthelme.co.uk/death-by-copy-paste/

  11. ScottHelme commented on Aug 15, 2016

    @ScottHelme

    FYI the Chromium HSTS Preload pending list is now located here: https://hstspreload.appspot.com/api/v2/pending

  12. rugk commented on Aug 15, 2016

    @rugk
    Contributor

    FYI the Chromium HSTS Preload pending list is now located here: https://hstspreload.appspot.com/api/v2/pending

    This loads way faster. 😃

  13. jeremyn commented on Aug 15, 2016

    @jeremyn
    Contributor

    @rugk I looked at the preload eligibility requirements here and you're right and I was wrong about a minimum certificate lifetime requirement. It's only the HSTS header that requires a minimum duration, but that doesn't necessarily have anything to do with the certificate lifetime. I misunderstood that, thanks for pointing it out.

    I want to make sure I'm not totally confused. Assume a simple site, with no subdomains, that redirects from http to https. Whether that site is in the HSTS preload list, or there's an HTTPS Everywhere rewrite rule, both of these cause the same user experience: the URL is changed from http to https before it leaves the user's browser. Is that right?

  14. rugk commented on Aug 15, 2016

    @rugk
    Contributor

    cause the same user experience: the URL is changed from http to https before it leaves the user's browser. Is that right?

    Yes.

  15. jeremyn commented on Aug 15, 2016

    @jeremyn
    Contributor

    @rugk Thanks.

  16. jsha commented on Aug 15, 2016

    @jsha
    Member

    Edit: Accidentally posted a partial comment; also I misread the original post so that partial comment was inaccurate. Removed. :-)

  17. reopened this on Aug 15, 2016
  18. fuglede commented on Aug 16, 2016

    @fuglede
    Contributor

    The overlap should not be very big; I ran some numbers for different reasons last year in https://lists.eff.org/pipermail/https-everywhere/2015-October/002639.html -- those only include the Chromium list and as mentioned above we want to take into account preload lists of all supported browsers, so the actual overlap is smaller. Moreover, afair the overlap was purged at some point between that mail and now.

  19. self-assigned this
    on May 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions