Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Rejecting targets without a working HTTP 200/3XX URL #7662

Closed

Description

@jeremyn

I've started rejecting targets where we can't find even one working HTTP 200 URL, for example see #7636 (comment) . Often both HTTP and HTTPS return the same non-200 code, like 403. My suggestion is to note these URLs under a new error category No working URL known.

@fuglede @gloomy-ghost @Hainish @J0WI What do you all think?

EDIT: We should count HTTP 3XX URLs as well as 200. I've updated the issue description.

Activity

  1. jeremyn commented on Nov 21, 2016

    @jeremyn
    ContributorAuthor

    I noticed @gloomy-ghost 's comment in #7481 (comment) , where they wrote:

    I'm quite aggressive on adding targets. And I think that these kind of risks are still acceptable since they are easy to fix as well.

    Problems caused by adding internal URLs may not be easy to fix. Internal URLs that are used to manage corporate firewalls, communications, VPNs etc may depend on complex applications from a third party. If a sysadmin finds a problem with one of these sites, they are going to spend a bunch of time narrowing the problem down to HTTPS Everywhere and then just disable it. If it ends up wasting enough of the third party vendor's (Dell, Cisco etc) time, the vendor will simply advise their customers not to use HTTPS Everywhere. The odds that someone from the third party will contact us to work out the problem are very low.

    Another problem is that the "easy to fix" approach means we are willing to disable rules because of problems we can't test just because someone asks us to do that. That opens an easy attack path.

  2. gloomy-ghost commented on Nov 21, 2016

    @gloomy-ghost
    Collaborator
  3. jeremyn commented on Nov 21, 2016

    @jeremyn
    ContributorAuthor

    I'm not sure how to measure risk here. I think that it's more likely a corporate-type web application will have trouble than a site intended for the general public, and that it's less likely a corporate user will report a problem than a general public user.

  4. changed the title [-]Rejecting targets without a working HTTP 200 URL[/-] [+]Rejecting targets without a working HTTP 200/3XX URL[/+] on Nov 22, 2016
  5. jeremyn commented on Nov 22, 2016

    @jeremyn
    ContributorAuthor

    We should count HTTP 3XX URLs as well as 200. I've updated the issue description. I've updated the issue description and first comment.

  6. gloomy-ghost commented on Nov 22, 2016

    @gloomy-ghost
    Collaborator

    I suggest to add a subdomain if it has the subdomain in certificate (not a wildcard) even we can't find any tests.

  7. jeremyn commented on Nov 22, 2016

    @jeremyn
    ContributorAuthor

    A subdomain being in the certificate just means that the site might, at some point, enable HTTPS on that domain.

  8. Hainish commented on Nov 29, 2016

    @Hainish
    Member

    I agree with @jeremyn (and in fact our testing generally reflects this) that the absence of a 2XX (not just 200, since something like 207 can have useful content) or 3XX response, we should not accept the target.

    Generally the absence of a 2XX or 3XX code indicates problems with the webserver, is hard to test, and may introduce problems in maintenance later.

  9. jeremyn commented on Nov 29, 2016

    @jeremyn
    ContributorAuthor

    Edge case: one of the domains in pull request #6230 is https://invite.hoccer.com , which gets a 404. I think it's okay to include because if you Google search for site:invite.hoccer.com, there's an invite. URL that looks like a personalized link which gets a 200. I don't want to include a personalized link in our tests though, so there's no test URL.

    @Hainish Are the tests supposed to complain about 404s? Because they didn't here.

  10. gloomy-ghost commented on Nov 29, 2016

    @gloomy-ghost
    Collaborator
  11. Hainish commented on Nov 30, 2016

    @Hainish
    Member

    @gloomy-ghost when the test url response is 2XX but the rewritten test url isn't.

    It seems like if we can not find a 2XX/3XX response within the https site, this is okay as long as the following conditions are met:

    EITHER the http version of pages for that target gives the same response code and content OR it is clear that the http version of pages for that target indends users to access the site over https.

    For this last point, an http site that responds with a 200 code and response body of "access this site over https", and when you do responds with a 404, or a 301/302 redirect from http to https are pretty much functionally equivalent.

  12. Hainish commented on Nov 30, 2016

    @Hainish
    Member

    For some of these edge cases, I think it's also difficult to provide a hard and fast rule, and it's better to just rely on your best judgement. Over-formalization of process can get in the way of making reasonable decisions in odd circumstances.

  13. jeremyn commented on Nov 30, 2016

    @jeremyn
    ContributorAuthor

    @Hainish

    and when you do responds with a 404

    I don't understand this, did you make a typo?

    For some of these edge cases, I think it's also difficult to provide a hard and fast rule, and it's better to just rely on your best judgement. Over-formalization of process can get in the way of making reasonable decisions in odd circumstances.

    I think it's better to have over-formal rules and the reviewer has the discretion of relaxing the rules, instead of too-few rules and the reviewer comes across as arbitrary.

  14. Hainish commented on Nov 30, 2016

    @Hainish
    Member

    @jeremyn no. I'm describing an edge-case where the http endpoint body just says "Use HTTPS" with a 200 code, and the HTTPS endpoint responds with 404 codes. Theoretically I don't see any problem with accepting a ruleset like this, since it is

    clear that the http version of pages for that target indends users to access the site over https

    Practically, it may not offer much utility to provide just 404 pages. But I could imagine even a 404 providing useful information to a user.

  15. jeremyn commented on Nov 30, 2016

    @jeremyn
    ContributorAuthor

    Do you have an example of that sort of page? The only thing I can think of is https://www.senate.gov , which is the reverse: a working HTTPS page that just advises the user to look at the HTTP page.

    Regardless, I don't want to sign off on pages until I can actually see something work. It's the simplest way to make sure that we, ultimately, aren't making things worse somehow.

    Also, you can imagine an MITM attack where the attacker sends fake HTTP content redirecting the user to HTTPS knowing that HTTPS doesn't work. That sounds silly but that could do some damage if the attacker could trick an HTTPS Everywhere reviewer into breaking the site for all users of the extension.

    Anyway I don't mean to drag out a conversation about an edge case.

  16. Hainish commented on Dec 2, 2016

    @Hainish
    Member

    Regardless, I don't want to sign off on pages until I can actually see something work. It's the simplest way to make sure that we, ultimately, aren't making things worse somehow.

    A site can exhibit signs of working without necessarily having a 2XX status. For instance, I could envision a capture-the-flag type game where a 404 status code is returned unless your path contains a specific password. So you go to the ctf website, and it explains the rules of the game, and on a subdomain (let's say challenge1.examplectf.com) you get a whole bunch of 404s. It may not be trivial to discover the password. But since you know the rules of the game, you know that 404 status codes are expected. Thus if we know that the rule is working, even though we've never seen a 200 response. This doesn't seem like a contrived example, there are probably ctf challenges of this type out there, and likely many other types of sites where you won't find a 200 page easily. But you may be reasonably assured that the rule doesn't break anything. If you're not reasonably assured of that, don't merge the rule.

    The purpose of this is to draw out an example of how we would be forced to break an overly stringent rule that we've set for ourselves under certain unforeseen circumstances.

  17. jeremyn commented on Dec 2, 2016

    @jeremyn
    ContributorAuthor

    The rules should cover most cases and give the reviewers something to refer contributors to. The typical case is not a security challenge site with intentional 404s. The typical case, I think, is more like a contributor not associated with the domain includes a REST endpoint where the root subdomain returns a 404. In that case I want to see some 200/3xx URL so I have some assurance we're not going to break something. A rule makes a request for this URL, and the review process generally, seem less arbitrary. If someone does submit a security challenge site with intentional 404s, we can possibly make an exception, which is also arbitrary, but people are happier with arbitrariness if it means less work rather than more work.

  18. Hainish commented on Dec 3, 2016

    @Hainish
    Member

    It seems clumsy to make stringent rules where there are obvious exceptions that can be demonstrated, rather than building the exceptions into the language of the rule. My example above demonstrates a possible exception, regardless of what is 'typical.' If you're creating a rule, the rule should be amenable to circumstances where an atypical site is encountered. You can easily account for this by changing the language to reflect our example:

    In most cases, a 2XX or 3XX endpoint is required for rule adoption, except when it is clear that the site functions as intended in the absence of such an endpoint.

    This last clause is sufficient to give leeway when such circumstances are encountered, but still strong enough to provide a high burden of proof.

  19. Hainish commented on Dec 7, 2016

    @Hainish
    Member

    I'll work this into documentation in CONTRIBUTING.md

  20. jeremyn commented on Dec 7, 2016

    @jeremyn
    ContributorAuthor

    Thanks @Hainish . I'll close this issue, and we can discuss it further when the documentation is ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions