Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Automatic generation of rulesets #7691

Closed

Description

@galeksandrp

https-everywhere-bot got mad?????

@J0WI @jeremyn

Submit app comparison (+sources) Generators comparison (+sources) Domain enumerators

Submit app comparison

Note that ✅ is clickable

Backend Frontend Prod License Dependencies GitHub caps
galeksandrp@webhook galeksandrp/httpse ✅ MIT Travis GitHub Docker (galeksandrp@docker-httpse)
galeksandrp@webhook-2 galeksandrp@gh-pages ✅ MIT Travis GitHub Docker (galeksandrp@docker-httpse) public_repo
galeksandrp@webhook-client galeksandrp/galeksandrp.github.io@master ✅ MIT Travis GitHub Docker (galeksandrp@docker-httpse) public_repo email

Generators support

Author/Link @Foorack #6912 (comment) gloomy-ghost/ew@master galeksandrp@check-sublist3r @pgerber #7706 Foorack/httpse-ruleset-generator@master guiweber/ssl-yeah@master
galeksandrp/httpse ✔️ ✅ ✔️ ✔️
galeksandrp@gh-pages ✔️ ✅ ✔️
galeksandrp/galeksandrp.github.io@master ✅ ✅ ✅

✔️ not deployed to prod
✅ deployed to prod

https://github.com/galeksandrp/httpse/issues/new
  • No progress shown, but you can just wait when your PR will be arrived or see Travis log for specified *-auto branch in repo below.
  • Incoming PRs will have mine as author. (fix can produce issues since github limit unanteficated requests to 60 per hour)
  • Use github-email for discovering email.
  • Single ruleset can't be PRed once. (branch will be rebased if existing branch have old test scripts)
  • If ruleset is already present in HTTPSE and have different filename than lowercase.domain.xml PR will be created, but PR's Travis will fail.
  • Issue number not passed to PR, so issue can't be automatically closed.
  • Even empty ruleset will be PRed.
  • ~~~Scan limit is 10m.~~~ Now 50m.
  • Use shallow clone for speedup.
  • Mention issue author in PR.
  • Automatically comment && close issue when HSTS or no rules found.
https://galeksandrp.github.io/https-everywhere/
Open PRs
  • Check empty comment
  • Rescan rulesets
  • Rescan rulesets for mixed content

My mixed content position

Achived copy of this post in my repository: okyes/httpse#122 (comment)

Activity

  1. jeremyn commented on Nov 24, 2016

    @jeremyn
    Contributor

    Thanks @galeksandrp . I think you mean @J0WI (with a zero). Also @gloomy-ghost is a new maintainer.

  2. gloomy-ghost commented on Nov 24, 2016

    @gloomy-ghost
    Collaborator

    It's a good idea, and I personally have a python script to do the same job. I'm not good at bash programming, but here is a suggestion:

    • Check http when https returns a non-2xx code.
    • If https also returns a non-2xx code, try to automatically find a test on Google/Bing/DuckDuckGo/whatever. (I'm currently doing it by hand, which is boring)
  3. jeremyn commented on Nov 24, 2016

    @jeremyn
    Contributor

    I think @Foorack also has a script as well.

  4. numismatika commented on Nov 25, 2016

    @numismatika
    Contributor

    In this context we also should revisit #2718 and #6912 as they are closely related.

  5. Foorack commented on Nov 25, 2016

    @Foorack

    Good day. Yes, I have for some time working on a ruleset generator, Current version is completely written in python. It currently integrates with Sublis3r to automatically find domains. It can also check for MCB, SSL issues, cert issues and a bunch of other things. I have recently been taking all feedback I got to improve the program and I'm currently in the process of rewriting it again due to the recent complexity.

  6. Foorack commented on Nov 25, 2016

    @Foorack

    Personally, I suggest we all should work on the same project. Both for code efficiency (stronger together) but also to save time. The goal would be to get the program into a state where it can cover nearly any domain. We currently have a golden situation with the vast amount of training data available for us. Ultimately I think it should be merged into the HTTPSE project. What is everyone's opinion on this?

    Update: My two responses above was written on my phone. I only responded to jeremyn's comment and did not read the original issue description. As I understand it you want people to create an issue with the domain in the title. Your program will then generate a ruleset and push it to the HTTPSE repository. To be honest, I am not in favour of your approach for several reasons.

    First, it will make up dependent on you and your system. Let's say if anything were to happen or you lost all interest; then we would be in a difficult situation. Secondly, your approach seems like it's closed-source, as I can't find anything about it on your profile. Best would be for it to be open source under a license like GPL where anyone can contribute to improving it and use it to generate rulesets on their own.

  7. galeksandrp commented on Nov 25, 2016

    @galeksandrp
    ContributorAuthor
  8. jeremyn commented on Nov 25, 2016

    @jeremyn
    Contributor

    As a maintainer, I would really prefer all of you who are using scripts work on them together and in the open. If you are going to take a role in a script writing process, please stay involved in discussions such as #6868 . If you are going to write a script that produces output that I need to review, then it should take into account how I review it, and I don't want to have separate, different conversations with each of you about that.

    If you want a script reviewed, put the script up for code review, not a hundred pull requests from the script that all have the same problems. I would rather spend my time encouraging new contributors and resolving tricky issues than robotically reviewing a ton of script output from from experienced contributors. You should submit only a few pull requests at a time and your script should take feedback into account before you make new pull requests with it.

    @galeksandrp In particular, you have put a lot of time into HTTPS Everywhere and obviously care about the project, and that's great. However you produce a lot of messy pull requests and then are slow about taking feedback. For the fnal.gov pull request that we discussed several times, I put a lot of time into an initial review, you didn't implement my feedback, and then I spent more time implementing my own changes onto your branch. I resubmitted your #5372 as my #7493 to get it off my list. Your example PR in this issue, #7692 , has a blank top comment, which is obviously wrong.

    Anyone who submits a pull request should be willing to take feedback on that pull request in a timely way. I am very against an automatic rule submission process where a person can submit a messy pull request under someone else's name that a maintainer then has to clean up after.

  9. jeremyn commented on Nov 25, 2016

    @jeremyn
    Contributor

    @smw-koops For your #7691 (comment) , see what I wrote in #6322 (comment) .

  10. gloomy-ghost commented on Nov 25, 2016

    @gloomy-ghost
    Collaborator

    @jeremyn I think we can make reviewing of simple rulesets (only http -> https) automatically. We were working like bots in most of PRs, and I really don't like that.

    Testing on travis-ci are conservative, and we can have some progress as reviewers that are more radical. The idea is to pass targets which are obviously okay, and filter out the others for further checking. For example, automatically finds MCB problems and throws decoding error, and reviewers can check these targets separately. Also we can set max_distance = 0.01, so we can check small difference between http and https.

  11. jeremyn commented on Nov 26, 2016

    @jeremyn
    Contributor

    @gloomy-ghost It should be possible but in reality, I guess well under 5% of the pull requests I review pass without having some problem. A valid script should be able to pass a ruleset Turing test, where the reviewer submits several flawless rulesets and only then reveals they were using a script to make them. That can't have happened yet because so few rulesets pass without a problem.

    To clarify my earlier comment, I don't mind reviewing a ruleset just because it came from a script. I do mind reviewing hundreds of rulesets that all have similar problems, or a ruleset with an obvious problem (#7692 with the blank top comment), or a super complicated ruleset with lots of problems (#7504 (comment) for SLU.se).

    I would like it to work like this:

    1. A contributor writes a script and makes it available for community review.
    2. They submit a ruleset with that script from some limited set of domains.
    3. I find a problem with the ruleset.
    4. The contributor either (a) manually fixes the ruleset, and I merge it, or (b) disagrees with my review and either I change my mind and merge it, or the contributor changes their mind and goes back to (a).
    5. If (a), the contributor also updates the public script so it doesn't make the same mistake again.

    Once enough rulesets are submitted without any problems found, then we can be confident enough to allow the script to automatically generate rulesets for whatever our limited set of domains is. We can then try expanding the ruleset to cover a larger set of domains.

    In the above sequence, by "I find a problem", I mean any of the reviewers. If the contributor gets conflicting feedback from different reviewers, then we need to fix the conflict before moving forward. Similarly, by "a contributor", I mean any contributor working on the common script project.

  12. galeksandrp commented on Dec 15, 2016

    @galeksandrp
    ContributorAuthor
  13. galeksandrp commented on Mar 30, 2017

    @galeksandrp
    ContributorAuthor

    Note that ✅ is clickable

    Author / Link License Language Dependencies Readme/Launch script Install info Example CI
    @Foorack #6912 (comment) custom nolicense python3 python2.7 (aboul3la/Sublist3r@master) ✅ see below ✅ ✅
    gloomy-ghost/ew@master GPLv3 python3 (gloomy-ghost/Sublist3r@1b12560) see below see below
    galeksandrp@check-sublist3r MIT bash python2 (aboul3la/Sublist3r@master https-everywhere-checker) ✅ #6912 (comment) ✅ ✅
    @pgerber #7706 GPLv2+ python3 python2 (aboul3la/Sublist3r@master) #7706 (comment) #7706 (comment) #7706 (comment) #7706 (comment) ✅
    Foorack/httpse-ruleset-generator@master GPLv3 python3.4 (aboul3la/Sublist3r@ebe9a05) ✅ Foorack/httpse-ruleset-generator#1 (comment) ✅ ✅
    guiweber/ssl-yeah@master GPLv3 node ✅ ✅
    Author / Link Refused Timeout Bad cert Wrong chain SSL error Redirect Different content Non 200 Mixed content
    @Foorack #6912 (comment) ✅ ✅ ✅ ✅ ✅ ✅ ✅
    gloomy-ghost/ew@master ✅ ✅ ✅ ✅ ✅ ✅ ✅
    galeksandrp@check-sublist3r ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
    @pgerber #7706 ✅ ✅ ✅ ✅ ✅
    Foorack/httpse-ruleset-generator@master ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅

    Achived copy of this post in my repository: okyes/httpse#120 (comment)

  14. galeksandrp commented on Mar 30, 2017

    @galeksandrp
    ContributorAuthor
    https://github.com/galeksandrp/https-everywhere/commit/check-sublist3r
    • When no issues detected, produces an empty comment
    • Produces comments for nonexisting domains.
    • Does not check mixed content. Mixed check #8960

    Bonus:

    Author / Link list of domains to STDIN xml to STDOUT list of domains from file xml to file Insert to existing xml DNS zone as input Check existing XML Top comment as input
    @Foorack #6912 (comment) ✅
    gloomy-ghost/ew@master ✅
    galeksandrp@check-sublist3r ✅ ✅
    @pgerber #7706 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
    Foorack/httpse-ruleset-generator@master ✅

    @Foorack #6912 (comment)

    git clone https://gist.github.com/51893f7d0f8b16d6d8be25073131d4b6.git
    chmod +x ct.sh
    git clone https://github.com/aboul3la/Sublist3r.git
    docker run --rm -ti -v $(pwd):/opt -w /opt electronicfrontierfoundation/https-everywhere-docker-base
    apt-get update
    apt-get install -y $(cat Sublist3r/requirements.txt | sed 's/^/python-/') python3-requests
    ./ct.sh europa.eu

    gloomy-ghost/ew@master

    git clone https://github.com/gloomy-ghost/ew.git
    git submodule init
    git submodule update
    docker run --rm -ti -v $(pwd):/opt -w /opt electronicfrontierfoundation/https-everywhere-docker-base
    ln -s Sublist3r/subbrute .
    python3 sub.py

    Achived copy of this post in my repository: okyes/httpse#121 (comment)

  15. 2 remaining items

  16. cschanaj commented on Aug 7, 2017

    @cschanaj
    Collaborator

    FYI, I have a C program to generate rulesets which pass Travis build in most of the time. Source code available at https://github.com/cschanaj/xhttpse2; for a list of error supported, see src/HttpseCode.h

    Please let me know if there is anything I can help with this issue. Thanks.

  17. self-assigned this
    on Sep 14, 2017
  18. jeremyn commented on Oct 10, 2017

    @jeremyn
    Contributor

    @strugee The issue bot is having trouble here, FYI.

  19. wonderchook commented on Feb 13, 2018

    @wonderchook
    Contributor

    @galeksandrp I was thinking it might be useful to move your tables to the wiki. Right now because of the way Github issues limits the width of an issue they are a little hard to read.

  20. zero77 commented on Apr 17, 2018

    @zero77
    Contributor

    @galeksandrp
    How is this coming along and do you have a proto type that can be tested.

  21. Hainish commented on Oct 31, 2018

    @Hainish
    Member

    I'm closing this issue. Please let me know if you want it re-opened, @galeksandrp. Please refer to #16984 for further discussion.

  22. self-assigned this
    on Jun 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions