You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
It's a good idea, and I personally have a python script to do the same job. I'm not good at bash programming, but here is a suggestion:
Check http when https returns a non-2xx code.
If https also returns a non-2xx code, try to automatically find a test on Google/Bing/DuckDuckGo/whatever. (I'm currently doing it by hand, which is boring)
Good day. Yes, I have for some time working on a ruleset generator, Current version is completely written in python. It currently integrates with Sublis3r to automatically find domains. It can also check for MCB, SSL issues, cert issues and a bunch of other things. I have recently been taking all feedback I got to improve the program and I'm currently in the process of rewriting it again due to the recent complexity.
Personally, I suggest we all should work on the same project. Both for code efficiency (stronger together) but also to save time. The goal would be to get the program into a state where it can cover nearly any domain. We currently have a golden situation with the vast amount of training data available for us. Ultimately I think it should be merged into the HTTPSE project. What is everyone's opinion on this?
Update: My two responses above was written on my phone. I only responded to jeremyn's comment and did not read the original issue description. As I understand it you want people to create an issue with the domain in the title. Your program will then generate a ruleset and push it to the HTTPSE repository. To be honest, I am not in favour of your approach for several reasons.
First, it will make up dependent on you and your system. Let's say if anything were to happen or you lost all interest; then we would be in a difficult situation. Secondly, your approach seems like it's closed-source, as I can't find anything about it on your profile. Best would be for it to be open source under a license like GPL where anyone can contribute to improving it and use it to generate rulesets on their own.
As a maintainer, I would really prefer all of you who are using scripts work on them together and in the open. If you are going to take a role in a script writing process, please stay involved in discussions such as #6868 . If you are going to write a script that produces output that I need to review, then it should take into account how I review it, and I don't want to have separate, different conversations with each of you about that.
If you want a script reviewed, put the script up for code review, not a hundred pull requests from the script that all have the same problems. I would rather spend my time encouraging new contributors and resolving tricky issues than robotically reviewing a ton of script output from from experienced contributors. You should submit only a few pull requests at a time and your script should take feedback into account before you make new pull requests with it.
@galeksandrp In particular, you have put a lot of time into HTTPS Everywhere and obviously care about the project, and that's great. However you produce a lot of messy pull requests and then are slow about taking feedback. For the fnal.gov pull request that we discussed several times, I put a lot of time into an initial review, you didn't implement my feedback, and then I spent more time implementing my own changes onto your branch. I resubmitted your #5372 as my #7493 to get it off my list. Your example PR in this issue, #7692 , has a blank top comment, which is obviously wrong.
Anyone who submits a pull request should be willing to take feedback on that pull request in a timely way. I am very against an automatic rule submission process where a person can submit a messy pull request under someone else's name that a maintainer then has to clean up after.
@jeremyn I think we can make reviewing of simple rulesets (only http -> https) automatically. We were working like bots in most of PRs, and I really don't like that.
Testing on travis-ci are conservative, and we can have some progress as reviewers that are more radical. The idea is to pass targets which are obviously okay, and filter out the others for further checking. For example, automatically finds MCB problems and throws decoding error, and reviewers can check these targets separately. Also we can set max_distance = 0.01, so we can check small difference between http and https.
@gloomy-ghost It should be possible but in reality, I guess well under 5% of the pull requests I review pass without having some problem. A valid script should be able to pass a ruleset Turing test, where the reviewer submits several flawless rulesets and only then reveals they were using a script to make them. That can't have happened yet because so few rulesets pass without a problem.
To clarify my earlier comment, I don't mind reviewing a ruleset just because it came from a script. I do mind reviewing hundreds of rulesets that all have similar problems, or a ruleset with an obvious problem (#7692 with the blank top comment), or a super complicated ruleset with lots of problems (#7504 (comment) for SLU.se).
I would like it to work like this:
A contributor writes a script and makes it available for community review.
They submit a ruleset with that script from some limited set of domains.
I find a problem with the ruleset.
The contributor either (a) manually fixes the ruleset, and I merge it, or (b) disagrees with my review and either I change my mind and merge it, or the contributor changes their mind and goes back to (a).
If (a), the contributor also updates the public script so it doesn't make the same mistake again.
Once enough rulesets are submitted without any problems found, then we can be confident enough to allow the script to automatically generate rulesets for whatever our limited set of domains is. We can then try expanding the ruleset to cover a larger set of domains.
In the above sequence, by "I find a problem", I mean any of the reviewers. If the contributor gets conflicting feedback from different reviewers, then we need to fix the conflict before moving forward. Similarly, by "a contributor", I mean any contributor working on the common script project.
@galeksandrp I was thinking it might be useful to move your tables to the wiki. Right now because of the way Github issues limits the width of an issue they are a little hard to read.
Activity
jeremyn commentedon Nov 24, 2016
Thanks @galeksandrp . I think you mean @J0WI (with a zero). Also @gloomy-ghost is a new maintainer.
gloomy-ghost commentedon Nov 24, 2016
It's a good idea, and I personally have a python script to do the same job. I'm not good at bash programming, but here is a suggestion:
jeremyn commentedon Nov 24, 2016
I think @Foorack also has a script as well.
numismatika commentedon Nov 25, 2016
In this context we also should revisit #2718 and #6912 as they are closely related.
Foorack commentedon Nov 25, 2016
Good day. Yes, I have for some time working on a ruleset generator, Current version is completely written in python. It currently integrates with Sublis3r to automatically find domains. It can also check for MCB, SSL issues, cert issues and a bunch of other things. I have recently been taking all feedback I got to improve the program and I'm currently in the process of rewriting it again due to the recent complexity.
Foorack commentedon Nov 25, 2016
Personally, I suggest we all should work on the same project. Both for code efficiency (stronger together) but also to save time. The goal would be to get the program into a state where it can cover nearly any domain. We currently have a golden situation with the vast amount of training data available for us. Ultimately I think it should be merged into the HTTPSE project. What is everyone's opinion on this?
Update: My two responses above was written on my phone. I only responded to jeremyn's comment and did not read the original issue description. As I understand it you want people to create an issue with the domain in the title. Your program will then generate a ruleset and push it to the HTTPSE repository. To be honest, I am not in favour of your approach for several reasons.
First, it will make up dependent on you and your system. Let's say if anything were to happen or you lost all interest; then we would be in a difficult situation. Secondly, your approach seems like it's closed-source, as I can't find anything about it on your profile. Best would be for it to be open source under a license like GPL where anyone can contribute to improving it and use it to generate rulesets on their own.
galeksandrp commentedon Nov 25, 2016
@Foorack
jeremyn commentedon Nov 25, 2016
As a maintainer, I would really prefer all of you who are using scripts work on them together and in the open. If you are going to take a role in a script writing process, please stay involved in discussions such as #6868 . If you are going to write a script that produces output that I need to review, then it should take into account how I review it, and I don't want to have separate, different conversations with each of you about that.
If you want a script reviewed, put the script up for code review, not a hundred pull requests from the script that all have the same problems. I would rather spend my time encouraging new contributors and resolving tricky issues than robotically reviewing a ton of script output from from experienced contributors. You should submit only a few pull requests at a time and your script should take feedback into account before you make new pull requests with it.
@galeksandrp In particular, you have put a lot of time into HTTPS Everywhere and obviously care about the project, and that's great. However you produce a lot of messy pull requests and then are slow about taking feedback. For the
fnal.govpull request that we discussed several times, I put a lot of time into an initial review, you didn't implement my feedback, and then I spent more time implementing my own changes onto your branch. I resubmitted your #5372 as my #7493 to get it off my list. Your example PR in this issue, #7692 , has a blank top comment, which is obviously wrong.Anyone who submits a pull request should be willing to take feedback on that pull request in a timely way. I am very against an automatic rule submission process where a person can submit a messy pull request under someone else's name that a maintainer then has to clean up after.
jeremyn commentedon Nov 25, 2016
@smw-koops For your #7691 (comment) , see what I wrote in #6322 (comment) .
gloomy-ghost commentedon Nov 25, 2016
@jeremyn I think we can make reviewing of simple rulesets (only http -> https) automatically. We were working like bots in most of PRs, and I really don't like that.
Testing on travis-ci are conservative, and we can have some progress as reviewers that are more radical. The idea is to pass targets which are obviously okay, and filter out the others for further checking. For example, automatically finds MCB problems and throws decoding error, and reviewers can check these targets separately. Also we can set
max_distance = 0.01, so we can check small difference between http and https.jeremyn commentedon Nov 26, 2016
@gloomy-ghost It should be possible but in reality, I guess well under 5% of the pull requests I review pass without having some problem. A valid script should be able to pass a ruleset Turing test, where the reviewer submits several flawless rulesets and only then reveals they were using a script to make them. That can't have happened yet because so few rulesets pass without a problem.
To clarify my earlier comment, I don't mind reviewing a ruleset just because it came from a script. I do mind reviewing hundreds of rulesets that all have similar problems, or a ruleset with an obvious problem (#7692 with the blank top comment), or a super complicated ruleset with lots of problems (#7504 (comment) for
SLU.se).I would like it to work like this:
Once enough rulesets are submitted without any problems found, then we can be confident enough to allow the script to automatically generate rulesets for whatever our limited set of domains is. We can then try expanding the ruleset to cover a larger set of domains.
In the above sequence, by "I find a problem", I mean any of the reviewers. If the contributor gets conflicting feedback from different reviewers, then we need to fix the conflict before moving forward. Similarly, by "a contributor", I mean any contributor working on the common script project.
galeksandrp commentedon Dec 15, 2016
Generators comparison
galeksandrp commentedon Mar 30, 2017
Note that ✅ is clickable
@Foorack#6912 (comment)@pgerber#7706@Foorack#6912 (comment)@pgerber#7706Achived copy of this post in my repository: okyes/httpse#120 (comment)
galeksandrp commentedon Mar 30, 2017
https://github.com/galeksandrp/https-everywhere/commit/check-sublist3r
Bonus:
@Foorack#6912 (comment)@pgerber#7706@Foorack#6912 (comment)gloomy-ghost/ew@master
Achived copy of this post in my repository: okyes/httpse#121 (comment)
2 remaining items
cschanaj commentedon Aug 7, 2017
FYI, I have a C program to generate rulesets which pass Travis build in most of the time. Source code available at https://github.com/cschanaj/xhttpse2; for a list of error supported, see
src/HttpseCode.hPlease let me know if there is anything I can help with this issue. Thanks.
jeremyn commentedon Oct 10, 2017
@strugee The issue bot is having trouble here, FYI.
wonderchook commentedon Feb 13, 2018
@galeksandrp I was thinking it might be useful to move your tables to the wiki. Right now because of the way Github issues limits the width of an issue they are a little hard to read.
zero77 commentedon Apr 17, 2018
@galeksandrp
How is this coming along and do you have a proto type that can be tested.
Hainish commentedon Oct 31, 2018
I'm closing this issue. Please let me know if you want it re-opened, @galeksandrp. Please refer to #16984 for further discussion.