You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
@guiweber, based on your latest comments it sounds like SSL-Yeah is pretty sophisticated at detecting mixed content and crawling multiple pages on a site, which are two things the https-everywhere-checker currently lacks. Can you modify it to do crawling and mixed content detection on existing rulesets, while performing rewrites? That would be hugely useful.
To answer your question about automatically generating test URLs: Yes, the https-everywhere-checker takes any non-wildcard target host tag as an implicit test URL. However, to get good test coverage we ideally want multiple URLs per hostname. For instance, on a news site we'd like to have the root page (implicit), plus a feature article page, plus a photo gallery page, plus a video page, etc. As a secondary goal, if your tool can generate a sampling of test URLs in addition to the implicit ones, that would be helpful.
Yes, this would definitely be possible, although I'll be finishing the rule writing functionality and do some polishing work before getting started on this.
So just to make sure I understand, here are the basic steps it would need to do:
Read ruleset file
Perform rewrite for each host and load it, detecting mixed content
Load a few more pages for each host, while performing the rewrites and detecting mixed content
What kind of reporting/messaging would be needed? How would you be invoking the script? Do you want to test all existing rules or just new rules are they are submitted?
I already plan on adding the creation of a varied sample of test URLs (with images, video, article) in the rule-writing functionality, but doing this while testing has limitations. You would need to either use the first pages crawled no matter what they contain, or to load more pages to get some variety, which would increase the testing time (I suggest keeping the rate at which pages are loaded at 1 per second, as some hosts don't play well if requesting pages faster).
Yep, you have the right idea with your list of actions.
The output should include, for each ruleset, which pages failed and why, similar to the 'Disabled because' reasons the https-everywhere-checker currently puts into failing rulesets.
I would want to run it the same way I currently run the https-everywhere-checker: Occasionally running it manually for all rules. Also, for both https-everywhere-checker, and potentially ssl-yeah, I want to run it on two other occasions:
Automatically, once per day, for all rulesets
Automatically, on any pull request, for any rulesets that changed in that pull request.
Neither of these is implemented yet but I plan to.
For mixed content you should have some heuristics, e.g.:
If 100% of CSS resources or 100% of JS resources on a page are mixed content, fail.
If some JS or some CSS fails, but not 100%, it may be JS ad tracking requests or auxiliary stylesheets, so we'll assume they're fine.
If any object resource on a page is mixed content, fail.
Otherwise pass.
Probably we want to ignore iframe mixed content as well, because those are likely to be from ad trackers as well. But we might change that depending on what testing shows in practice.
Does it make sense to use PhantomJS? It might do a better job of trying to render a page.
Sounds good. I will get started with this and will let you know if I need more details or have other questions along the way.
I was planning on implementing some sort of headless browser at some point in the future, as a way of detecting MCB triggered by XMLHttpRequests. It looks like Phantom JS would do perfectly, but I'll have to give it a try. There seem to be good libraries that allow controlling it from node.js, so I'm hopeful it won't be too complicated.
The other possibility, BTW, is to just run Firefox headless using Xvfb, and have a driver extension that loads URLs, crawls, and watches for MCB. There's already some ~2 year old code that does this inside the HTTPS Everywhere extension, called the "ruleset tests." You could pull that out and revitalize that. The advantage would be that you'd be running against the actual extension code, with all its quirks and bug fixes, so you'd minimize false positives and false negatives.
Activity
guiweber commentedon Mar 15, 2015
Yes, this would definitely be possible, although I'll be finishing the rule writing functionality and do some polishing work before getting started on this.
So just to make sure I understand, here are the basic steps it would need to do:
What kind of reporting/messaging would be needed? How would you be invoking the script? Do you want to test all existing rules or just new rules are they are submitted?
I already plan on adding the creation of a varied sample of test URLs (with images, video, article) in the rule-writing functionality, but doing this while testing has limitations. You would need to either use the first pages crawled no matter what they contain, or to load more pages to get some variety, which would increase the testing time (I suggest keeping the rate at which pages are loaded at 1 per second, as some hosts don't play well if requesting pages faster).
jsha commentedon Mar 16, 2015
Yep, you have the right idea with your list of actions.
The output should include, for each ruleset, which pages failed and why, similar to the 'Disabled because' reasons the https-everywhere-checker currently puts into failing rulesets.
I would want to run it the same way I currently run the https-everywhere-checker: Occasionally running it manually for all rules. Also, for both https-everywhere-checker, and potentially ssl-yeah, I want to run it on two other occasions:
Neither of these is implemented yet but I plan to.
For mixed content you should have some heuristics, e.g.:
Probably we want to ignore iframe mixed content as well, because those are likely to be from ad trackers as well. But we might change that depending on what testing shows in practice.
Does it make sense to use PhantomJS? It might do a better job of trying to render a page.
guiweber commentedon Mar 23, 2015
Sounds good. I will get started with this and will let you know if I need more details or have other questions along the way.
I was planning on implementing some sort of headless browser at some point in the future, as a way of detecting MCB triggered by XMLHttpRequests. It looks like Phantom JS would do perfectly, but I'll have to give it a try. There seem to be good libraries that allow controlling it from node.js, so I'm hopeful it won't be too complicated.
jsha commentedon Mar 23, 2015
The other possibility, BTW, is to just run Firefox headless using Xvfb, and have a driver extension that loads URLs, crawls, and watches for MCB. There's already some ~2 year old code that does this inside the HTTPS Everywhere extension, called the "ruleset tests." You could pull that out and revitalize that. The advantage would be that you'd be running against the actual extension code, with all its quirks and bug fixes, so you'd minimize false positives and false negatives.
pipboy96 commentedon May 9, 2019
Project seems to no longer be developed, last update in 2016. Last update of issue in 2015. If this should be re-opened, tell me.