Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Added ulaval.ca - #1015

Merged
jsha merged 1 commit into
EFForg:masterfrom
guiweber:ulaval2
Feb 16, 2015
Merged

jsha merged 1 commit into
EFForg:masterfrom
guiweber:ulaval2

Conversation

@guiweber

@guiweber guiweber commented Feb 4, 2015

Copy link
Copy Markdown
Contributor

No description provided.

@jsha

jsha commented Feb 15, 2015

Copy link
Copy Markdown
Member

Per the new ruleset testing requirements and ruleset style guide, can you change this rule to list specify hosts in the target hosts, rather than wildcards?

Thanks,
Jacob

@guiweber

Copy link
Copy Markdown
Contributor Author

This one has been updated as well.

After creating a rule with so many sub domains, I am wondering... has anybody ever thought about creating a bot for creating rules? It seems to me that a script that would check all known sub-domains for a given domain and verify if they accept https or not would be tremendously helpful.

jsha added a commit that referenced this pull request Feb 16, 2015
@jsha
jsha merged commit ca7399f into EFForg:master Feb 16, 2015
@jsha

jsha commented Feb 16, 2015

Copy link
Copy Markdown
Member

We've thought about that type of automated adding, but our biggest problem is not lack of rulesets, it's low quality rulesets. In particular, a number of web pages trigger mixed content blocking when loaded over HTTPS, which isn't immediately obvious when checking whether there is HTTPS with a valid certificate available. So implementing a bot would simply encourage creation of a lot of new low quality rulesets.

@guiweber
guiweber deleted the ulaval2 branch February 16, 2015 06:19
@guiweber

Copy link
Copy Markdown
Contributor Author

I understand your concern, and I totally agree that it cannot be fully automated. But I think the most tedious part of it can be scripted by checking the following:
1- If the server accepts https and has a valid certificate (only this will eliminate many sub domains)
2- If there are hard-linked active or passive http resources on the pages
3- try to load external resources where http/https isn't specified over https

With this info, the domains could be put under three categories: secure, insecure, and mixed. Only the mixed category would then have to be checked manually. I think a bot like this could even help improve the quality of rulesets since it would be able to check many more pages per sub domain than a human.

I'll try and see if I can come up with something that works.

@jsha

jsha commented Feb 17, 2015

Copy link
Copy Markdown
Member

Got it, thanks! If you're interested in adding the check for "hard-linked active or passive http resources on the pages," it would be most useful to add that in the https-everywhere-checker: https://github.com/hiviah/https-everywhere-checker/. That way we can use it to check existing rules for issues as well.

try to load external resources where http/https isn't specified over https

Keep in mind that if this is necessary, the page is already in the 'mixed' category, since both FF and Chrome now block those resources before they have a chance to be rewritten by HTTPS Everywhere.

@guiweber

Copy link
Copy Markdown
Contributor Author

Sure thing, I'll take a look at the https-everywhere-checker a well.

Regarding testing external resources where http/https isn't specified... maybe I'm missing something, but my thinking is that if a page links a resource like shown below, the browser will try to load the resource using the same protocol that was used to load the base page. It is not automatically blocked as mixed content.

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>

By only looking at the page source, I have no way to determine if the resource will actually load or not over https. Since I'm not doing this through a browser, the resources are not loaded when I load the page, so I believe I need to explicitly try loading them to see if the servers on which they located are accept ssl and have valid certificates.

@guiweber

Copy link
Copy Markdown
Contributor Author

Hi Jacob,

I just uploaded a first version of the rule creator: https://github.com/guiweber/ssl-yeah

It still needs some improvements to make it great, but it works. I tested it on a few large websites with good results. If you have time to try it out, let me know what you think, I'd like to have your comments/suggestions if you have any.

As for https-everywhere-checker, would you be able to invoke a nodejs script from python to test the content? I need to cleanup my code and make it more modular, but the functionality is there so it would be easy to create a module that you can call to verify the content of the pages that (I believe) are already loaded by the checker.

Cheers

@jsha

jsha commented Mar 10, 2015

Copy link
Copy Markdown
Member

Hi @guiweber,

According to the README this currently doesn't detect mixed content, which means I would not be comfortable promoting it as a method to generate rulesets. Ideally it should:

  • detect mixed content
  • do some degree of crawling the site beyond the home page
  • output some of the URLs crawled as test URLs in the ruleset.

https-everywhere-checker could potentially call out to nodejs but I would definitely prefer a tool that integrates with the existing code, because it will be easier to maintain.

@guiweber

Copy link
Copy Markdown
Contributor Author

Actually it does detect mixed content. Sorry if the readme is not clear, the limitation I am referring to is that it does not check within the js scripts on the page to see if those scripts do http calls on their own, after the page is loaded. But everything that is loaded with the page is detected, including the scripts themselves.

In addition to the rule xml file, it also outputs a detailed report where you can see all the resources that were detected as mixed content for each page of each domain, as well as any http errors or redirects.

Here is a screenshot of the report for reference:
large

It also crawls beyond the home page and crawls subdomains as well (these things can be customized in the settings).

The only thing it does not do for now is add test URLs in the ruleset, but I was wondering if this is necessary since each sub-domain has its own "target" tag (no wildcard used) in the ruleset. I thought that these were used as implicit test URLs on your side.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants