You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
How to handle domains with very large subdomains list? #8545
I browsed through the previous issues and couldn't find anything related to my question. I apologize if it was already asked and I simply missed it.
Recently, I came across a lot of potential candidates for a new ruleset but after looking them up with Sublist3r, I found out they have a huge number of subdomains.
A good example here would be mnhn.fr.
Sublist3r finds 744 subdomains for it. I'm betting a lot of them are not functional but still...
I am wondering what should be done in this case:
A wildcard rule is out of question because only some subdomains have a valid HTTPS configuration
I could check every single subdomain but even then, it would result in a huge ruleset, leading to a painful review and a difficult maintaining. I am not even sure such a ruleset would be accepted due to its size.
Don't do anything and wait for the domain to have HSTS on all subdomains or at least valid HTTPS on all of them
It depends on the subdomains. In pull request #6899 we covered about 80 subdomains using a wildcard and a special rule, but the subdomains apparently had identical content, so that's fine. On the other end is pull request #5265 for fnal.gov where we want to add mentions of over a hundred subdomains. These are I suppose managed by a variety of groups at a huge government science lab so a wildcard will not work. It's still waiting for review by someone other than me (since I made modifications to it).
Huge rulesets are unattractive to review. You can put a lot of time into one ruleset and then the contributor doesn't respond quickly. It can be tricky to discuss a multiple problems in a single unthreaded GitHub pull request.
I see about 240 mnhn.fr subdomains using the recommended Sublist3r settings. A lot of the subdomains are internal/technical like autodiscover, horde5, imap*, test*, pop3, proxy, and webmail. In my opinion these can be ignored entirely.
At https://mnhn.fr I get an invalid certificate/incomplete certificate chain error, which is not a good sign.
If I were you, I might submit a small pull request with some of the most important subdomains in a ruleset with (partial) in the name. If that went well then I would submit more pull requests later adding more subdomains. Add a comment in the pull request that you understand there are a lot more subdomains than you included but that you don't want to overwhelm the reviewer.
Activity
jeremyn commentedon Feb 20, 2017
It depends on the subdomains. In pull request #6899 we covered about 80 subdomains using a wildcard and a special rule, but the subdomains apparently had identical content, so that's fine. On the other end is pull request #5265 for fnal.gov where we want to add mentions of over a hundred subdomains. These are I suppose managed by a variety of groups at a huge government science lab so a wildcard will not work. It's still waiting for review by someone other than me (since I made modifications to it).
Huge rulesets are unattractive to review. You can put a lot of time into one ruleset and then the contributor doesn't respond quickly. It can be tricky to discuss a multiple problems in a single unthreaded GitHub pull request.
I see about 240 mnhn.fr subdomains using the recommended Sublist3r settings. A lot of the subdomains are internal/technical like
autodiscover,horde5,imap*,test*,pop3,proxy, andwebmail. In my opinion these can be ignored entirely.At https://mnhn.fr I get an invalid certificate/incomplete certificate chain error, which is not a good sign.
If I were you, I might submit a small pull request with some of the most important subdomains in a ruleset with
(partial)in thename. If that went well then I would submit more pull requests later adding more subdomains. Add a comment in the pull request that you understand there are a lot more subdomains than you included but that you don't want to overwhelm the reviewer.