This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Unnecessary overlapping targets in rulesets #12322
Copy link
Copy link
Closed
Labels
Description
#12320 helped to find a list of rulesets where targets overlap, and so unnecessarily duplicate each other. Reducing number of those might help with #12232:
- 16163.com.xml:
*.16163.comalso coverswww.16163.com,app.16163.com,bbs.16163.com,m.16163.com - 3min.xml:
*.3min.dealso coverswww.3min.de - Acenet.xml:
*.acenet-inc.netalso covers*.esupport.acenet-inc.net - Advertising.com.xml:
*.ace.advertising.comalso coverssecure.ace.advertising.com - African-Network-Information-Center.xml:
*.afrinic.netalso covers*.meeting.afrinic.net - Airtricity.xml:
*.airtricity.comalso coverswww.airtricity.com - AliceDSL.xml:
*.alice-dsl.dealso coverswww.alice-dsl.de - American-University.xml:
*.american.edualso covers*.wcl.american.edu - American_Society_of_Media_Photographers.xml:
*.asmp.orgalso coverswww.admin.asmp.org - Argonne-National-Laboratory.xml:
*.anl.govalso coverswww.*.anl.gov - Argos.xml:
*.argos.co.ukalso coversimage.email.argos.co.uk - Audiko.xml:
*.audiko.netalso coverscss.cdn.audiko.net,jpg.st.audiko.net - Bauhaus-University_Weimar.xml:
*.uni-weimar.dealso covers*.webmail.uni-weimar.de - Bennetts.xml:
*.bennetts.co.ukalso covers*.quotes.bennetts.co.uk - BreNet.xml:
*.brenet.dealso covers*.webmail.brenet.de - Caller.com.xml:
*.caller.comalso coverswww.caller.com,login.caller.com - Catalog-of-Domestic-Federal-Assistance.xml:
*.cfda.govalso covers*.www.cfda.gov - Champs_Sports.xml:
*.champssports.comalso covers*.e.champssports.com,*.www.champssports.com - City_University_London.xml:
*.city.ac.ukalso coverswww.soi.city.ac.uk,*.www.city.ac.uk - Claranet.xml:
*.clara.netalso coverswebmail.bln.de.clara.net,nswebmail.uk.clara.net,portal.uk.clara.net - Cloudhexa_Network.xml:
*.cloudhexa.comalso covers*.www.cloudhexa.com - Columbia_University-problematic.xml:
*.hr.columbia.edualso covers*.managers.hr.columbia.edu - Compendium.xml:
*.compendiumblog.comalso coverscdn2.content.compendiumblog.com,cdn.content.compendiumblog.com,global.content.compendiumblog.com - Council_on_Foreign_Relations.xml:
*.cfr.orgalso coverssecure.www.cfr.org - Cox_Communications.xml:
*.cox.comalso covers*.store.cox.com - Cox_Communications.xml:
*.cox.netalso coversidm.east.cox.net - Crain-Communications.xml:
*.adage.comalso coverswww.amiga.adage.com - Dailymotion.xml:
*.dailymotion.comalso coversak2.static.dailymotion.com - Digia.xml:
*.digia.comalso coversblog.qt.digia.com - Digital_Photography_Review.xml:
*.img-dpreview.comalso covers*.static.img-dpreview.com - Digitaria.xml:
*.insightmgr.comalso covers*.digi.insightmgr.com,*.www.insightmgr.com - DotCOM-host.xml:
*.dotcomhost.comalso covers*.webmail.dotcomhost.com - dpfile.com.xml:
*.dpfile.comalso coversqcloud.dpfile.com,www.dpfile.com - Eastbay.xml:
*.eastbay.comalso covers*.teamsales.eastbay.com,*.www.eastbay.com - Economic-Policy-Institute.xml:
*.epi.orgalso covers*.secure.epi.org - Electronic-Arts.xml:
*.thesims3.comalso covers*.store.thesims3.com - Epson.xml:
*.epson.comalso coversglobal.latin.epson.com - Eyeviewads.com.xml:
*.eyeviewads.comalso coverstrack.eyeviewads.com - Familie-redlich.xml:
*.familie-redlich.dealso coverswww.systeme.familie-redlich.de - Federal_Business_Opportunities.xml:
*.fbo.govalso covers*.www.fbo.gov - Film-Threat.xml:
*.filmthreat.comalso covers*.www.filmthreat.com - Final-Score.xml:
*.final-score.comalso covers*.e.final-score.com,*.www.final-score.com - Footaction_USA.xml:
*.footaction.comalso covers*.www.footaction.com - Freelancer.xml:
*.freelancer.comalso covers*.www.freelancer.com - Freelancer.xml:
*.freelancer.co.ukalso covers*.www.freelancer.co.uk - FreeWheel.xml:
*.fwmrm.netalso covers2912a.v.fwmrm.net - FutureQuest.net.xml:
*.futurequest.netalso coverswww.service.futurequest.net - GaiaOnline.xml:
*.gaiaonline.comalso covers*.cdn.gaiaonline.com - Game-Show-Network.xml:
*.gsn.comalso coverswww.tv.gsn.com - garmin.xml:
*.garmin.comalso coverswww.garmin.com - Gigaserver.xml:
*.gigaserver.czalso covers*.www.gigaserver.cz - Gizmodo.com.xml:
*.gizmodo.comalso covers20khz.gizmodo.com,es.gizmodo.com,factually.gizmodo.com,fieldguide.gizmodo.com,homeofthefuture.gizmodo.com,indefinitelywild.gizmodo.com,io9.gizmodo.com,lego.gizmodo.com,offworld.gizmodo.com,paleofuture.gizmodo.com,reframe.gizmodo.com,space.gizmodo.com,sploid.gizmodo.com,throb.gizmodo.com,toyland.gizmodo.com,us.gizmodo.com,www.gizmodo.com,cache.gizmodo.com - GoogleImages.xml:
google.*also coversgoogle.com - GoogleImages.xml:
images.google.*also coversimages.google.com - GoogleServices_Complex.xml:
*.googleusercontent.comalso covers*.corp.googleusercontent.com - GoogleVideos.xml:
google.*also coversgoogle.com - GoStats.xml:
*.gostats.comalso coverswww.ssl.gostats.com - Griffith-University.xml:
*.griffith.edu.aualso covers*.secure.griffith.edu.au - HitBTC.com.xml:
*.hitbtc.comalso coversaffiliate.hitbtc.com,auth.hitbtc.com,blog.hitbtc.com,demo.hitbtc.com,forum.hitbtc.com,www.hitbtc.com - Home.pl.xml:
*.home.plalso covers*.akcje.home.pl,*.panel.home.pl,*.poczta.home.pl,*.m.poczta.home.pl - Home.pl.xml:
*.poczta.home.plalso covers*.m.poczta.home.pl - Honest.com.xml:
*.honest.comalso coversblog.honest.com,img.honest.com,www.honest.com - Hot_Pics_Amateur.xml:
*.hotpics-amateur.comalso coverswww.collection.hotpics-amateur.com - ImageShack.xml:
*.imageshack.usalso coverswww.imageshack.us,imagizer.imageshack.us,post.imageshack.us,a.imageshack.us - kantonalbanken.xml:
*.bkb.chalso coversintegration.quotes.bkb.ch - Kids_Foot_Locker.xml:
*.kidsfootlocker.comalso covers*.www.kidsfootlocker.com - KIXEYE.xml:
*.kixeye.comalso covers*.cdn.kixeye.com - Lady_Foot_Locker.xml:
*.ladyfootlocker.comalso covers*.www.ladyfootlocker.com - LastPass.com.xml:
*.lastpass.comalso coverswww.lastpass.com,0.lastpass.com,account.lastpass.com,accounts.lastpass.com,blog.lastpass.com,download.lastpass.com,enterprise.lastpass.com,forums.lastpass.com,helpdesk.lastpass.com,localvault.lastpass.com,m.lastpass.com,manda.lastpass.com,pollserver.lastpass.com,portable.lastpass.com,rodan.lastpass.com,service.lastpass.com,teams.lastpass.com,uber.lastpass.com,vaul.lastpass.com - Linux-New-Media.xml:
*.linuxnewmedia.comalso covers*.shop.linuxnewmedia.com - Linux-New-Media.xml:
*.linuxnewmedia.dealso covers*.shop.linuxnewmedia.de - Liquid-Web.xml:
*.liquidweb.comalso coversmedia.cdn.liquidweb.com - London-2012.xml:
*.london2012.comalso coverswww.festival.london2012.com,tickets.london2012.com,www.tickets.london2012.com - Loopia.xml:
*.loopia.sealso covers*.www.loopia.se - Lumosity.xml:
*.lumosity.comalso coversstatic.sl.lumosity.com - Mail.ru.xml:
*.foto.mail.rualso coversavt.foto.mail.ru,content.foto.mail.ru - Mail.ru.xml:
*.my.mail.rualso coverscontent.foto.my.mail.ru,stat.my.mail.ru,videoapi.my.mail.ru - Maricopa-Community-Colleges.xml:
*.maricopa.edualso covers*.sis.maricopa.edu - Maxymiser.xml:
*.maxymiser.comalso covers*.www.maxymiser.com - MediaFire.com.xml:
*.mediafire.comalso coversstaticcdn.mediafire.com,www.mediafire.com,www1.mediafire.com,www2.mediafire.com,cdn.mediafire.com,cdnssl.mediafire.com,m.mediafire.com - MeiTuan.com.xml:
*.meituan.comalso coverswww.meituan.com,analytics.meituan.com,b.meituan.com,daili.meituan.com,hotel.meituan.com,mos.meituan.com,p0.meituan.com,p1.meituan.com,passport.meituan.com,report.meituan.com,s0.meituan.com,s1.meituan.com,waimaie.meituan.com,i.meituan.com,waimai.meituan.com,kaidian.waimai.meituan.com - Mentor-Graphics.xml:
*.mentor.comalso covers*.store1.mentor.com - MIC-Gadget.xml:
*.micgadget.comalso covers*.store.micgadget.com - MSN-mismatches.xml:
*.msnbc.msn.comalso covers*.today.msnbc.msn.com - mytalkdesk.com.xml:
*.mytalkdesk.comalso coverswww.mytalkdesk.com - National_Park_Service.xml:
*.nature.nps.govalso coversscience.nature.nps.gov - NAVTEQ.xml:
*.navteq.comalso coverscss.mapreporter.navteq.com - NetMediaEurope.xml:
*.itespresso.fralso coversquiz.itespresso.fr - Oberlin_College.xml:
*.oberlin.edualso covers*.cs.oberlin.edu,oncampus.csr.oberlin.edu - ONEsite.xml:
*.onesite.comalso covers*.admin.onesite.com - OnSugar.xml:
*.onsugar.comalso coverssecure.*.onsugar.com,www.*.onsugar.com - Pair-Networks.xml:
*.pair.comalso covers*.webmail.pair.com - Polytechnic-University-of-Catalonia.xml:
*.upc.edualso covers*.blog.upc.edu - Poppy-Sports.xml:
*.poppysports.comalso covers*.www.poppysports.com - Prxy.com.xml:
*.prxy.comalso coverswww.prxy.com - Radboud-University-Nijmegen.xml:
*.ru.nlalso covers*.cmbi.ru.nl,*.hosting.ru.nl,*.portalhelp.hosting.ru.nl - Radboud-University-Nijmegen.xml:
*.hosting.ru.nlalso covers*.portalhelp.hosting.ru.nl - Royal_Mail.xml:
*.royalmail.comalso covers*.shop.royalmail.com - RTEMS.xml:
*.rtems.orgalso coversdevel.rtems.org,docs.rtems.org,git.rtems.org,lists.rtems.org,wiki.rtems.org,www.rtems.org - Secunet.xml:
*.secunet.comalso coverswww.secunet.com - SexNarod.xml:
*.superforum.orgalso covers*.dating.superforum.org - SexNarod.xml:
*.sxnarod.comalso coverswap.dating.sxnarod.com - Sheet-Music-Plus.xml:
*.sheetmusicplus.comalso coversssl.assets.sheetmusicplus.com - Skrill.com.xml:
*.skrill.comalso coversaccount.skrill.com,help.skrill.com,sso.skrill.com,www.skrill.com - Snagajob.xml:
*.snagajob.comalso coverswww.snagajob.com - Spoki.xml:
*.spoki.lvalso covers*.www.spoki.lv - StumbleUpon.xml:
*.stumbleupon.comalso covers*.b9.stumbleupon.com - Symantec.xml:
*.symanteccloud.comalso coversbuy.symanteccloud.com,static.symanteccloud.com,static1.symanteccloud.com,static2.symanteccloud.com,static3.symanteccloud.com - Target_Performance.xml:
*.ad-srv.netalso covers*.ad.ad-srv.net - Telefonica.xml:
*.o2.czalso covers*.www.o2.cz - Textbooks.com.xml:
*.textbooks.comalso covers*.www.textbooks.com - The-Escapist-Expo.xml:
*.escapistexpo.comalso coverswww.sec.escapistexpo.com - UBS.xml:
*.ubs.comalso covers*.ibb.ubs.com - UCSD.edu.xml:
*.ucsd.edualso coversa4.ucsd.edu,acs-webmail.ucsd.edu,altng.ucsd.edu,aventeur.ucsd.edu,cinfo.ucsd.edu,facilities.ucsd.edu,gradapply.ucsd.edu,graduateapp.ucsd.edu,jacobsstudent.ucsd.edu,myucsdchart.ucsd.edu,sdacs.ucsd.edu,shs.ucsd.edu,ted.ucsd.edu,ucsdbkst.ucsd.edu,a.ucsd.edu,acms.ucsd.edu,bookstore.ucsd.edu,www.bookstore.ucsd.edu,cs.ucsd.edu,www.cs.ucsd.edu,cse.ucsd.edu,www.cse.ucsd.edu,ece.ucsd.edu,www.ece.ucsd.edu,hdh.ucsd.edu,www.hdh.ucsd.edu,hds.ucsd.edu,www.hds.ucsd.edu,maeweb.ucsd.edu,nanoengineering.ucsd.edu,www.nanoengineering.ucsd.edu,ne-web.ucsd.edu,ne.ucsd.edu,neweb.ucsd.edu,roger.ucsd.edu,se.ucsd.edu,structures.ucsd.edu,www.structures.ucsd.edu,uxt.ucsd.edu,www-cs.ucsd.edu,www-cse.ucsd.edu,www-ne.ucsd.edu,www-structures.ucsd.edu,act.ucsd.edu,health.ucsd.edu,libraries.ucsd.edu,studenthealth.ucsd.edu,www-act.ucsd.edu,accesslink.ucsd.edu,acs.ucsd.edu,cri.ucsd.edu,desktop.ucsd.edu,financiallink.ucsd.edu,iwdc.ucsd.edu,marketplace.ucsd.edu,mytritonlink.ucsd.edu,www.mytritonlink.ucsd.edu,resnet.ucsd.edu,software.ucsd.edu,sysstaff.ucsd.edu,tritonlink.ucsd.edu,www.tritonlink.ucsd.edu,uclearning.ucsd.edu,webmail.ucsd.edu,www-acs.ucsd.edu - United-States-Department-of-Energy.xml:
*.doe.govalso coverswww.*.doe.gov - UniversalSubtitles.xml:
*.universalsubtitles.orgalso coverss3.www.universalsubtitles.org - University-of-Alaska.xml:
*.alaska.edualso coverswww.*.alaska.edu,biotech.inbre.alaska.edu,lib.uaa.alaska.edu,*.vpn.alaska.edu - University-of-Bern.xml:
*.unibe.chalso coverswww.*.unibe.ch - University-of-Delaware.xml:
*.udel.edualso covers*.facilities.udel.edu,*.nss.udel.edu - University-of-Groningen.xml:
*.rug.nlalso coverswww.astro.rug.nl - University-of-Idaho.xml:
*.uidaho.edualso coverswww.*.uidaho.edu,www2.sites.uidaho.edu - University-of-Massachusetts-Amherst.xml:
*.umass.edualso covers*.oit.umass.edu,*.spire.umass.edu,*.umii.umass.edu - University-of-South-Florida.xml:
*.usf.edualso covers*.stpete.usf.edu - University-of-Southampton.xml:
*.soton.ac.ukalso coverswww.*.soton.ac.uk - University_of_Houston.xml:
*.uh.edualso coverswww.*.uh.edu,fp.my.uh.edu,*.nsm.uh.edu - University_of_Maine.xml:
*.umaine.edualso coverswww.*.umaine.edu - University_of_Salford.xml:
*.salford.ac.ukalso covers*.www.salford.ac.uk - University_of_Waikato.xml:
*.waikato.ac.nzalso coverstools.its.waikato.ac.nz,www.mngt.waikato.ac.nz - University_of_Wisconsin-Madison.xml:
*.wisc.edualso covers*.library.wisc.edu - uptodown.com.xml:
*.uptodown.comalso coverswww.uptodown.com,api.uptodown.com,blog.uptodown.com,dw.uptodown.com,feeds.uptodown.com,gstatic.uptodown.com,img.uptodown.com,stat.uptodown.com,stc.uptodown.com - US-Dept-of-Veterans-Affairs.xml:
*.vaforvets.va.govalso coverswww.*.vaforvets.va.gov - US-Dept-of-Veterans-Affairs.xml:
*.vba.va.govalso coverswww.*.vba.va.gov - US_State_Department.xml:
*.history.state.govalso coverswww.history.state.gov - UserEcho.com.xml:
*.userecho.comalso coverswww.userecho.com,blog.userecho.com,feedback.userecho.com - Vdopia.xml:
*.vdopia.comalso coversmobile.sb.vdopia.com - Wikidot.xml:
*.wdfiles.comalso covers1.*.wdfiles.com,2.*.wdfiles.com,3.*.wdfiles.com,4.*.wdfiles.com,5.*.wdfiles.com,6.*.wdfiles.com,7.*.wdfiles.com,8.*.wdfiles.com,9.*.wdfiles.com - Wikinvest.xml:
*.wikinvest.comalso covers*.www.wikinvest.com - Wiley.xml:
*.wiley.comalso coversonlinelibrarystatic.wiley.com,sp.onlinelibrary.wiley.com - Wolfram_Alpha.xml:
*.wolframalpha.comalso coversapi.wolframalpha.com,api-cn.wolframalpha.com,api-maps.wolframalpha.com,api-tw.wolframalpha.com,developer.wolframalpha.com,m.wolframalpha.com,preview.wolframalpha.com,products.wolframalpha.com,volunteer.wolframalpha.com,wc.wolframalpha.com,www1.wolframalpha.com,www3.wolframalpha.com,www4b.wolframalpha.com,www4c.wolframalpha.com,www4d.wolframalpha.com,www4f.wolframalpha.com,www5a.wolframalpha.com,www5b.wolframalpha.com - Woot.xml:
*.woot.comalso coversimages.deals.woot.com,gzip.static.woot.com
Activity
RReverser commentedon Sep 1, 2017
Updated to group output, it's less noisy this way.
RReverser commentedon Sep 1, 2017
Not sure if I should attempt to remove these automatically or going manually one by one will be easier...
Bisaloo commentedon Sep 1, 2017
For some of them, conceptually, it kind of makes sense to be that way. For example,
UserEcho.com.xmllists "actual" subdomains first and then the wildcard is used to cover customer subdomains.What I like about it is that the day custom subdomains support is removed, we are just one line away to fix this ruleset, we don't have to find and check every "regular" subdomain again.
A similar alternative would be:
But I have no idea how big an impact this overlapping targets have on performance.
Bisaloo commentedon Sep 1, 2017
That's interesting, looking at cdc90c4, it looks like most of the overlaps are due to the fact that the ruleset author didn't account for the fact that left wildcards affect more than first-level subdomains.
RReverser commentedon Sep 1, 2017
Well, I noticed them as part of hunt for wildcard-in-the-middle regexps, and thought that it makes sense to remove overlaps completely as they're somewhat confusing.
On the other hand, I see your point too, even though 1)
*.somehosthosts are better for both CPU and memory performance than long lists of precise subdomains (especially with reverse-trie approach) and 2) it's future-proof for cases when domain gets new subdomains that also need http: -> https: rewrite and so I'd rather wish to keep these than long lists of specific targets.But I know not everyone in HTTPS Everywhere team agrees with this 😄
Anyway, in cases where they overlap, I think using
<test url />makes more sense if it's just for documentation purposes, although by the time custom subdomains are gone, ruleset will likely have to update list of targets anyway as new subdomains might appear or host might get HSTS etc.RReverser commentedon Sep 1, 2017
Yup, I got the same impression.
cschanaj commentedon Sep 2, 2017
AFAIK, the lookup performance will be improved as the complicated operations in
rules.js#L347-L354can be avoid.I do not know this until recently when I create #11998 looking at rules.js. Maybe we should document this behavior.
RReverser commentedon Sep 2, 2017
It is documented at https://www.eff.org/https-everywhere/rulesets like some other details, but yes, looks like that information is out of sync with contributing guidelines in the repo.
RReverser commentedon Sep 14, 2017
Automated fix was merged, so closing this as resolved.