Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Signify that a PR affects a site in the Alexa top 1M #6424

Closed

Description

@Hainish

It would be nice if we had a badge next to PRs (similar to the Travis checkmark) which signifies if a PR affects any domains in the Alexa top X sites. Perhaps different color badges if it's in the Alexa top 1M, 100k, 10k, etc.

This would make it easy to see which PRs are most important for inclusion in an at-a-glace way.

I don't know what the easiest way to do this is, though...

Activity

  1. Hainish commented on Aug 18, 2016

    @Hainish
    MemberAuthor

    Barring that, maybe just utilizing the Github API to create a script which can go over all pending ruleset PRs and list those that are in the Alexa top X

  2. added and removed on Aug 19, 2016
  3. Hainish commented on Aug 20, 2016

    @Hainish
    MemberAuthor

    @J0WI, @fuglede, @jeremyn if you take a look at the list of PRs currently, all the open ones before and including 6476 will have been measured up against the Alexa top 1M sites.

    Based on if any of the hosts in any file modified by the PR is contained in the Alexa top 1M, you may or may not see one of the following labels applied based on their placement within the list:

    1. top-100
    2. top-1k
    3. top-10k
    4. top-100k
    5. top-1m

    This corresponds with how popular the site is. I've also color-coded the labels so that (1) is bright red, and as you move down to (5) the color gets progressively less red. This is to make it easy to see which ruleset PRs are more important than others. My hope is that this will make it easier to determine which PRs to prioritize.

    Currently I'm going over the PRs with a script manually. Next week I'll try to make it an automated process, so that any new PR that is opened will automatically hit the script and have a tag applied if needed.

  4. Hainish commented on Aug 20, 2016

    @Hainish
    MemberAuthor

    Updated to include up to & including #6485

  5. J0WI commented on Aug 20, 2016

    @J0WI
    Contributor

    That's really awesome!

  6. J0WI commented on Aug 20, 2016

    @J0WI
    Contributor

    Not that this depends on the Alexa global rank.
    Some rules might be much more important in a specific country. E.g. news sites often have a global rank > 100K, but are in the top ten of one country.

    Can we also scan rulesets/domains in the issues to prioritize them too?

  7. luixxiul commented on Aug 20, 2016

    @luixxiul
    Contributor

    Can we use this? http://www.alexa.com/topsites/countries

    Actually it is the top 500 sites on the web, so it seems few a bit

  8. Hainish commented on Aug 20, 2016

    @Hainish
    MemberAuthor

    Updated to include up to & including #6504

    I think in terms of prioritization, we should go with the global rank. It doesn't make sense to focus on one particular country more than any other, as this extension is global in focus (it is called HTTPS Everywhere, after all :]). Also, I'm afraid that if we add labels like japan-top100 for a bunch of countries, the labels will get way too cluttered and the utility of seeing clearly which rulesets to prioritize on the list page will be impeded.

    @J0WI I don't think there's any good way to prioritize for the issues. This is because the issues can not be programmatically parsed as the PRs can. If someone mentions that they want their site, https://www.somepersonalblog.com/ to be in the rulesets, then mentions that https://www.google.com/ ranks them highly, there's no good way to figure out programatically which site, google.com or somepersonalblog.com, the issue is actually for.

  9. jeremyn commented on Aug 20, 2016

    @jeremyn
    Contributor

    I think it's neat we have more information about the sites so thanks @Hainish for that. Some comments though:

    I'm uncomfortable triaging based entirely or mostly on top-N rank. A very small site might nevertheless have extreme, potentially life-threatening impact on its users. Also if we use global rank then I'm concerned we'll bias the project (even more) toward users in developed, English-speaking countries. And, especially at the lower levels, it's not clear what I'm supposed to do differently depending on the tag. What extra review should a top-100k site get that a top-1m site doesn't? Without a specific answer, I'm not sure we need to make that distinction.

    Here are some ideas:

    1. Definitely keep top-100, top-1k, and maybe top-10k. We can explore what extra review they should get in Shall we require additional review for top-100 rules? #6492 .
    2. Add country-specific tags. This can help connect volunteers to issues in areas they know about or areas they are interested in focusing on for whatever reason.
    3. Encourage users to use the 'High Priority' tag if their issue or PR is actually high priority. There have only been 17 High Priority marked issues and two High Priority marked pull requests ever.

    What do you all think?

  10. Hainish commented on Aug 20, 2016

    @Hainish
    MemberAuthor

    @jeremyn: You're right - you shouldn't make the decision to prioritize solely on the Alexa rank. The Alexa rank is meant to be a helpful guide for ruleset maintainers, not the sole criteria.

    I don't agree with the assertion that global rank weighs more toward users in developed countries. Just looking at the top 20, the list includes baidu.com, qq.com, google.co.in, taobao.com, google.co.jp, sina.com.cn, weibo.com, and yahoo.co.jp. These sites are all non-English by default, and most of the others in the top 20 are localized based on IP and browser headers. I think global ranking is the best way to have as unbiased a sample as you can get.

    I'm worried about the clutter country-specific tags will create. The high-priority tag is something that HTTPSE collaborators can tag for specific issues if they thing it's important enough, but Github doesn't allow non-collaborators to create issues with a prioritization specified. I think this is a good thing - it prevents anyone from coming along and jumping stack to get their own site included. Users can always tag collabs in order to ask for a prioritization.

  11. jeremyn commented on Aug 20, 2016

    @jeremyn
    Contributor

    This raises a philosophical question about HTTPS Everywhere's mission. Imagine two local governments that each represent a million people, each with a website. One website is in a developed country with 90% internet usage, so 900k possible site users, and the other is in a developing country with 40% internet usage, so 400k possible site users. Which site deserves more attention from HTTPS Everywhere? You could make an argument for the site in the developed country based on the number of internet users, or you could argue that both sites deserve equal attention based on population.

    I'm speculating here but I expect that the global Alexa ranking depends both on the total population a site serves and also the percentage of that population who use the internet. The second number is higher in developed countries so I expect the global ranking to favor developed countries. Anyway I don't have an alternate to the global Alexa rating to suggest, and I think in this case some information about site importance is better than none, so I support the tags. It's worth keeping this in mind though.

    The clutter from country tags would only be one extra tag per PR or issue. It's also hard to say how useful they would be because the people who might really benefit from them aren't well represented in this discussion. They might be super useful to some people. We could add them and see how it goes.

    If GitHub doesn't allow non-collaborators to tag an issue then that's a technical limitation and so be it. But my suspicion is that people are on the whole too shy rather than too aggressive, especially if they're not very technical and if they don't speak English well. I would prefer dealing with 10 aggressive stack-jumpers if it meant we get extra visibility on one more small but critical broken site.

  12. jeremyn commented on Aug 20, 2016

    @jeremyn
    Contributor

    By the way I do recognize I'm new here and I appreciate your listening to my ideas. I hope I'm not coming across as rude.

  13. jeremyn commented on Aug 21, 2016

    @jeremyn
    Contributor

    A more technical problem: if you search for both top-100 and top-1k at the same time, you get no results. Is there a way to get the top 1-1000 in one search?

  14. Hainish commented on Aug 22, 2016

    @Hainish
    MemberAuthor

    Updated to include up to & including #6550

    @jeremyn if you're willing to go through and tag those that apply to a specific country, you're welcome to. I suspect it will be more trouble than it's worth, though. @J0WI, @fuglede what do you think about this idea? If we do implement country labels, the one thing I ask is that it be prefixed. So instead of japan, cc-jp for the Japanese country code. This way it's easier to see all the countries there are labels for when typing cc- into the filter, and if we want to get rid of it later we can see easily which tags to delete.

    @jeremyn you're not being rude, open discussion is important for making building a sustainable project, and I appreciate the input.

    On the technical problem, seems like it's not possible: https://stackoverflow.com/questions/29136057/can-i-search-github-labels-with-logical-operator-or

  15. 10 remaining items

  16. Hainish commented on Aug 25, 2016

    @Hainish
    MemberAuthor

    Updated to include up to & including #6602

  17. J0WI commented on Aug 25, 2016

    @J0WI
    Contributor

    #3985 is now labeled twice for some reason.

  18. Hainish commented on Aug 26, 2016

    @Hainish
    MemberAuthor

    I now have this running as a cron job on a server, running at minute 14 and 44 of every hour.

  19. Hainish commented on Aug 26, 2016

    @Hainish
    MemberAuthor

    It looks like the same PR was gone over with the script twice, and one of the hosts in the PR changed positions within the top million. It shouldn't happen again, the script has stored state that tells it not to go over PRs it has already seen.

    On August 25, 2016 4:25:05 PM PDT, J0WI notifications@github.com wrote:

    #3985 is now labeled twice for some reason.

    You are receiving this because you were mentioned.
    Reply to this email directly or view it on GitHub:
    #6424 (comment)

    Sent from my Android device with K-9 Mail. Please excuse my brevity.

  20. cooperq commented on Aug 26, 2016

    @cooperq
    Contributor

    So, this seems like a desired outcome to me, don't we want to know when the popularity of a site jumps by an order of magnitude on alexa, for better prioritization? (I assume that in this case the site jumped from 1001 to 999 or something similar so not really an order of magnitude, but you get my point).

  21. Hainish commented on Aug 26, 2016

    @Hainish
    MemberAuthor

    Currently we have > 700 PRs open. With the cron job that runs every half hour, it is prudent to only scan the PRs that have been opened since we last checked. Otherwise we come up against API limits and the running time increases dramatically. For registered users, Github gives 3000 API calls/hour, which is pretty low when you consider that for each ruleset PR we have to look at all the ruleset files changed via the API.

    Perhaps every day we can do a more thorough scan for every PR that is open at all, so for older PRs we'll get an idea of if a host popularity has changed.

  22. jeremyn commented on Sep 4, 2016

    @jeremyn
    Contributor

    @Hainish possible bug: pull request #6279 did not get marked top-100k even though a duplicate pull request #6750 did.

  23. reopened this on Sep 4, 2016
  24. Hainish commented on Sep 6, 2016

    @Hainish
    MemberAuthor

    I'm not certain this is a bug. This may accurately reflects the Alexa ranking at the time the script was run. Currently, this site ranks at ~31k. I'll close this, and we could reassess it if we see similar. If we run the script on the full list every week as I suggest in #6424 (comment), this should be tagged regardless, even if we miss it for some reason on the first pass.

  25. jeremyn commented on Sep 6, 2016

    @jeremyn
    Contributor

    Okay, that's fine. I guess I thought that it could matter that the unmarked PR had Nvcc.edu in the description, and the marked one had nvcc.edu. I'm not sure what your script is reading to compare to the Alexa data.

  26. jeremyn commented on Mar 9, 2017

    @jeremyn
    Contributor

    @Hainish Pull request #6771 got tagged with both top-100k and (months later) top-10k. A bug in the tag process?

  27. jeremyn commented on Mar 9, 2017

    @jeremyn
    Contributor

    Pull request #7519 got double-tagged with top-100 and top-1k.

  28. Hainish commented on Mar 9, 2017

    @Hainish
    MemberAuthor

    @jeremyn more probably double-tagged due to targets for that site moving in the Alexa list. I've had to manually delete the placeholder file when the labeller periodically gets stuck.

    The fact that both of these were double-labelled on the same day (14 days ago) indicates to me that it was one of the days I did some manual work on the labeller. Indeed, SSHing in to that server just now gives me Last login: Wed Feb 22 19:52:37 2017 which corresponds to this double-labelling.

  29. cschanaj commented on Jul 23, 2017

    @cschanaj
    Collaborator

    @Hainish rakuten.co.jp from #11265 did not got the top-1k label. I suspect this is a bug caused by the lack of ^ and www in the respective ruleset. Can you confirm this? Related index.js#L104.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions