Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

Implement issue labeller bot - #11615

Merged
Hainish merged 96 commits into
EFForg:masterfrom
strugee:issue-format-bot
Sep 19, 2017
Merged

Hainish merged 96 commits into
EFForg:masterfrom
strugee:issue-format-bot

Conversation

@strugee

@strugee strugee commented Aug 1, 2017 •

Copy link
Copy Markdown
Contributor

Reached a point where the bot's basic functionality works pretty well, so I thought I'd open a WIP PR for review.

Note that if you can think of any other weird input that users might give, I'd love to expand the test suite with that type of stuff.

Still TODO:

  • Manually test the editing logic on GitHub
  • Implement Alexa labeling
  • Hook into Travis CI
  • Linting
  • Style checking
  • Let users add stuff that's not key-value
  • Expand tests (see in-source TODOs)
  • Check for users doing weird stuff with the domain, like adding paths or something
  • Extract Alexa loading/mocks into a separate npm module since that's actually pretty useful
  • Label for edits as well as creates
  • File headers
  • Don't process Markdown links (which have :s in them)
  • Try to handle cases where the user's free-form text has :s in it
  • Upgrade Probot
  • Better deployment docs
  • Allow "other" as an issue type
  • Trim ESLint config
  • Support multiple domains
  • Check for more than one type specified
  • Refresh Alexa data daily
  • Remove all top-X labels before relabeling

Ref #6322

strugee added 28 commits June 19, 2017 02:21
This template exactly matches what the bot will parse.
Spoiler: it can't as of this commit. But it should.
This makes it much easier to test.
I was just being dumb. Obviously we want the functions _inside_ the
returned context to be Sinon stubs, not the factory function itself.
@strugee

strugee commented Aug 1, 2017

Copy link
Copy Markdown
Contributor Author

Introduced ESLint to do linting/style checking - it's possible it could be a little better, but I think it's good for now.

@strugee
strugee deleted the issue-format-bot branch September 21, 2017 13:58
@strugee

strugee commented Sep 21, 2017

Copy link
Copy Markdown
Contributor Author

Ah yeah, sorry @Hainish - upstream docs are a little scarce but it looks like if you set NODE_ENV=production it won't try to do any of the tunneling stuff. See https://github.com/probot/probot/blob/master/bin/probot-run.js

@Hainish

Hainish commented Sep 21, 2017

Copy link
Copy Markdown
Member

Looks like it has crashed again. Thanks for the tip @strugee - I've stopped the bot and removed the issue template for the time being. I'll re-enable it once we resolve a public EFF subdomain to the bot and start running it that way.

@strugee

strugee commented Sep 21, 2017

Copy link
Copy Markdown
Contributor Author

OK. Keep me posted.

@Hainish

Hainish commented Sep 27, 2017

Copy link
Copy Markdown
Member

Waiting on our internal techops team to do some server management before running bot again.

@Hainish

Hainish commented Sep 28, 2017

Copy link
Copy Markdown
Member

The issue labeller bot is re-activated, and I've incorporated a (slightly) modified version of @jeremyn's suggested ISSUE_LABELLER.md text.

@Hainish

Hainish commented Sep 28, 2017

Copy link
Copy Markdown
Member

I just changed the issue template to use a different style for comments, which I think looks much better.

@jeremyn

jeremyn commented Oct 5, 2017 •

Copy link
Copy Markdown
Contributor

The bot's behavior in issue #12956 isn't great, with four requests for clarification in one day. I think the bot should ask for clarification at most one time.

@strugee

strugee commented Oct 6, 2017

Copy link
Copy Markdown
Contributor Author

So sorry all - I've been swamped with midterm papers due this week. I'll try to fix the bug as soon as I get the time.

@strugee strugee mentioned this pull request Nov 2, 2017
@Hainish

Hainish commented Jan 26, 2018

Copy link
Copy Markdown
Member

The bot seems to have crashed - here's the log:

  Error: No X-Hub-Signature found on request
      at hasError (/opt/https-everywhere/utils/issue-format-bot/node_modules/github-webhook-handler/github-webhook-handler.js:46:17)
      at handler (/opt/https-everywhere/utils/issue-format-bot/node_modules/github-webhook-handler/github-webhook-handler.js:57:14)
      at Server.http.createServer (/opt/https-everywhere/utils/issue-format-bot/node_modules/probot/lib/server.js:5:5)
      at emitTwo (events.js:125:13)
      at Server.emit (events.js:213:7)
      at parserOnIncoming (_http_server.js:602:12)
      at HTTPParser.parserOnHeadersComplete (_http_common.js:116:23)
Error: No X-Hub-Signature found on request
    at hasError (/opt/https-everywhere/utils/issue-format-bot/node_modules/github-webhook-handler/github-webhook-handler.js:46:17)
    at handler (/opt/https-everywhere/utils/issue-format-bot/node_modules/github-webhook-handler/github-webhook-handler.js:57:14)
    at Server.http.createServer (/opt/https-everywhere/utils/issue-format-bot/node_modules/probot/lib/server.js:5:5)
    at emitTwo (events.js:125:13)
    at Server.emit (events.js:213:7)
    at parserOnIncoming (_http_server.js:602:12)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:116:23)
07:50:07.370Z DEBUG PRobot: Refreshed Alexa rankings.
npm info lifecycle issue-format-bot@1.0.0~poststart: issue-format-bot@1.0.0
npm info ok 

@Hainish

Hainish commented Jan 26, 2018

Copy link
Copy Markdown
Member

I've restarted the service for now. @jeremyn has noted that the bot has had limited utility. I can shut it down again or we can try to improve it. I'm interested in community input on this.

@jeremyn

jeremyn commented Jan 26, 2018

Copy link
Copy Markdown
Contributor

The bot would be helpful as part of a larger automated process, for example in migrating simple "add this domain" issues to the wiki or some collection issue or something else, like we discuss in issue #13890.

The Alexa labeling is helpful. I hope the current bot is an improvement over @Hainish 's previous process.

The required Type:/Domain: categorization has not been very useful so far.

The bot only processes new issues, but we have a huge issue backlog, which means that even the best-performing bot will only process a small percentage of open issues. Probably more extensive bot actions, if we want to have them, should wait until the backlog has been reduced.

I would be okay with reducing the bot's behavior to the Alexa labeling for now.

@Hainish

Hainish commented Jan 26, 2018

Copy link
Copy Markdown
Member

Pinging @strugee for input as well, since he reached out to me recently to check up on the bot and how it has worked out.

@Hainish

Hainish commented Jan 26, 2018

Copy link
Copy Markdown
Member

@jeremyn my previous process for issues was nonexistent. For PRs, the labeller is still behaving as it was before the creation of the issue labeller bot. Unlike the issues bot, it did dig through the PR backlog, but this is because it was easy to do so: you can simply parse the PR to see what its targets are. For issues, things are a little more complicated. Without the issue template, it's hard for a bot to parse common vernacular for what is actually being requested.

@jeremyn

jeremyn commented Jan 26, 2018

Copy link
Copy Markdown
Contributor

I should add that I appreciate @strugee 's willingness to contribute. Also, while the bot has had some technical problems, that is to be expected with new software and I don't mean to criticize @strugee 's skill as a programmer. My complaints (if I am complaining) are related to whether the bot is a good fit for the project, or whether we are putting it to good use.

@jeremyn

jeremyn commented Jan 26, 2018

Copy link
Copy Markdown
Contributor

@Hainish In that case it would be nice to merge the issue bot and the Alexa bot together so there is less code hanging around. We might re-envision the issue bot as a broader "GitHub bot" and in that way find more stuff to do with it.

@Hainish

Hainish commented Jan 26, 2018

Copy link
Copy Markdown
Member

Thanks @jeremyn - that's probably a good to-do item.

@strugee

strugee commented Jan 26, 2018

Copy link
Copy Markdown
Contributor Author

@Hainish that stack trace is being thrown in a Probot dependency, not our custom code. Apparently someone has POSTed to the API endpoint and the code is complaining because the code doesn't contain the usual signature from GitHub. The fact that it's throwing instead of just dropping the request is a bug in Probot (the bot framework).

We should try just upgrading Probot and see if it's still reproducible (assuming of course that my theory is correct and it is reproducible). Otherwise someone could file a bug or PR upstream.

@strugee

strugee commented Jan 26, 2018

Copy link
Copy Markdown
Contributor Author

As for the usefulness, I agree it doesn't do as much as it could. The original thinking behind the Type: categorization was to allow separating different types of issues with labels or something - in particular I noticed that code issues tend to get lost among ruleset bugs or ruleset suggestions. However I never got around to actually bringing that up and trying to find consensus on whether community members agreed with that assessment or not, so Type: right now is kinda pointless.

@jeremyn what do you mean reducing the functionality to Alexa labeling? AFAIK that's basically all it does right now. I mean it says some other stuff but most of that is handling user error. Also I too have always thought it felt kinda weird to have a separate bot for PRs. (Also I appreciate you saying the bit about not criticizing me, but absolutely don't worry! I didn't take it that way at all 👍)

Here's what I would suggest for some broad TODOs, in no particular order:

  • Upgrade Probot and see if that fixes the crash we've been discussing
  • Merge the PR labeler into the issue format bot
  • Create a mechanism for processing the issue backlog - something where you could ping the bot on an older issue to disable the date cutoff functionality so the bot would be "activated" on that issue (we could write a one-off script to label "unmanaged" issues too, to make it easier to do this processing)
  • Start a discussion about possibly making better use of Type: information, or dropping the Type: field if that ends up not being something people want

I also wonder if @brainwane has any thoughts on this since I believe she's been looking into similar workflow questions?

@jeremyn

jeremyn commented Jan 26, 2018

Copy link
Copy Markdown
Contributor

@strugee Building consensus and then acting on it is a serious challenge for this project. Probably the bot shouldn't do anything unless it is directly supporting something that already has overwhelming support from the community, or unless the EFF has some firm vision it's willing to support. Implementing something that only might be used (Type:) is where things fall apart.

My earlier comment was confused, I thought that the issue bot was also handling Alexa labeling for PRs.

Handling old issues is a problem and not one that a bot can solve. I opened issue #13890 in December to talk about it but it hasn't really gone anywhere (see above about building consensus).

I don't think anyone thinks the required fields are very useful right now. I would disable them entirely.

@strugee

strugee commented Jan 26, 2018

Copy link
Copy Markdown
Contributor Author

@jeremyn well, Type: already exists. Depending on how much it's getting in the way (i.e. how much people are annoyed by it) I think we could make a decision and either start using it or drop it, and just see what people think. Adding extra labels to issues doesn't seem like a big enough change to worry about.

My earlier comment was confused, I thought that the issue bot was also handling Alexa labeling for PRs.

Gotcha, no worries.

Handling old issues is a problem and not one that a bot can solve. I opened issue #13890 in December to talk about it but it hasn't really gone anywhere (see above about building consensus).

Well, the bot could probably help. At least then we'd get the Alexa labeling for old issues which would help with triage. Some things you mention too could help, e.g.:

Another thing to do is to improve the workflow around the low effort "please add example.com, thanks!" type issues. These are good to have but they can add a lot of clutter to the issues page when domains are reported one at a time.

This could be helped by using Type: information to label issues to give the issue tracker better organization. You mention this further down in the second comment.

I don't think anyone thinks the required fields are very useful right now. I would disable them entirely.

Type: can be argued but if we want to keep the Alexa labeling for issues, Domain: is a hard requirement. It's simply not possible to extract domain info from reports unless it's structured.

@wonderchook

Copy link
Copy Markdown
Contributor

@brainwane and I will be drafting some specs for tool and workflow improvements. We'll be sure to review this conversation and summarize into those specs, then sharing for comment.

@jeremyn

jeremyn commented Jan 26, 2018

Copy link
Copy Markdown
Contributor

@strugee The bot could help with old issues, but probably won't, unless people agree on what it could do to help and the bot then does those things. The agreeing step is the problem.

@strugee

strugee commented Jan 27, 2018

Copy link
Copy Markdown
Contributor Author

The bot could help with old issues, but probably won't, unless people agree on what it could do to help

I mean, I was assuming it'd do the same thing it does for new issues...? That is, give Alexa labels and (possibly) label based on Type:?

@jeremyn

jeremyn commented Jan 27, 2018

Copy link
Copy Markdown
Contributor

For Alexa labelling the old issues, it probably makes more sense for someone to go through the list manually.

@Hainish Hainish mentioned this pull request Jan 30, 2018
@strugee strugee mentioned this pull request Feb 13, 2018
5 of 16 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants