discuss: Should we alter the behavior of make test? #5607

The behavior was changed and then reverted after objections. I take this to mean that the initial discussion didn't include enough folks.

This is an attempt to document why we might want to make certain changes, and what those options are.

Option 1: Status Quo (Change Nothing)

Pros

• Battle-tested. If it mostly works, why fix it?
• Doesn't move anyone's cheese. People are used to it, let's not mess up their current workflows.

Cons

• Linting runs fast. Tests are slow. With the current set up, you can run tests and wait many minutes before finding out that your tests pass but you have linting errors. At this point, you fix the linting errors, but then you have to run all the tests again. Hope you wanted to go get a second cup of coffee while the tests run a second time.
• If a test fails, you won't be told about linting errors, even if those linting errors might point you at precisely the problem. ("Oh, I didn't mean to have that case block fall through to the next one! Thanks for pointing it out, ESLint!")

Option 2: Lint Before Testing

Pros

• Linting runs fast. So tell me the linting errors now rather than making me wait through the long test run before bothering to lint.

Cons

• This will mess with some people's current workflow and that can be really annoying, especially if you've used the same workflow for years.

Option 3: Keep Testing Before Linting, But Lint Even If Tests Fail

Pros

• If your linting errors are related to the cause of your test failures, it's nice to be told about them.
• If you have a locally flaky test, you still get lint results, rather than only finding out about your lint errors in CI. (Sure, you can run the lint job via make or directly, but we've seen that people don't.)

Cons

• Code to do this in Makefile will be ugly. I suspect it will be portable, but I don't know that and it may be annoying to test across all relevant platforms in advance
• Output may be confusing. If there was a test error or a jslint error, but cpplint reports "No errors found!" then this may confuse newcomers. This is likely fixable, but there are general possible issues around changing output from the job that may only come to light as we use it.
• You still have to wait through the tests to get your lint results.

Refs: #5602
Refs: #5470

/cc @Fishrock123 @bnoordhuis @thealphanerd @evanlucas @jasnell @jbergstroem