[WIP] Tests without builtins fixtures by pkch · Pull Request #3189 · python/mypy

pkch · 2017-04-18T11:50:40Z

Based on #3111. I am using this PR to examine the impact of removing builtins fixtures (# of failing tests and performance impact).

pkch · 2017-04-20T12:42:42Z

When env. var. NO_BUILTINS_FIXTURES is present and evaluates to True (using `os.environ['NO_BUILTINS_FIXTURES']), the tests run without fixtures.

On this branch, the tests run without fixtures on Travis / python 3.6, and the usual way in every other configuration.

For no-fixture tests, I temporarily disabled units tests for testsemanal, testtypegen and testtransform because they kept parsing too many modules. In addition, I disabled pytest testfinegrained because it has not implemented enough functionality yet to parse real stdlib modules. The only other tests that fail are under pytest.

When NO_BUILTINS_FIXTURES is enabled, differences in error messages are ignored if the expected and actual messages are on the same lines. In addition, duplicate messages on the same line are removed. The idea is that those things are not important to fix (or at very lower priority).

The current status for pytest (I'll update it as problems are resolved on master): 9 failed, 2415 passed.

I fixed everything that requires just minor edits to tests. The rest requires merging some PR referenced from #3111.

* Env. var. NO_BUILTINS_FIXTURES=1 triggers using regular version for tests * If NO_BUILTINS_FIXTURES=1, minor test error message differences are ignored * Travis under python 3.6 build will run tests without fixtures

ilevkivskyi · 2017-04-25T11:28:29Z

 [builtins fixtures/dict.pyi]

 [case testNoReturnImportFromTyping]
-from typing import NoReturn


This is not right. If it fails, then typeshed should be updated, this is why this test is called testNoReturnImportFromTyping.

But python runtime would crash on that. Why would we want to allow it in mypy?

NoReturn is in typing now.

Oh I'm on python 3.6.0 so didn't realize that haha

BTW, you mean typing.NoReturn is in cpython master right? It's not yet in 3.6.1. I think it's fine for mypy to allow things that are valid on cpython master.

It will be included in 3.6.2

ilevkivskyi · 2017-04-25T11:31:30Z

 [case testTypeCheckNamedModule]
 # cmd: mypy -m m.a
 [file m/__init__.py]
-None + 1


Why do you replace None with object() everywhere? This does not seem right.

I only replaced the cases where the intent is to check whether mypy gets to a given line. The test basically assumes that None + 1 will cause type error. But they don't fail because --strict-optional isn't the default. object() + 1 will fail under any conditions, so I thought I'd use it.

IIRC, some tests are actually run with automatic addition of --strict-optional flag. Why this changes with your PR?

It shouldn't change.

But apart from flags added to tests, and tests inside typeshed, and mypy mypy tests, I can't find tests that have automatic strict-optional. Are you sure it was done to the pytest tests?

Are you sure it was done to the pytest tests?

I am not, but I assumed this, since the tests were passing before, i.e. were giving an error for None + 1.

It was just because in the builtins fixture int.__radd__ isn't defined (only int.__add__). And the subtyping is only checked for the rhs, not for the lhs of these operators.

ilevkivskyi · 2017-04-25T11:32:28Z


 x = foo()
 x = 1
-x = x + 1


What is the reason for this change? This looks strange.

Because of #3122. This test doesn't actually care about that behavior, it simply wanted to see if x + 1 is allowed. But until #3122 is fixed, the side effect is that x changes type in a inconsistent manner without fixtures - making the test fail.

pkch · 2017-04-25T12:01:20Z

The main purpose of this PR is to allow everyone to run no-fixtures tests locally by setting NO_BUILTINS_FIXTURES env. var. I removed the Travis setting that made this happen automatically.

pkch · 2017-04-25T12:14:15Z

After you review, I was going to rebase this off the current master to make it less likely to cause weird merge conflicts.

pkch · 2017-04-27T08:44:34Z

This PR has a lot of old stuff that would make merging it very messy. I'm closing this in favor of #3268. @ilevkivskyi I'm not just rebasing on master within this PR because you were reviewing it, and you'll lose the context if I do so.

pkch force-pushed the nostubs branch 2 times, most recently from 809baf6 to 632e780 Compare April 20, 2017 05:21

Fix crash when looking up _importlib_modulespec names

14a8534

pkch force-pushed the nostubs branch 2 times, most recently from f9a7cfa to 60ba056 Compare April 20, 2017 13:29

pkch mentioned this pull request Apr 20, 2017

Add mypy.test.config.ALLOW_FIXTURES #3112

Closed

pkch added 3 commits April 20, 2017 07:01

Add test without builtins fixtures

16a0ef6

* Env. var. NO_BUILTINS_FIXTURES=1 triggers using regular version for tests * If NO_BUILTINS_FIXTURES=1, minor test error message differences are ignored * Travis under python 3.6 build will run tests without fixtures

Merge branch 'master' into nostubs

25ca546

Merge branch 'fix-incr-crash' into nostubs

afb7aae

pkch force-pushed the nostubs branch from 60ba056 to afb7aae Compare April 20, 2017 14:02

pkch changed the title ~~[WIP] Benchmarking tests without builtins fixtures~~ Tests without builtins fixtures Apr 20, 2017

pkch changed the title ~~Tests without builtins fixtures~~ [WIP] Tests without builtins fixtures Apr 20, 2017

pkch added 2 commits April 20, 2017 07:16

Suppress output of actual errors beyond 25 lines

5364461

Suppress modules from stdlib from polluting mergetest

206eecb

pkch force-pushed the nostubs branch from 79354b9 to 206eecb Compare April 20, 2017 14:28

pkch added 3 commits April 20, 2017 16:42

Merge branch 'master' into nostubs

01cd538

Merge branch 'master' into nostubs

97fdddf

Merge branch 'master' into nostubs

cd32a05

pkch force-pushed the nostubs branch 3 times, most recently from aa46617 to b070899 Compare April 25, 2017 11:22

Make minor fixes to tests to pass without builtins fixtures

5989124

pkch force-pushed the nostubs branch from b070899 to 5989124 Compare April 25, 2017 11:25

ilevkivskyi reviewed Apr 25, 2017

View reviewed changes

pkch added 2 commits April 25, 2017 04:43

Address CR

575c0e8

Don't run no-fixture tests (just leave the option to do so)

00e7b63

pkch changed the title ~~[WIP] Tests without builtins fixtures~~ Tests without builtins fixtures Apr 25, 2017

Merge branch 'master' into nostubs

564aeae

pkch changed the title ~~Tests without builtins fixtures~~ [WIP] Tests without builtins fixtures Apr 27, 2017

pkch mentioned this pull request Apr 27, 2017

Add option to run tests without builtins stub fixtures #3268

Closed

pkch closed this Apr 27, 2017

Conversation

pkch commented Apr 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkch commented Apr 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pkch Apr 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pkch commented Apr 25, 2017

Uh oh!

pkch commented Apr 25, 2017

Uh oh!

pkch commented Apr 27, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pkch commented Apr 18, 2017 •

edited

Loading

pkch commented Apr 20, 2017 •

edited

Loading

pkch Apr 25, 2017 •

edited

Loading