hatch swallows test failures ?

It seem that hatch (or something else?) jus ignore some failure.

see this run of current main branch:

https://github.com/ipython/ipykernel/actions/runs/13286863326/job/37097508006 which is aborted,
and this comment:

That point to a run with actual failures that appear as passing.

It's because we rerun last failed tests here.

That seems wrong, it can hide test that modify global state and make subsequent test to fail.
Plus the abort I linked to does not seem to be reran.

Plus the abort I linked to does not seem to be reran.

You mean this? https://github.com/ipython/ipykernel/actions/runs/13286863326/job/37097508006
Yes it is rerun (look for --lf).

No, your confusing the two run because hatch run tests twice .qt = ["qt5", "qt6"]

The first run abort at 14%, Fatal Python error: Aborted, pytest does not write it's json file with failing test (there are no failed test, and the ones after test_qt_enable_gui are not run, so the second --lf does nothing.

Compare with https://github.com/ipython/ipykernel/actions/runs/13324822814/job/37215765326?pr=1324, where you get two complete full runs.

Or a more explicit question/prof to show you that test are not properly ran:

Show me where tests/test_heartbeat.py::test_port_bind_failure_raises is ran under cov.qt5 ?,

It is ran neither by the hatch first invocation (it would be ran after the Abort),
It is not ran under --lf that reruns nothing.

Same for tests/test_eventloop.py::test_cocoa_loop, which is the crashing test on cov.qt5:

where is it reran ?

Show me where tests/test_heartbeat.py::test_port_bind_failure_raises is ran under cov.qt5 ?

Here?

Here?

And no, again this is not under cov.qt5, it's under test, qt is not available !

I also meant q6 insted of 5 as well but it does not change much

None of the qt6 test are run.

This is current report on main branch:

2025-02-12T13:48:26.3781030Z ##[group]Run hatch run cov:test --cov-fail-under 50 || hatch run test:test --lf


2025-02-12T13:48:26.3918128Z + hatch run cov:test --cov-fail-under 50
2025-02-12T13:48:27.3793726Z ─────────────────────────────────── cov.qt5 ────────────────────────────────────
2025-02-12T13:48:27.3796015Z Creating environment: cov.qt5

...  
2025-02-12T13:48:57.2991349Z tests/test_async.py::test_async_await �[32mPASSED�[0m�[33m                             [  0%]�[0m
...
2025-02-12T13:49:34.6306217Z tests/test_eventloop.py::test_enable_gui �[32mPASSED�[0m�[33m                          [ 13%]�[0m
2025-02-12T13:49:34.6322729Z tests/test_eventloop.py::test_cocoa_loop �[33mSKIPPED�[0m (MacOS-only)�[33m            [ 14%]�[0m
2025-02-12T13:49:36.3876305Z Fatal Python error: Aborted
...

^ Here we aborted, all of the test after this won't be marked as failed, all of the qt6 test will not run.

2025-02-12T13:49:38.2130862Z + hatch run test:test --lf

## rerun all the test under Creating environment: test
## You can see the pyqt test are skipped :


2025-02-12T13:50:15.2029784Z tests/test_eventloop.py::test_qt_enable_gui �[33mSKIPPED�[0m (No viable version of PyQt or PySide installed.)�[33m                                            [ 14%]�[0m

^ See SKIPPED No viable version of PyQt or PySide installed

But

If you remove the --lf the failure is not hidden ! This is an actual failure, and then you mark the aborting test as skipping you run test that otherwise are not ran:

In PR #1324:

2025-02-14T07:49:44.9124248Z ─────────────────────────────────── cov.qt5 ────────────────────────────────────
2025-02-14T07:49:44.9126900Z Creating environment: cov.qt5

...Ok same as main with no failure I'm not going to rewrite this but this is with qt5
So everything after tests/test_eventloop.py::test_enable_gui is actually passing with qt5

All of QT6 is is missing from main as well but present in #1324

2025-02-14T07:52:19.1069595Z ─────────────────────────────────── cov.qt6 ────────────────────────────────────
2025-02-14T07:52:19.1071179Z Creating environment: cov.qt6

^ This is not created or ran on main !

...
2025-02-14T07:53:22.0112802Z tests/test_eventloop.py::test_asyncio_loop �[32mPASSED�[0m�[33m                        [ 13%]�[0m
2025-02-14T07:53:23.7892640Z tests/test_eventloop.py::test_enable_gui �[32mPASSED�[0m�[33m                          [ 14%]�[0m

^ This is not ran with qt6 !

Again I'm asking is it reran under qt5 or qt6 not is it reran under test !

The --lf line is

hatch run cov:test --cov-fail-under 50 || hatch run test:test --lf

not

hatch run cov:test --cov-fail-under 50 || hatch run cov:test --lf

and the test env does not have qt, only the cov env has:

# pyproject.toml
[tool.hatch.envs.test]
features = ["test"]
[tool.hatch.envs.cov]
features = ["test", "cov"]

...

[[tool.hatch.envs.cov.matrix]]
qt = ["qt5", "qt6"]

[tool.hatch.envs.cov.overrides]
matrix.qt.features = [
  { value = "pyqt5", if = ["qt5"] },
  { value = "pyside6", if = ["qt6"] },
]

Now you can argue, , yes but for most tests whether qt is installed or not does not matter, well maybe, but for some it does, for example tests/test_eventloop.py::test_enable_gui, and tests/test_eventloop.py::test_enable_gui failure under qt5 is swallowed and nothing get ran under qt6 because tests/test_eventloop.py::test_enable_gui['qt5'] crashes, and because of || --lf

One more argument that ||--lf is swallowing errors

In #1234 I actually marked some qt5 test as @skip as they were segfaulting, and now we see previously unknown failure on python39 and qt6 !

Navigation Menu

Repository navigation

hatch swallows test failures ? #1323

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

hatch swallows test failures ? #1323

Description

Activity

davidbrochart commented on Feb 13, 2025

Carreau commented on Feb 13, 2025

davidbrochart commented on Feb 13, 2025

Carreau commented on Feb 14, 2025

Carreau commented on Feb 14, 2025

davidbrochart commented on Feb 14, 2025

Carreau commented on Feb 14, 2025

Carreau commented on Feb 19, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions