Skip to content

test_typing leaked [1, 1, 1] memory blocks: fail randomly on Refleak buildbots #121084

Closed
Closed
test_typing leaked [1, 1, 1] memory blocks: fail randomly on Refleak buildbots#121084
Labels
testsTests in the Lib/test dirtopic-typing

Description

@vstinner
vstinner
opened on Jun 27, 2024
Re-running test_typing in verbose mode
test_typing leaked [1, 1, 1] memory blocks, sum=3

Examples of failure:

Linked PRs

Activity

  1. vstinner
    added
    testsTests in the Lib/test dir
    on Jun 27, 2024
  2. JelleZijlstra
    added
    topic-typing
    on Jun 27, 2024
  3. sobolevn
    self-assigned this
    on Jun 27, 2024
  4. sobolevn
    added
    OS-windows
    on Jun 27, 2024

  5. sobolevn commented on Jun 27, 2024

    @sobolevn
    sobolevn
    on Jun 27, 2024
    Member

    It does not reproduce on macos, so it is windows-specific.
    I don't have access to Windows machince, so I cannot help with this :(

  6. sobolevn
    removed their assignment
    on Jun 27, 2024

  7. vstinner commented on Jun 27, 2024

    @vstinner
    vstinner
    on Jun 27, 2024
    MemberAuthor

    ./python -m test test_typing -R 3:3 result is very random. Example of 3 runs on Linux:

    • test_typing leaked [3, 0, 0] memory blocks, sum=3 (this is fine)
    • test_typing leaked [2, 1, 0] memory blocks, sum=3 (this is fine)
    • test_typing leaked [1, 0, 1] memory blocks, sum=2 (this is fine)

  8. vstinner commented on Jun 27, 2024

    @vstinner
    vstinner
    on Jun 27, 2024
    MemberAuthor

    Even if I run more iterations, I only see a total of sum=3. "Leaks" are only in the first 5 runs (after the 3 warmups):

    ./python -m test test_typing -R 3:20:

    • test_typing leaked [2, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=3 (this is fine)
    • test_typing leaked [0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=3 (this is fine)
    • test_typing leaked [1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=3 (this is fine)

  9. vstinner commented on Jun 27, 2024

    @vstinner
    vstinner
    on Jun 27, 2024
    MemberAuthor

    I tried tracemalloc:

    diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py
index 25e71305402..5da93117945 100644
--- a/Lib/test/libregrtest/refleak.py
+++ b/Lib/test/libregrtest/refleak.py
@@ -120,6 +120,7 @@ def get_pooled_int(value):
         numbers = numbers[:warmups] + ':' + numbers[warmups:]
         print(numbers, file=sys.stderr, flush=True)
 
+    import tracemalloc
     xml_filename = 'refleak-xml.tmp'
     result = None
     dash_R_cleanup(fs, ps, pic, zdc, abcs)
@@ -137,6 +138,11 @@ def get_pooled_int(value):
         dash_R_cleanup(fs, ps, pic, zdc, abcs)
         support.gc_collect()
 
+        if i == 2:
+            tracemalloc.take_snapshot().dump('/tmp/snapshot.before')
+        elif i == 3:
+            tracemalloc.take_snapshot().dump('/tmp/snapshot.after')
+
         # Read memory statistics immediately after the garbage collection.
         # Also, readjust the reference counts and alloc blocks by ignoring
         # any strings that might have been interned during test_func. These

    Script:

    import tracemalloc
snapshot1 = tracemalloc.Snapshot.load('/tmp/snapshot.before')
snapshot2 = tracemalloc.Snapshot.load('/tmp/snapshot.after')
top_stats = snapshot2.compare_to(snapshot1, 'lineno')

print("[ Top 10 differences ]")
for stat in top_stats[:10]:
    print(stat)

    Run tests: ./python -X tracemalloc -m test test_typing -R 3:3.

    Example of script output:

    unittest/result.py:170: size=25.0 KiB (-17.9 KiB), count=350 (-327), average=73 B
test/test_typing.py:1169: size=9240 B (+9240 B), count=1 (+1), average=9240 B
collections/__init__.py:510: size=52.6 KiB (-4672 B), count=259 (-2), average=208 B
typing.py:3000: size=5656 B (+4624 B), count=19 (+1), average=298 B
test/test_typing.py:7002: size=0 B (-4624 B), count=0 (-1)
enum.py:559: size=46.1 KiB (+1104 B), count=144 (+1), average=328 B
enum.py:496: size=44.9 KiB (-568 B), count=152 (-1), average=302 B
test/test_typing.py:8101: size=568 B (+568 B), count=1 (+1), average=568 B
test/test_typing.py:5595: size=568 B (+568 B), count=1 (+1), average=568 B
test/test_typing.py:10258: size=0 B (-568 B), count=0 (-1)

    Only growing counts are interesting:

    • test/test_typing.py:1169: count=1 (+1) -- class Old(Generic[*Ts]): ...
    • typing.py:3000: count=19 (+1) -- nm_tpl.__bases__ = bases
    • enum.py:559: count=144 (+1) -- enum_class = super().__new__(metacls, cls, bases, classdict, **kwds)
    • test/test_typing.py:8101: count=1 (+1) -- class Meta(type):
    • test/test_typing.py:5595: count=1 (+1) -- class D(type(ClassVar[int])):

    Many creation of classes. The question is why these classes still seem to be alive after the test completed.

    Maybe something is delaying the deletion of these temporary classes. Maybe they are registered in caches and caches are not fully cleared "immediately" but "later".

  10. JelleZijlstra commented on Jun 27, 2024

    @JelleZijlstra
    JelleZijlstra
    on Jun 27, 2024
    Member

    I recall @encukou telling me it had something to do with materialization of __dict__.

  11. AlexWaygood commented on Jun 27, 2024

    @AlexWaygood
    AlexWaygood
    on Jun 27, 2024
    Member

    I recall @encukou telling me it had something to do with materialization of __dict__.

    see #115822

  12. encukou commented on Jun 27, 2024

    @encukou
    encukou
    on Jun 27, 2024
    Member

    This is a different pattern than the one related to dict rematerialization (#115822)

  13. picnixz commented on Jun 27, 2024

    @picnixz
    picnixz
    on Jun 27, 2024
    Member

    I don't know if its helpful but test.test_typing.ProtocolTests.test_protocols_isinstance_py36 seems to leak quite consistently just after the warm-up. If you write something like:

    for i in {1..5} ; do ./python -m test test_typing -R3:4 -m test_protocols_isinstance_py36 ; done

    You'll see patterns of the form

    # 123:4567
  XX* *...
# or
  XX. *...

    (The test invokes isinstance(X, Y) with Y a Protocol class).

    In addition, we seem to have some pattern:

    Either it is

    beginning 43 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:4567890123456789012345678901234567890123
XX1 2.......................................
test_typing leaked [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=2 (this is fine)

    or it is

    123:4567890123456789012345678901234567890123
XX1 1....1..................................
test_typing leaked [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=2 (this is fine)

    (either you have a 2 at position 4 and nothing afterwards or a 1 at 4 and a 1 at 9).

  14. vstinner commented on Jun 27, 2024

    @vstinner
    vstinner
    on Jun 27, 2024
    MemberAuthor

    GenericAlias and Union have no clear() function, but it doesn't seem to make any difference to add them:

    diff --git a/Objects/genericaliasobject.c b/Objects/genericaliasobject.c
index f5fefd65653..59ef57acc4e 100644
--- a/Objects/genericaliasobject.c
+++ b/Objects/genericaliasobject.c
@@ -50,6 +50,16 @@ ga_traverse(PyObject *self, visitproc visit, void *arg)
     return 0;
 }
 
+static int
+ga_clear(PyObject *self)
+{
+    gaobject *alias = (gaobject *)self;
+    Py_CLEAR(alias->origin);
+    Py_CLEAR(alias->args);
+    Py_CLEAR(alias->parameters);
+    return 0;
+}
+
 static int
 ga_repr_item(PyUnicodeWriter *writer, PyObject *p)
 {
@@ -974,6 +984,7 @@ PyTypeObject Py_GenericAliasType = {
     .tp_getattro = ga_getattro,
     .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_VECTORCALL,
     .tp_traverse = ga_traverse,
+    .tp_clear = ga_clear,
     .tp_richcompare = ga_richcompare,
     .tp_weaklistoffset = offsetof(gaobject, weakreflist),
     .tp_methods = ga_methods,
diff --git a/Objects/unionobject.c b/Objects/unionobject.c
index 7931f4345f7..192ba4ef1ed 100644
--- a/Objects/unionobject.c
+++ b/Objects/unionobject.c
@@ -36,6 +36,15 @@ union_traverse(PyObject *self, visitproc visit, void *arg)
     return 0;
 }
 
+static int
+union_clear(PyObject *self)
+{
+    unionobject *alias = (unionobject *)self;
+    Py_CLEAR(alias->args);
+    Py_CLEAR(alias->parameters);
+    return 0;
+}
+
 static Py_hash_t
 union_hash(PyObject *self)
 {
@@ -380,6 +389,7 @@ PyTypeObject _PyUnion_Type = {
     .tp_free = PyObject_GC_Del,
     .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
     .tp_traverse = union_traverse,
+    .tp_clear = union_clear,
     .tp_hash = union_hash,
     .tp_getattro = union_getattro,
     .tp_members = union_members,

  15. vstinner commented on Jun 27, 2024

    @vstinner
    vstinner
    on Jun 27, 2024
    MemberAuthor

    test_functools has a similar issue: #120163

  16. 23 remaining items

  17. Akasurde
    added a commit that references this issue on Jul 3, 2024

    pythongh-121084: Call _abc_registry_clear() when checking refleaks (p…

    da14728
  18. vstinner
    added 3 commits that reference this issue on Jul 4, 2024

    pythongh-121084: Clear typing ABC caches when checking for refleaks

    0f41e17

    pythongh-121084: Fix test_typing random leaks

    1a30e79

    gh-121084: Fix test_typing random leaks (#121360)

    5f660e8
  19. miss-islington
    added 2 commits that reference this issue on Jul 4, 2024

    pythongh-121084: Fix test_typing random leaks (pythonGH-121360)

    066e648

    pythongh-121084: Fix test_typing random leaks (pythonGH-121360)

    ca77d5a

  20. vstinner commented on Jul 4, 2024

    @vstinner
    vstinner
    on Jul 4, 2024
    MemberAuthor

    test_typing was fixed by 5f660e8.

  21. vstinner
    closed this as completedon Jul 4, 2024
  22. vstinner
    added 2 commits that reference this issue on Jul 4, 2024

    [3.12] gh-121084: Fix test_typing random leaks (GH-121360) (#121372)

    4bf4cc2

    [3.13] gh-121084: Fix test_typing random leaks (GH-121360) (#121373)

    abefbcb

  23. Eclips4 commented on Jul 4, 2024

    @Eclips4
    Eclips4
    on Jul 4, 2024
    Member

    Thanks Victor for solving this. I was a little late to the "party", but I can confirm that it solves the leak at least in every one of my setups (Linux, Windows, MacOS).

  24. noahbkim
    added 2 commits that reference this issue on Jul 11, 2024

    pythongh-121084: Call _abc_registry_clear() when checking refleaks (p…

    7c568e4

    pythongh-121084: Fix test_typing random leaks (python#121360)

    ae52fac
  25. estyxx
    added 2 commits that reference this issue on Jul 17, 2024

    pythongh-121084: Call _abc_registry_clear() when checking refleaks (p…

    fecbd1d

    pythongh-121084: Fix test_typing random leaks (python#121360)

    289b85b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtopic-typing

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions