test_dis should test the dis module, not everything else #90916

Activity

1. markshannon commented on Feb 15, 2022

   

   markshannon

   on Feb 15, 2022

   Last edited by terryjreedy

   MemberAuthor

   More actions

   <rant>

   This is getting really annoying.
   It takes longer to fix all the heavily coupled and poorly written tests in test_dis than to make the real changes.

   Tiny changes in the calling sequence, or reordering CFGs, cause huge diffs in the test_dis module.
   No one ever checks these changes, they are just noise.

   I've put this under "enhancement" as there is no "wastes a huge amount of time" category.

   </rant>

   The test_dis should not:

   • Contain offsets; they turn one line diffs into 100 line diffs.
   • Contain tests for the compiler; they belong elsewhere.
   • Contain big strings; write proper tests not just regex matches.
   • Tests for Instruction should should not depend on the compiler output; create the bytecode directly.

   This is not a new problem, but it does seem to be getting progressively worse.

   A lot of the irritation stems from
   b39fd0c
   although the tests from before that still hardcode offsets.
2. markshannon

   added

   testsTests in the Lib/test dirTests in the Lib/test dir

   type-featureA feature request or enhancementA feature request or enhancement

   on Feb 15, 2022
3. terryjreedy commented on Feb 18, 2022

   

   terryjreedy

   on Feb 18, 2022

   Member

   More actions

   With IDLE, I have issues with trying to test IDLE code without retesting tkinter, as well as deciding on the proper units of behavior to test.

   Some suggestions:

   1. Add a docstring to the module with guidelines, after review from a couple of others.
      For instance,I believe you are saying that no test should explicitly call compile. Rather the test writer should call compile and extract the bytecode to copy into a test.

   2. Make yourself a module code owner so you get informed and can review. Reviewing would be easier with guidelines to refer to, instead of repeating them.

   3. An example issue is that some callables take many types of arguments. If not already, tests that the function can extract the bytecode from various objects should be separate from tests that extracted bytecode is then handled properly. Would any internal refactoring of dis and addition of _test functions make it easier to make test more independent of each other?

   4. I would rather read the multiple long lists like

   expected_opinfo_outer = [
     Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=3, argrepr='3', offset=0, starts_line=2, is_jump_target=False),
     Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, argrepr='4', offset=3, starts_line=None, is_jump_target=False),
     ...
   ]

   condensed to

   expected_opinfo_outer = [Instruction(opname, opcode, arg, argval, argrepr,
                                        offset, starts_line, is_jumps_target)
       for opname, opcode, arg, argval, argrepr, offset, starts_line, is_jumps_target in
   (('LOAD_CONST', 100,    1,   3,     '3',      0,      2,           False),
    ('LOAD_CONST', 100,    2,   4,     '4',      3,      None,        False),
    ...
   )]
4. ezio-melotti

   transferred this issue fromon Apr 10, 2022
5. martindemello commented on Aug 31, 2022

   

   martindemello

   on Aug 31, 2022

   Contributor

   More actions

   i'm interested in working on this (and refactoring dis to support better testability if needed). i like @terryjreedy 's ideas in general so i can start by implementing some of them.
6. iritkatriel commented on Aug 31, 2022

   

   iritkatriel

   on Aug 31, 2022

   Member

   More actions

   I think the most complex part of this issue is to make sure that the things that are covered by test_dis (and which do not belong there) are covered by other tests. Only then we can reduce test_dis to the right scope.

   👍React with 👍1Reacted by Martin DeMello
7. terryjreedy commented on Sep 1, 2022

   

   terryjreedy

   on Sep 1, 2022

   Last edited by terryjreedy

   Member

   More actions

   I added checkboxes to Mark's todo list in the initial post. The first item is already fixed.
8. martindemello commented on Sep 1, 2022

   

   martindemello

   on Sep 1, 2022

   Contributor

   More actions

   there are still offsets in the Instruction(...) lists; I'll get rid of them in the next PR, so that this one can remain a pure refactoring.
9. martindemello commented on Sep 1, 2022

   

   martindemello

   on Sep 1, 2022

   Contributor

   More actions

   by this:

   Contain big strings; write proper tests not just regex matches.

   do you mean that we should always be matching the structured output of _get_instruction_bytes and not the formatted string output of _disassemble_bytes?
10. markshannon

    added a commit that references this issue on Mar 4, 2025

    GH-90916: Add helper function to dis.Instruction and use it to reduce…

    ...

    885c3d1