Skip to content

argparse: mutually exclusive groups full of help-suppressed args can cause AssertionErrors #62090

Closed
Closed
argparse: mutually exclusive groups full of help-suppressed args can cause AssertionErrors#62090
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@gholms
gholms
mannequin
opened on May 2, 2013
BPO 17890
Nosy @terryjreedy, @bitdancer, @vadmium, @iritkatriel
Files
  • argparse-assertfail.py: Test case
  • 7691d1d4b955.diff: gholms repo
  • Issue17890-27.patch: Resolution for Python 2.7 issue17890
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2013-05-02.00:43:02.387>
labels = ['type-bug', 'library']
title = 'argparse: mutually exclusive groups full of help-suppressed args can cause AssertionErrors'
updated_at = <Date 2021-12-10.16:06:27.619>
user = 'https://bugs.python.org/gholms'

    bugs.python.org fields:

    activity = <Date 2021-12-10.16:06:27.619>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-05-02.00:43:02.387>
creator = 'gholms'
dependencies = []
files = ['30102', '30108', '30110']
hgrepos = ['187']
issue_num = 17890
keywords = ['patch']
message_count = 17.0
messages = ['188250', '188253', '188278', '188291', '188295', '188306', '188337', '188379', '188413', '188984', '189054', '189064', '189065', '189355', '193191', '259189', '408217']
nosy_count = 8.0
nosy_names = ['terry.reedy', 'bethard', 'r.david.murray', 'martin.panter', 'paul.j3', 'Yogesh.Chaudhari', 'gholms', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'pending'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17890'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

    Linked PRs

    Activity

    1. gholms commented on May 2, 2013

      @gholms
      gholmsmannequin
      on May 2, 2013
      MannequinAuthor

      When it goes to format a usage message, argparse seems to (correctly) fail to satisfy one of its assertions when all of the following are true:

      1. A mutually exclusive group contains only args that are suppressed
      2. An unsuppressed arg follows that group
      3. The usage is long enough to need to line-wrap

      The cause seems to be that the set of regular expressions that argparse uses to clean up mutually exclusive groups' separators doesn't handle the space that follows what would otherwise be an empty pair of square braces, sort of like this:

      1. [-h] [ ] [--spam] ...
      2. [-h] [] [--spam] ...
      3. [-h] [--spam] ...

      A test case is attached. I was able to reproduce this with python-2.7.3-13.fc18.x86_64 on Fedora as well as with commit 83588:e6b962fa44bb in 3.4 mainline. I have a small patch for the latter that I'll submit shortly.

      Sorry if I missed anything. This is my first bug report against python proper.

    2. gholms
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on May 2, 2013

    3. paulj3 commented on May 2, 2013

      @paulj3
      paulj3mannequin
      on May 2, 2013
      Mannequin

      Looks like the

      text = text.strip()

      at the end of the set of regex (in _format_actions_usage) needs to be replaced with something that removes all excess spaces, e.g.

      text = _re.sub( '\s+', ' ', text ).strip()

    4. YogeshChaudhari commented on May 2, 2013

      @YogeshChaudhari
      YogeshChaudharimannequin
      on May 2, 2013
      Mannequin

      Made similar required changes for version 2.7

    5. paulj3 commented on May 3, 2013

      @paulj3
      paulj3mannequin
      on May 3, 2013
      Mannequin

      In the test case: class TestMutuallyExclusiveManySuppressed
      even with a short 'eggs' argument, there is a difference

      Old usage would be:

      usage: PROG [-h] [--eggs EGGS]

      new

      usage: PROG [-h] [--eggs EGGS]

      i.e. 2 v 1 space. But extra spaces are not as dramatic a failure as an assertion error.

      It would also be good to check what happens when there are 2 suppressed groups. If the text before all trimming is:

      [ -h ] [] () [ --eggs EGGS ]

      does it reduce to?

      [-h] [--eggs EGGS]

      The old code would have left 3 spaces.

      I can't think of a situation in which a user would want a (generated) usage line with multiple spaces. If some sort of special formatting is needed, there is always the option of specifying an explicit usage line.

      parser = ArugmentParser(usage='one \ttwo  \nthree   four')

    6. YogeshChaudhari commented on May 3, 2013

      @YogeshChaudhari
      YogeshChaudharimannequin
      on May 3, 2013
      Mannequin

      A user generated line with multiple spaces may be essential if we are getting the usage data (particularly from) a csv file (that may have random spaces in certain fields) or some other form of stored or auto-generated data.

    7. paulj3 commented on May 3, 2013

      @paulj3
      paulj3mannequin
      on May 3, 2013
      Mannequin

      I see three solutions -

      1. gholms' patch which removes '() ' and [] '

      2. Yogesh's patch which removes all duplicated spaces.

      3. remove the 2 asserts.

      The first 2 do the same thing most of the time, but may differ if the user somehow inserts spaces into names. The third leaves the extra blanks, but renders them innocuous. I doubt if the asserts were written to catch this problem. They probably were included to verify the usage line had been split up as expected prior to reassembling on multiple lines.

      As best I can tell test_argparse.py does not test for these spaces. Curiously though a port of argparse to javascript does have a test case with the extra space.

    8. terryjreedy commented on May 3, 2013

      @terryjreedy
      terryjreedy
      on May 3, 2013
      Member

      Garrett and Yogesh, please submit contributor license agreements
      http://www.python.org/psf/contrib/contrib-form/
      if you have not yet. When one is properly recorded, a * appears after your name.

    9. YogeshChaudhari commented on May 4, 2013

      @YogeshChaudhari
      YogeshChaudharimannequin
      on May 4, 2013
      Mannequin

      I am not sure if I am missing something. I had filled out the form at http://www.python.org/psf/contrib/contrib-form/ on the day I submitted the patch and even got back an email from Ewa Jodlowska. However, I don't see any "*" after my name. I submitted the same form to contributors@python.org and shot a mail to python-dev mailing list but there is no change. Any suggestions?

    10. terryjreedy commented on May 4, 2013

      @terryjreedy
      terryjreedy
      on May 4, 2013
      Member

      Wait a week and see what happens.

    11. YogeshChaudhari commented on May 12, 2013

      @YogeshChaudhari
      YogeshChaudharimannequin
      on May 12, 2013
      Mannequin

      @terry: Thanks for the info. I seem to have the elusive "*" after my username now. I am not sure how this works, but can you review/test/apply the patch now?

    12. terryjreedy commented on May 12, 2013

      @terryjreedy
      terryjreedy
      on May 12, 2013
      Member

      Argparse is out of my area of competence/experience. I have added a couple of people who have worked on argparse in the past and should be better able to review or suggest another reviewer.

    13. paulj3 commented on May 12, 2013

      @paulj3
      paulj3mannequin
      on May 12, 2013
      Mannequin

      I'm following a dozen argparse issues with patches. I haven't seen much posting by argparse experts (like bethard, david.murry) since last December.

    14. YogeshChaudhari commented on May 12, 2013

      @YogeshChaudhari
      YogeshChaudharimannequin
      on May 12, 2013
      Mannequin

      @terry and @paul:
      I have 2 argparse related patches, sitting idle for quiet sometime now. Can you suggest any more names who can take this forward? (I could not find anyone else related to argparse in http://docs.python.org/devguide/experts.html)

    15. bitdancer commented on May 16, 2013

      @bitdancer
      bitdancer
      on May 16, 2013
      Member

      I've been observing the activity on the argparse issues and am appreciating the work, but I don't have time right now to review the patches. I should have more time next month, and expect to get to them then, if no one else gets to them before I do.

    16. paulj3 commented on Jul 16, 2013

      @paulj3
      paulj3mannequin
      on Jul 16, 2013
      Mannequin

      I just submitted a patch to http://bugs.python.org/issue11874 that substantially rewrites _format_actions_usage(). It generates the group and action parts separately, and does not do the kind of cleanup that produces this issue.

    17. vadmium commented on Jan 29, 2016

      @vadmium
      vadmium
      on Jan 29, 2016
      Member

      See duplicate bpo-22363 for the traceback, and a workaround. There is also a patch there similar to Garrett’s original fix, but using the RE r'%s *%s ?'.

      I restored the diff from Garrett’s repository, in case it is still useful.

      Yogesh: I am not familiar with how the code works, and I struggle to be excited at using regular expressions to build a usage message :). But can you explain why you changed Garrett’s original fix? Does it affect intentional double spaces (or tabs, non-ASCII spaces, etc) in customizable parts of the usage message?

    18. vadmium
      changed the title [-]argparse: mutually exclusive groups full of suppressed args can cause AssertionErrors[/-] [+]argparse: mutually exclusive groups full of help-suppressed args can cause AssertionErrors[/+] on Jan 29, 2016

    19. iritkatriel commented on Dec 10, 2021

      @iritkatriel
      iritkatriel
      on Dec 10, 2021
      Member

      I'm unable to reproduce this problem on 3.11:

      >>> import argparse
>>> parser = argparse.ArgumentParser()
>>> group = parser.add_mutually_exclusive_group()
>>> group.add_argument('--spam', help=argparse.SUPPRESS)
_StoreAction(option_strings=['--spam'], dest='spam', nargs=None, const=None, default=None, type=None, choices=None, help='==SUPPRESS==', metavar=None)
>>> parser.add_argument('--' + 'eggs' * 20, dest='eggs')
_StoreAction(option_strings=['--eggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggs'], dest='eggs', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
>>> parser.print_usage()
usage: [-h]  [--eggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggs EGGS]
>>>
    20. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    21. erlend-aasland
      added this to Argparse issueson May 19, 2022
    22. erlend-aasland
      moved this to Bugs in Argparse issueson May 19, 2022
    23. hamdanal
      added a commit that references this issue on May 28, 2023

      Simplify argparse usage formatting

      4336ac7
    24. encukou
      added a commit that references this issue on May 7, 2024

      gh-62090: Simplify argparse usage formatting (GH-105039)

      de1428f

    25. encukou commented on May 7, 2024

      @encukou
      encukou
      on May 7, 2024
      Member

      The reproducer needs a narrow enough console, or a bigger number for the 'eggs' * 20.

      This was fixed in #96311.

    26. encukou
      closed this as completedon May 7, 2024
    27. github-project-automation
      moved this from Bugs to Doc issues in Argparse issueson May 7, 2024
    28. SonicField
      added a commit that references this issue on May 8, 2024

      pythongh-62090: Simplify argparse usage formatting (pythonGH-105039)

      8047c0b
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions