Skip to content

3.11 exec raises SystemError instead of SyntaxError on char after line continuation #89901

Closed
Closed
3.11 exec raises SystemError instead of SyntaxError on char after line continuation#89901
Labels
3.11only security fixes3.9 (EOL)end of lifetype-bugAn unexpected behavior, bug, or error

Description

@pewscorner
pewscorner
mannequin
opened on Nov 6, 2021
BPO 45738
Nosy @aroberge, @Carreau, @lysnikolaou, @Zac-HD, @pablogsal, @miss-islington, @isidentical, @pewscorner
PRs
  • bpo-45738: Fix computation of error location for invalid continuation characters in the parser #29550
  • [3.10] bpo-45738: Fix computation of error location for invalid continuation (GH-29550) #29551
  • [3.9] bpo-45738: Fix computation of error location for invalid continuation (GH-29550) #29552
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-11-14.13:59:48.259>
created_at = <Date 2021-11-06.13:10:51.997>
labels = ['type-bug', '3.9', '3.11']
title = '3.11 exec raises SystemError instead of SyntaxError on char after line continuation'
updated_at = <Date 2021-11-14.18:39:21.628>
user = 'https://github.com/pewscorner'

    bugs.python.org fields:

    activity = <Date 2021-11-14.18:39:21.628>
actor = 'mbussonn'
assignee = 'none'
closed = True
closed_date = <Date 2021-11-14.13:59:48.259>
closer = 'pablogsal'
components = []
creation = <Date 2021-11-06.13:10:51.997>
creator = 'pewscorner'
dependencies = []
files = []
hgrepos = []
issue_num = 45738
keywords = ['patch']
message_count = 12.0
messages = ['405860', '406188', '406223', '406224', '406235', '406236', '406301', '406303', '406304', '406305', '406322', '406329']
nosy_count = 8.0
nosy_names = ['aroberge', 'mbussonn', 'lys.nikolaou', 'Zac Hatfield-Dodds', 'pablogsal', 'miss-islington', 'BTaskaya', 'pewscorner']
pr_nums = ['29550', '29551', '29552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45738'
versions = ['Python 3.9', 'Python 3.11']

    Activity

    1. pewscorner commented on Nov 6, 2021

      @pewscorner
      pewscornermannequin
      on Nov 6, 2021
      MannequinAuthor

      When exec'ing code with an illegal character after the line continuation character '\', python 3.11.0a2 raises this strange exception:

      SystemError: Negative size passed to PyUnicode_New

      Here's an example where '#' is the illegal character:

        exec('1,\\#\n2')

      I expected the following exception (which is raised by Python 3.10.0, and also by Python 3.11.0a2 when using eval instead of exec, or when the string content is put into its own file and run as a normal script):

      SyntaxError: unexpected character after line continuation character

    2. pewscorner
      added
      3.11only security fixes
      type-bugAn unexpected behavior, bug, or error
      on Nov 6, 2021

    3. Zac-HD commented on Nov 12, 2021

      @Zac-HD
      Zac-HDmannequin
      on Nov 12, 2021
      Mannequin

      It looks like this also affects Python 3.9.8, which makes me very suspicious of https://bugs.python.org/issue45494 as the probable cause.

      See Zac-HD/hypothesmith#16 and psf/black#2592 (comment) for reports.

    4. Zac-HD
      added
      3.9 (EOL)end of life
      on Nov 12, 2021

    5. Carreau commented on Nov 12, 2021

      @Carreau
      Carreaumannequin
      on Nov 12, 2021
      Mannequin

      Yep, this also break IPython command completions when users press tab in some places in multiline code.

      88f4ec8 is the first offending commit for me in the 3.9 branch:

      [3.9] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) (bpo-29071)

There are two errors that this commit fixes:

* The parser was not correctly computing the offset and the string
  source for E_LINECONT errors due to the incorrect usage of strtok().
* The parser was not correctly unwinding the call stack when a tokenizer
  exception happened in rules involving optionals ('?', [...]) as we
  always make them return valid results by using the comma operator. We
  need to check first if we don't have an error before continuing..
(cherry picked from commit a106343f632a99c8ebb0136fa140cf189b4a6a57)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

NOTE: unlike the cherry-picked original, this commit points at a crazy location
due to a bug in the tokenizer that required a big refactor in 3.10 to fix.
We are leaving as-is for 3.9.

    6. Carreau commented on Nov 12, 2021

      @Carreau
      Carreaumannequin
      on Nov 12, 2021
      Mannequin

      Note that with the following example, exec and CommandCompiler return a different error:

          from codeop import CommandCompiler
      # identical errors
#exec('1,\\#\n2') SystemError
#CommandCompiler()('1,\\#\n2', symbol='exec') SystemError

          # one is syntax the other is System.
    exec('a \ \n')  # SyntaxError
    CommandCompiler()('a \ \n', symbol='exec') # SystemError

    7. pablogsal commented on Nov 12, 2021

      @pablogsal
      pablogsal
      on Nov 12, 2021
      Member

      I will be able to take a look later in the weekend, so if someone is able to give it a go before, that would be great.

    8. Carreau commented on Nov 12, 2021

      @Carreau
      Carreaumannequin
      on Nov 12, 2021
      Mannequin

      I will be able to take a look later in the weekend

      Please take some time for yourself, I added you as you were the original author of the patch to make you aware. I believe most projects can workaround that for now.

    9. pablogsal commented on Nov 14, 2021

      @pablogsal
      pablogsal
      on Nov 14, 2021
      Member

      New changeset 25835c5 by Pablo Galindo Salgado in branch 'main':
      bpo-45738: Fix computation of error location for invalid continuation (GH-29550)
      25835c5

    10. miss-islington commented on Nov 14, 2021

      @miss-islington
      miss-islington
      on Nov 14, 2021
      Contributor

      New changeset bf26a6d by Miss Islington (bot) in branch '3.10':
      bpo-45738: Fix computation of error location for invalid continuation (GH-29550)
      bf26a6d

    11. pablogsal commented on Nov 14, 2021

      @pablogsal
      pablogsal
      on Nov 14, 2021
      Member

      This should be fixed by now, but please, check if everything is in order in your respective test suites and confirm here.

    12. pablogsal commented on Nov 14, 2021

      @pablogsal
      pablogsal
      on Nov 14, 2021
      Member

      New changeset 142fcb4 by Pablo Galindo Salgado in branch '3.9':
      bpo-45738: Fix computation of error location for invalid continuation characters in the parser (GH-29550) (GH-29552)
      142fcb4

    13. pewscorner commented on Nov 14, 2021

      @pewscorner
      pewscornermannequin
      on Nov 14, 2021
      MannequinAuthor

      Seems to work. Thanks!

    14. pablogsal
      closed this as completedon Nov 14, 2021

    15. Carreau commented on Nov 14, 2021

      @Carreau
      Carreaumannequin
      on Nov 14, 2021
      Mannequin

      Works for me as well. Thanks.

    16. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixes3.9 (EOL)end of lifetype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions