Skip to content

Reduce memory usage of urllib.unquote and unquote_to_bytes #88500

Closed
#96763
Closed
Reduce memory usage of urllib.unquote and unquote_to_bytes#88500
#96763
Assignees
gpshead
Labels
3.12only security fixesperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

Description

@mustafaelagamey
mustafaelagamey
mannequin
opened on Jun 7, 2021
BPO 44334
Nosy @terryjreedy, @gpshead, @orsenthil, @mustafaelagamey
PRs
  • bpo-44334: Use bytearray in urllib.unquote_to_bytes #26576
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2021-06-07.15:07:38.990>
labels = ['extension-modules', '3.11', '3.9', '3.10', 'performance']
title = 'Use bytearray in urllib.unquote_to_bytes'
updated_at = <Date 2021-06-07.20:09:11.522>
user = 'https://github.com/mustafaelagamey'

    bugs.python.org fields:

    activity = <Date 2021-06-07.20:09:11.522>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2021-06-07.15:07:38.990>
creator = 'eng.mustafaelagamey'
dependencies = []
files = []
hgrepos = []
issue_num = 44334
keywords = ['patch']
message_count = 2.0
messages = ['395280', '395281']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'gregory.p.smith', 'orsenthil', 'eng.mustafaelagamey']
pr_nums = ['26576']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue44334'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. mustafaelagamey
      added
      3.8 (EOL)end of life
      extension-modulesC modules in the Modules dir
      performancePerformance or resource usage
      on Jun 7, 2021
    2. mustafaelagamey
      changed the title [-]urllib cannot parse large data[/-] [+]urllib.parse.parse_qsl cannot parse large data[/+] on Jun 7, 2021
    3. mustafaelagamey
      changed the title [-]urllib cannot parse large data[/-] [+]urllib.parse.parse_qsl cannot parse large data[/+] on Jun 7, 2021

    4. terryjreedy commented on Jun 7, 2021

      @terryjreedy
      terryjreedy
      on Jun 7, 2021
      Member

      'eng' claimed in original title that "urllib.parse.parse_qsl cannot parse large data". On original PR, said problem with 6-7 millions bytes.

      Claim should be backed up by a generated example that fails with original code and succeeds with new code. Claims of 'faster' also needs some examples.

      Original PRs must nearly all propose merging a branch created from main into main. Performance enhancements are often not backported.

    5. terryjreedy
      added
      3.11only security fixes
      and removed
      3.8 (EOL)end of life
      on Jun 7, 2021
    6. terryjreedy
      changed the title [-]urllib.parse.parse_qsl cannot parse large data[/-] [+]Use bytearray in urllib.unquote_to_bytes[/+] on Jun 7, 2021
    7. terryjreedy
      added
      3.11only security fixes
      and removed
      3.8 (EOL)end of life
      on Jun 7, 2021
    8. terryjreedy
      changed the title [-]urllib.parse.parse_qsl cannot parse large data[/-] [+]Use bytearray in urllib.unquote_to_bytes[/+] on Jun 7, 2021

    9. 4 remaining items

    10. iritkatriel commented on Sep 11, 2022

      @iritkatriel
      iritkatriel
      on Sep 11, 2022
      Member

      The PR was closed due to technicalities (pointing to the wrong branch, CLA) and the OP didn’t follow up.

      Unless someone object I will close this issue as well.

    11. iritkatriel
      added
      pendingThe issue will be closed if no feedback is provided
      on Sep 11, 2022
    12. gpshead
      added a commit that references this issue on Sep 12, 2022

      pythongh-88500: Reduce memory use of urllib.unquote

      c02c357
    13. gpshead
      added
      stdlibStandard Library Python modules in the Lib/ directory
      3.12only security fixes
      and removed
      extension-modulesC modules in the Modules dir
      3.9 (EOL)end of life
      on Sep 12, 2022
    14. gpshead
      changed the title [-]Use bytearray in urllib.unquote_to_bytes[/-] [+]Reduce memory usage of urllib.unquote and unquote_to_bytes[/+] on Sep 12, 2022
    15. gpshead
      removed
      3.11only security fixes
      3.10 (EOL)end of life
      on Sep 12, 2022

    16. gpshead commented on Sep 12, 2022

      @gpshead
      gpshead
      on Sep 12, 2022
      Member

      I created a new PR and included fixing a similar legacy design issue in unquote() as well as the original report's unquote_to_bytes(). Some performance microbenchmarks need running before I'll consider moving forward with it.

      If someone wanted to consider this a security issue it could be backported. It is at most a fixed constant factor (roughly $len(input) * sizeof(PyObject)$ memory consumption vs a maximally antagonistic input though. That doesn't smell DoS worthy.

    17. iritkatriel
      removed
      pendingThe issue will be closed if no feedback is provided
      on Sep 12, 2022
    18. gpshead
      self-assigned this
      on Nov 11, 2022
    19. gpshead
      closed this as completedin #96763on Dec 11, 2022
    20. gpshead
      added a commit that references this issue on Dec 11, 2022

      gh-88500: Reduce memory use of `urllib.unquote` (#96763)

      2e279e8
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.12only security fixesperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions