Add the close method for ElementTree.iterparse() object

@scoder

BPO	25707
Nosy	@scoder, @serhiy-storchaka, @Vgr255, @furkanonder, @jacobtylerwalls
PRs	bpo-43292: Fix file leak in `ET.iterparse()` when not exhausted #31696
Dependencies	bpo-25638: Verify the etree_parse and etree_iterparse benchmarks are working appropriately

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2015-11-23.13:57:25.201>
labels = ['3.8', 'library', 'performance']
title = 'Add the close method for ElementTree.iterparse() object'
updated_at = <Date 2022-03-05.15:26:50.497>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2022-03-05.15:26:50.497>
actor = 'jacobtylerwalls'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-11-23.13:57:25.201>
creator = 'serhiy.storchaka'
dependencies = ['25638']
files = []
hgrepos = []
issue_num = 25707
keywords = ['patch']
message_count = 8.0
messages = ['255159', '255164', '255171', '255173', '341003', '341253', '341258', '368308']
nosy_count = 5.0
nosy_names = ['scoder', 'serhiy.storchaka', 'abarry', 'furkanonder', 'jacobtylerwalls']
pr_nums = ['31696']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue25707'
versions = ['Python 3.8']

Linked PRs

gh-69893: Add the close() method for xml.etree.ElementTree.iterparse() iterator #114534

If ElementTree.iterparse() is called with file names, it opens a file. When resulting iterator is not exhausted, the file lefts not closed.

>>> import xml.etree.ElementTree as ET
>>> import gc
>>> ET.iterparse('/dev/null')
<xml.etree.ElementTree._IterParseIterator object at 0xb6f9e38c>
>>> gc.collect()
__main__:1: ResourceWarning: unclosed file <_io.BufferedReader name='/dev/null'>
34

Martin Panter proposed in bpo-25688 to add an explicit way to clean it up, like a generator.close() method.

I am unable to reproduce the issue on Windows 7 with 3.5.0; I have tried opening a small (non-empty) text. Here's the result:

>>> import xml.etree.ElementTree as ET
>>> import gc
>>> ET.iterparse("E:/New.txt")
<xml.etree.ElementTree._IterParseIterator object at 0x0023ABB0>
>>> gc.collect()
59

You have to enable deprecation warnings. Run the interpreter with the -Wa option.

Oh, my bad. Ignore my last message, behaviour is identical then. Thanks for clearing that up.

I don't think there is a need for a close() method. Instead, the iterator should close the file first thing when it's done with it, but only if it owns it. Therefore, the fix in bpo-25688 seems correct.

Closing can also be done explicitly in a finaliser of the iterator, if implicit closing via decref is too lax.

Implicit closing an exhausted iterator helps only the iterator is iterated to the end. If the iteration has been stopped before the end, we get a leak of the file descriptor. Closing the file descriptor in the finalizer can be deferred to undefined term, especially in implementations without reference counting. Since file descriptors are limited resource, this can cause troubles in real programs.

Reasons for close() in iterparse objects are the same as for close in files and generators.

Maybe we will need to implement the full generator protocol (send() and throw()) in the iterparse objects, but currently I do not know use cases for this.

Ok, I think it's reasonable to make the resource management explicit for the specific case of letting iterparse() open the file. That suggests that there should also be context manager support, given that safe usages would often involve a try-finally.

Since it might not always be obvious for users when they need to close the iterator or not, I would also suggest to not let it raise an error on a double-close, i.e. if .close() was already called or the iterator was already exhausted (and the file closed automatically), calling .close() should just do nothing.

Python 3.8.2 (default, Apr  8 2020, 14:31:25) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.etree.ElementTree as ET
>>> import gc
>>> ET.iterparse('/dev/null')
<xml.etree.ElementTree.iterparse.<locals>.IterParseIterator object at 0x7fb96f679d00>
>>> gc.collect()
34

The warning(main:1: ResourceWarning: unclosed file <_io.BufferedReader name='/dev/null'>) is no longer available in python3.8.2

@scoder

I recently did a fully compatible implementation that solves an issue discussed here:

class iterparse:
    __slots__ = '_source', '_opened', 'root', '_next'

    def __init__(self, /, source, events = None, parser = None):
        # If source cannot be opened,
        # an error is emitted and object is moved to gc,
        # gc calls __del__ which calls close().
        # opened flag must be set to false before opening,
        # so close() won't emit AttributeError.
        self._opened = False
        if hasattr(source, 'read'):
            self._source = source
        else:
            self._source = open(source, 'rb')
            self._opened = True

        self.root = None
        self._next = self._iterator(XMLPullParser(events=events, _parser=parser)).__next__

    def __iter__(self, /):
        return self

    def _iterator(self, parser: XMLPullParser, /):
        source = self._source
        try:
            data = source.read(16 * 1024)
            while data:
                parser.feed(data)
                yield from parser.read_events()
                data = source.read(16 * 1024)

            root = parser._close_and_return_root()
            yield from parser.read_events()  # is it necessary?
            self.root = root
        finally:
            self.close()

    def __next__(self, /):
        return self._next()

    def close(self, /):
        if self._opened:
            self._source.close()
            self._opened = False

    def __del__(self, /):
        self.close()

Features:

Fully compatible with current implementation.
According to my tests, implementation above has faster creation time (no new function and class each time), but a bit slower traverse time.
Has close method that does not raise an exception on double-close as @scoder asked (file descriptors also do not raise an error in such cases).
If there is no more references to iterparse object, it is destroyed closing opened fd.

Benchmarks:
I used this XML file for testing.

def test_creation(file: str, impls: list[type], /):
    code = f'iterparse({file!r})'

    return tuple(
        repeat(
            code,
            repeat=50,
            number=1000,
            globals=dict(iterparse=t),
            )
        for t in impls
        )


def test_traverse(file: str, impls: list[type], /):
    setup = f'it = iterparse({file!r})'
    code = f'for _ in it: pass'

    return tuple(
        repeat(
            code,
            setup=setup,
            repeat=50,
            number=10,
            globals=dict(iterparse=t),
            )
        for t in impls
        )


def main():
    file_path = '20220628-FULL-1_1(xsd).xml'
    impls = [iterparse_new, iterparse_old]

    creation = test_creation(file_path, impls)
    for i, time in enumerate(creation):
        print(f'Creation time of {impls[i].__name__}:', min(time))

    print()

    traverse = test_traverse(file_path, impls)
    for i, time in enumerate(traverse):
        print(f'Traverse time of {impls[i].__name__}:', min(time))


if __name__ == '__main__':
    main()

Creation time of iterparse_new: 0.08623739999984537
Creation time of iterparse: 0.10833569999977044

Traverse time of iterparse_new: 0.3242015000005267
Traverse time of iterparse: 0.3198846000004778

fixing ResourceWarning due to unclosed file when aborting iterparse i…

pythongh-69893: Add the close() method for xml.etree.ElementTree.iter…

Recent improvements (#101438) made explicit close() not needed in CPython. But it can still be useful in alternative implementations like PyPy, and maybe in future Python versions, so it is worth to add it.

gh-69893: Add the close() method for xml.etree.ElementTree.iterparse(…

pythongh-69893: Add the close() method for xml.etree.ElementTree.iter…

Navigation Menu

Repository navigation

Add the close method for ElementTree.iterparse() object #69893

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add the close method for ElementTree.iterparse() object #69893

Description

Linked PRs

Activity

serhiy-storchaka commented on Nov 23, 2015

Vgr255 commented on Nov 23, 2015

serhiy-storchaka commented on Nov 23, 2015

Vgr255 commented on Nov 23, 2015

scoder commented on Apr 27, 2019

serhiy-storchaka commented on May 2, 2019

scoder commented on May 2, 2019

furkanonder commented on May 6, 2020

Prometheus3375 commented on Jun 30, 2022

serhiy-storchaka commented on Jan 24, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions