Skip to content

Unbound stream upload buffers use about twice the documented memory #666

Open

Description

@sophiecarreras

What happens

For uploads from an unbound stream, each buffer is a bytearray, and UnboundSourceBytes wraps it in IOWrapper, a BytesIO subclass that copies the data. Each in-flight chunk is therefore held twice, so the memory in use is about double what buffer_size_bytes and queue_size suggest.

Where

b2sdk/_internal/transfer/emerge/unbound_write_intent.py:51 (IOWrapper.__init__), :78-86 (UnboundSourceBytes.__init__), :188-195 (_trim_to_leftovers copies the buffer).

Why it matters

Impact: Availability (memory use above the documented amount), silent. Severity: Low, because it happens on the normal unbound-stream path, but no concrete failure is shown.

How to check

Upload a stream with a large buffer_size_bytes and watch peak memory (for example with tracemalloc). Found by reading the code, not reproduced.

Version

Seen at v2.13.4. Present on the default branch: present.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions