S3Object streaming: iterating a stream corrupts subsequent seek()/read() by silently returning the wrong bytes #8388

Expected Behaviour

Consuming an S3Object/_S3SeekableIO stream via the iterator protocol (for line in s3_object: ... or next(s3_object)) should keep the stream's position tracking accurate, so any subsequent seek() or read() operates from the correct byte offset.

Current Behaviour

_S3SeekableIO.__next__ and __iter__ bypass position tracking entirely, so any seek() after iterating the stream computes the wrong S3 byte range, and the following read() silently returns the wrong slice of the object — no exception is raised, the data is just wrong (skipped or duplicated bytes).

Root cause

aws_lambda_powertools/utilities/streaming/_s3_seekable_io.py:

Every other read path in this class carefully advances self._position by the number of bytes actually consumed, specifically so seek() can compute the correct absolute S3 byte offset:

def read(self, size: int | None = -1) -> bytes:
    size = None if size == -1 else size
    data = self.raw_stream.read(size)
    if data is not None:
        self._position += len(data)
    return data

def readline(self, size: int | None = None) -> bytes:
    data = self.raw_stream.readline(size)
    self._position += len(data)
    return data

But __next__/__iter__ don't do this bookkeeping at all:

def __next__(self):
    return self.raw_stream.__next__()

def __iter__(self):
    return self.raw_stream.__iter__()

They hand iteration control directly to botocore's StreamingBody, whose own iterator reads chunks straight off the underlying stream and tracks its own internal counter — one _S3SeekableIO never looks at. self._position stays at whatever it was before iteration began.

seek() uses self._position as ground truth to build the next Range header on raw_stream access:

@property
def raw_stream(self) -> PowertoolsStreamingBody:
    if self._raw_stream is None:
        range_header = f"bytes={self._position}-"
        ...

So after iterating N bytes and then seeking, the resulting Range header is computed from the stale pre-iteration position instead of the true current position — the reopened stream starts at the wrong byte offset.

Concrete reproduction

s3obj = S3Object(bucket=bucket, key=key)

for line in s3obj:          # consumes bytes via __iter__/__next__; self._position never advances
    ...

s3obj.seek(5, io.SEEK_CUR)  # computed as 0 + 5, not (actual bytes consumed) + 5
s3obj.read(...)             # silently reads the wrong slice of the object

Note: this is masked when a GzipTransform/CsvTransform is applied, because GzipFile/TextIOWrapper internally call .read() (which is position-tracked correctly) rather than .__iter__(). The bug is exposed specifically on the raw/untransformed stream's iteration protocol — a fully supported, documented usage (S3Object's class docstring advertises being both seekable and streamable/iterable).

Possible Solution

Route __next__ through the already position-tracked readline(), and have __iter__ return self — matching the standard Python file-iterator protocol. I have a fix + regression tests ready and will open a PR referencing this issue.

Steps to Reproduce

Reproducible via unit test against _S3SeekableIO: iterate a multi-line payload fully, then check tell() — it stays at 0 instead of reflecting bytes consumed. A subsequent seek() + raw_stream access then requests the wrong Range header.

Powertools for AWS Lambda (Python) version

latest (develop, v3.34.0)

AWS Lambda function runtime

3.12

Debugging logs

N/A — logic bug reproducible via unit test, not runtime-log-dependent.