Skip to content

Bound streaming JSONL record buffering and recover from oversized input - #2

Merged
bconnell merged 9 commits into
mainfrom
work/bounded-jsonl-input-20261108
Nov 9, 2026
Merged

bconnell merged 9 commits into
mainfrom
work/bounded-jsonl-input-20261108

Conversation

@bconnell

@bconnell bconnell commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Change

Replace the unbounded per-line read_line allocation with a bounded BufRead::fill_buf scanner. Each record is limited to 1,048,576 bytes including its newline. The scanner drains the rest of an oversized record without buffering it and can resume on the next line. Malformed UTF-8 records are counted as malformed in non-strict mode; strict mode emits a safe line-numbered failure. The core parsing, filtering, normalized timestamp comparison, original-byte JSONL output, and exact summary semantics remain otherwise intact.

Tests include oversized lines followed by valid JSON, strict oversize rejection without logging input content, malformed UTF-8 recovery, and the exact memory-limit boundary. README discloses the per-record bound and remaining unbounded growth in distinct summary codes.

Gold Standard merge gates

Repository baseline lacks the Cargo.lock required by its validation contract. The temporary PR job first creates a reviewable lockfile artifact, followed by source tests. Before merge, commit that exact verified lockfile, switch workflow to --locked for tests and Release build, run text gate, rustfmt, Clippy and CLI on the exact updated head, verify jobs and re-review. Remain draft until complete. No permanent dependency-generation, external telemetry or background service is intended.

@bconnell
bconnell marked this pull request as ready for review November 9, 2026 02:58
@bconnell
bconnell merged commit 4d94d16 into main Nov 9, 2026
1 check passed
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T03:00:37.423932Z 47eb2dc Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant