Skip to content

[Rust] Explicitly seeking a BufReader will discard the internal buffer #23926

Closed

Description

@asfimport

This behavior was observed in the Parquet Rust file reader (parquet/src/util/io.rs).

 

Pull request: #6280

 

From the Rust documentation for BufReader:

 

"Seeking always discards the internal buffer, even if the seek position would otherwise fall within it. This guarantees that calling .into_inner() immediately after a seek yields the underlying reader at the same position."

 

https://doc.rust-lang.org/std/io/struct.BufReader.html#impl-Seek

Reporter: Max Burke / @maxburke

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-7681. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Mar 27, 2020

    @asfimport
    Author

    Andy Grove / @andygrove:
    Deferring this to 1.0.0 due to the concerns about the PR adding further dependencies on unstable Rust features

  2. asfimport commented on Apr 15, 2020

    @asfimport
    Author

    Rémi Dettai / @rdettai:
    I've proposed a fix in #6949 that uses an inlined modified version of BufReader instead of nightly methods.

     

    I've tested it on a large parquet of mine:

    • on fast disk, both versions take ~30s to read the column
    • on a slow mount, the the old version takes ~160s and the fixed one still takes ~30s (still CPU bounded)
  3. asfimport commented on Apr 27, 2020

    @asfimport
    Author

    Chao Sun / @sunchao:
    Issue resolved by pull request 6949
    #6949

  4. added this to the 1.0.0 milestone on Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions