Skip to content

FileHandle.stream #38350

Description

@ronag

It might be possible to have a more efficient way to pipe a file to a stream by adding a .stream method to FileHandle.

e.g.

FileHandle.prototype.stream = async function (dst, { start = 0, end, signal } = {}) {
 // TODO: this[kRef](), this[kUnref](), if (signal.aborted) throw AbortError()
  let pos = start
  while (true) {
    if (end && pos >= end) {
      break
    }
  
    const { buffer, bytesRead } = await this.read({
      buffer: Buffer.allocUnsafe(Math.min(end - pos, 16384)),
      position: pos,
    })
  
    if (bytesRead === 0) {
      break
    }
  
    if (dst.writableNeedDrain) {
      await EE.once(res, 'drain', { signal })
    }
  
    if (dst.destroyed) {
      break // TODO: break or throw?
    }
  
    dst.write(buffer)
  
    pos += bytesRead
  }
}
const fileHandle = fsp.open(filePath)
try {
  await fileHandle.stream(dst)
} finally {
  await fileHandle.close()
}

vs

await pipeline(fs.createReadStream(filePath), dst)

Less ergonomic but potentially better performance. This idea needs benchmarking.

Activity

  1. added
    fsIssues and PRs related to file-system APIs and the fs module.
    feature requestIssues requesting new Node.js features.
    good first issueIssues that are suitable for first-time contributors.
    on Apr 22, 2021
  2. ronag commented on Apr 22, 2021

    @ronag
    MemberAuthor

    Could also have a static alternative.

    await fsp.stream(filePath, dst)
  3. ronag commented on Apr 22, 2021

    @ronag
    MemberAuthor

    Another (less efficient) idea is to have a method that returns an async generator.

    await pipeline(fsp.generator(filePath), dst)
  4. kuzmaMinin commented on Apr 22, 2021

    @kuzmaMinin
  5. Ayase-252 commented on Apr 22, 2021

    @Ayase-252
    Member

    Could I take this challenge? 😁

  6. benjamingr commented on Apr 22, 2021

    @benjamingr
    Member

    Why a stream method rather than adding Symbol.asyncIterator? Is the goal perf or ergonomics?

  7. ronag commented on Apr 22, 2021

    @ronag
    MemberAuthor

    Why a stream method rather than adding Symbol.asyncIterator? Is the goal perf or ergonomics?

    Performance.

  8. jasnell commented on Apr 23, 2021

    @jasnell
    Member

    Just to expand on this a bit to add to the discussion. I don't have a timeline yet on exactly when I'd be able to continue working on this, but as part of the effort around enabling fetch (and a few other things), I've been looking at support for the web platform API standard Body mixin. My preference would be for whatever we do here to be aligned with that API.

    So, for instance, let's assume that a FileHandle implemented the Body mixin:

    const file = await fs.promises.open('file', 'rw');
    
    file.body;            // A WHATWG ReadableStream
    file.arrayBuffer();   // The content of the file as an ArrayBuffer
    file.blob();          // The content of the file a Blob
    file.formData();      // Doesn't really make sense here so probably good to omit
    file.json();          // The content of the file as JSON
    file.text();          // The content of the file as a string

    Following this pattern, I'd suggest a file.readable() that returns a stream.Readable if the file is readable, and a file.writable() that returns a stream.Writable if the file is writable.

  9. benjamingr commented on Apr 24, 2021

    @benjamingr
    Member

    What about adding a Symbol.readable symbol like Symbol.asyncIterator that enables getting a readable stream version of stuff (like FileHandles)?

    (Edit: obviously it'd be an import { ReadableSymbol } from 'stream' rather than monkey patching the global)

  10. ronag commented on Apr 24, 2021

    @ronag
    MemberAuthor

    What about adding a Symbol.readable symbol like Symbol.asyncIterator that enables getting a readable stream version of stuff (like FileHandles)?

    I would kind of like to move away from readables and towards async iterables. But maybe that’s a bigger discussion.

  11. benjamingr commented on Apr 25, 2021

    @benjamingr
    Member

    @ronag

    I would kind of like to move away from readables and towards async iterables. But maybe that’s a bigger discussion.

    Deprecating streams in favour of async iterables as the contract would be ideal (one stream type, only pull, all modern promises with ease of debugging and syntax assist) eventually would be ideal - it would also make incorporating whatwg streams nicer - but IIRC performance really isn't there.

  12. 5 remaining items

  13. ronag commented on Apr 28, 2021

    @ronag
    MemberAuthor

    Because there is an obvious answer to that: Because we're not using a consistent streams model on the native side.

    Also because streams are slowish. Both solutions here live in JS land and one is faster than the other.

  14. ronag commented on Apr 28, 2021

    @ronag
    MemberAuthor

    I'm fine with not doing this. Was just an idea on how get a little better perf with FileHandle (I'm doing it like this in our perf sensitive code). In an ergonomic sense I would be fine with having FileHandle.readable() and/or FileHandle[Symbol.AsyncIteartor]().

  15. jimmywarting commented on Sep 8, 2021

    @jimmywarting

    little late to the party... but how about something like

    const file = await fs.getFile('./readme.md')
    // file instanceof window.File
    file.stream() // new whatwg:stream ReadableStream()
    file.text() // Promise<string>
    file.arrayBuffer() // Promise<ArrayBuffer>
    await new Response(file).json()
    await new Response(file).formData()
  16. github-actions commented on Apr 4, 2022

    @github-actions
    Contributor

    There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

    For more information on how the project manages feature requests, please consult the feature request management document.

  17. added
    staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.
    on Apr 4, 2022
  18. moved this from Pending Triage to Stale in Node.js feature requestson Apr 4, 2022
  19. moved this to Pending Triage in Node.js feature requestson Apr 4, 2022
  20. moved this to Pending Triage in Node.js feature requestson Apr 4, 2022
  21. moved this from Pending Triage to Stale in Node.js feature requestson Apr 4, 2022
  22. github-actions commented on May 4, 2022

    @github-actions
    Contributor

    There has been no activity on this feature request and it is being closed. If you feel closing this issue is not the right thing to do, please leave a comment.

    For more information on how the project manages feature requests, please consult the feature request management document.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature requestIssues requesting new Node.js features.fsIssues and PRs related to file-system APIs and the fs module.good first issueIssues that are suitable for first-time contributors.staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions