Repository navigation
Avoid PassThrough to avoid buffering in pipeline #32039
Description
Activity
This is only when the last argument in pipeline is not a stream, thus we must create a stream since
pipelineshould always return a stream.I don't think this is a big issue. If we want to fix it we need to change the return signature of
pipeline.i.e. it's to make the following valid:
pipeline(src, function*(source) { for await (const chunk of source) { yield chunk } }, err => { }).pipe(dst);
Notice the
.pipeat the end.I think the current code is correct then, maybe we should add some comments to
pipelineas it's not exactly clear what that block would doReacted by Robert NagyI think the current code is correct then, maybe we should add some comments to pipeline as it's not exactly clear what that block would do
I'll prepare a PR this week.
@mcollina: there is however one case I would like to optimize, consider:
const src = Readable.from(asyncGenerator()) pipeline(src, dst, err => {})
In this case I think pipeline should be able to unwrap the original generator, which would however require
Readable.fromto somehow re-use the generator function inSymbol.asyncIterator/Symbol.iteratorinstead of creating a new one.In the future I would like encourage users to implement readable, transform and writable in terms of generators and wrap only for compatibility.
- added a commit that references this issue
on Mar 9, 2020 - added a commit that references this issue
on Jul 27, 2026
In the new async-terator capable pipeline as implemented in #31223, we support passing in an async generator function.
Reading from
node/lib/internal/streams/pipeline.js
Line 191 in 6bcea0a
We should avoid wrapping it in a
PassThroughto avoid said buffering and overhead.