fs: add WriteStream.prototype.fsync

Right now it's pretty complicated to intermix fs.fsync() calls with ws.write() calls, to the point that you lose most of the benefits of using fs.WriteStream. Example:

const fs = require('fs');
const ws = fs.createWriteStream('test.txt');
ws.write('important data', () => {
  fs.fsync(ws.fd, () => {
    // only now is it safe again to call ws.write()
    ws.write('more important data', () => {
      fs.fsync(ws.fd, () => { /* etc. */ });
    });
  });
});

It would be exceedingly helpful if fs.WriteStream grew a .fsync() method that preserves order with respect to writes so that the following example works like I would expect it to:

const ws = require('fs').createWriteStream('test.txt');
ws.write('important data');
ws.fsync();
ws.write('more important data');
ws.fsync();

It's not quite impossible to accomplish the above today but it's not very ergonomic. Here is an async/await example:

const util = require('util');
const fs = require('fs');
const ws = fs.createWriteStream('test.txt');
ws.once('open', (fd) => go(fd));
async function go(fd) {
  const write = util.promisify(ws.write.bind(ws));
  const fsync = util.promisify(fs.fsync.bind(null, fd));
  await write('important data');
  await fsync();
  await write('more important data');
  await fsync();
}

I don't know, the fact that you need to know about the 'open' event doesn't give me warm fuzzies. Proper synchronization is important enough that I feel it merits a place in core.

My idea is to refactor the buffer pool on WriteStream to the task pool which can write chunks or do fsync. I don't know if it is correct.

for example:

@bnoordhuis

It's not quite impossible to accomplish the above today but it's not very ergonomic. Here is an async/await example:
…
I don't know, the fact that you need to know about the 'open' event doesn't give me warm fuzzies. Proper synchronization is important enough that I feel it merits a place in core.

Hey @bnoordhuis, first off thank you for your contributions to Node. I’m trying to get my mind around this… would the following have the same effect (safe* writes) if you’re writing out an append-only log?

const fs = require('fs')
const ws = fs.createWriteStream('test.txt', {flags: 'as'})
ws.write('important data')
ws.write('more important data')

(In other words, having the stream open the file in kernel-level synchronous mode.)

In my tests (on my dev laptop running Pop!_OS/Ubuntu 20.04) I’m seeing the following behaviour:

I reach highWaterMark on the calls when using 'as', where I do not with 'a'
I am oddly seeing equivalent performance (even ever so slightly faster?) with 'as' which doesn’t make sense to me given all the warnings you read everywhere about synchronous IO being slower (even at the kernel level) unless there are some optimisations for what I’m testing with that I don’t know about (I’m appending a ~2KB string in every write).

If, by chance, you have a moment, I’d really love your thoughts on this. From what I’ve read in the docs and around the web, in my mind, opening the stream with 'as' is equivalent to issuing an fsync() after every write. Am I wrong on that?

Thanks in advance!

* as safe as the OS / device allows, I mean. Which I realise isn’t necessary “safe” :)

opening the stream with 'as' is equivalent to issuing an fsync() after every write

Yes, that's correct barring bugs. E.g., 'as' is unreliable on NFS mounts, and I'm not sure if Node.js actually implements it on Windows.

I am oddly seeing equivalent performance (even ever so slightly faster?) with 'as'

That's possible (equivalent performance, at least), it depends on the file system, the mount flags and the workload.

I suspect a lot of fsync() folklore stems from the days when file systems would flush all pending data, not just for the selected file descriptor.

@bnoordhuis

@bnoordhuis Thanks so much, Ben. Hopefully this will also help anyone else looking for this information in the future. Appreciate your help.

There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

For more information on how the project manages feature requests, please consult the feature request management document.

There has been no activity on this feature request and it is being closed. If you feel closing this issue is not the right thing to do, please leave a comment.

For more information on how the project manages feature requests, please consult the feature request management document.

Navigation Menu

Repository navigation

fs: add WriteStream.prototype.fsync #28513

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fs: add WriteStream.prototype.fsync #28513

Description

Activity

himself65 commented on Jul 7, 2019

aral commented on Sep 27, 2020

bnoordhuis commented on Sep 28, 2020

aral commented on Sep 28, 2020

github-actions commented on Mar 18, 2022

github-actions commented on Apr 18, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions