Repository navigation
can the fs module operate on fd's opened by a native add-on? #15433
Description
Activity
I think if you use libuv to obtain the fd, and wrap it appropriately for use in JavaScript like the
fsmodule does, you should be fine. If the wayfsworks ever changes, you could get broken though.FWIW, I've done this and yes, it works and yes, it turns out to be a bad idea. Your best bet to to use
fsto open/close the fd. At least for now.As well, it won't work if your add-on - or, more commonly, one of your add-on's libraries - is linked against a different CRT.
node.exeis linked with/MT, the static multi-threaded CRT. If a library is linked with e.g./MD(dynamic multi-threaded), it gets a separate file descriptor table that won't be visible touv__get_osfhandle().Thanks to everyone for the feedback.
node.exe is linked with /MT
Will
node-gyp rebuildon Windows link with/MTby default?I'm assuming that
fds on Mac and Linux are no problem at all and that the "bad idea" part is really only with Windows?The reason for asking is I'd like to add support for
O_DIRECTandO_SYNCon Windows (viaFILE_FLAG_NO_BUFFERINGandFILE_FLAG_WRITE_THROUGH
respectively). See https://ayende.com/blog/174785/fast-transaction-log-windows for more details.If converting Windows handles to
fds and back is going to be problematic for a native add-on, then I don't mind doing the work to add this to Node'sfs.open()directly, if someone else can step up to write the test.Will node-gyp rebuild on Windows link with /MT by default?
Yes (although any libraries your add-on links against may not be, of course.)
I tried to open a file from a native add on, just doing this:
HANDLE file = CreateFileW( pathw, FILE_GENERIC_READ | FILE_GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, NULL, OPEN_EXISTING, FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, NULL ); free(pathw); if (file == INVALID_HANDLE_VALUE) { return Nan::ThrowError("CreateFileW failed"); } int fd = _open_osfhandle((intptr_t) file, _O_RDWR | _O_EXCL); if (fd < 0) { CloseHandle(file); return Nan::ThrowError("_open_osfhandle failed"); } info.GetReturnValue().Set(Nan::New<v8::Number>(fd));That's all the native add on does. It's just there to open the file with Windows' O_DIRECT and O_DSYNC equivalents since these are not yet supported by
libuv.I get the
fdreturned successfully from the native add on, but when I pass it back to Node's fs.fstatSync() or any other fs method I getEBADF: bad file descriptor, fstat.I thought the native add on would share the static multi-threaded CRT?
Now that you mention it: add-ons probably inherit the
/MTswitch from node's common.gypi, which would mean they each get their own CRT copy. That would be... sub-optimal.@nodejs/platform-windows Can one of you check that hypothesis?
add-ons probably inherit the
/MTswitch from node's common.gypi, which would mean they each get their own CRT copy.Hypothesis checks out. Removing
/MTfrom common.gypi makes the compiled modules much smaller, and they still seem to work well. However, they will have to use the CRT installed in the machine, and this is probably an issue when distributing compiled modules.@joaocgreis Anything we can do about that? I think ideally we'd want add-ons to use the CRT that's linked with node; i.e., we re-export the CRT's symbols. Are there downsides to that?
I guess this one is pretty dormant but does anyone know a workaround for this problem? I have the same problem (although in my case I want to pass a pipe handle to
uv_pipe_open).(The only way I see this working is if node.exe exports
_open_osfhandleas it does with_get_osfhandle(viauv_get_osfhandle) to convert the other way.)Reacted by Joran Dirk Greef@eriohl with named pipes specifically you have to be very careful, as they have multiple combinations of mode that could make trouble (PIPE_NOWAIT or TYPE_MESSAGE/PIPE_READMODE_MESSAGE)
Anyway there's a PR that was already approved for libuv, hopefully it will make it to nodejs soon!
Reacted by eriohl- added a commit that references this issue
on Aug 20, 2018
Is it safe to use a native add-on to open a file, and then pass the file descriptor to methods of the
fsmodule?My concern is especially for Windows. Will everything just work because Libuv uses
uv__get_osfhandle?Usually, people want to open a file in Node, and pass the file descriptor to a native add-on and this used to give problems on Windows before
uv__get_osfhandlewas made public.