Repository navigation
Change Dirent with stat in fs.readdir with withFileTypes #23055
Description
Activity
There are cases where
statdata is not required andwithFileTypesallows for a significant performance increase for that case (I observe up to 300% performance gain using asyncwithFileTypesover callingreaddirandstatseparately), so ifstatdata is added todirent, it must be optional. I'm actually slightly in favor of having such an option because it's pretty common that one needs timestamps and the like.Reacted by coderaiser, Govind Rai and Vas SudanaguntawithFileTypes allows for a significant performance increase for that case (I observe up to 300% performance gain using async withFileTypes over calling readdir and stat separately
That's it, I'm also would like to have benefit of 300% performance gain with no need to call
statseparately :).The thing is what the option
withFileTypesis , it's justreaddir+stat, so I can't understand why to make simple things so hard :).The thing is what the option
withFileTypesis , it's justreaddir+stat, so I can't understand why to make simple things so hard :).It isn't just
readdir+stat. It's actually quite a bit less than that in most cases, on most platforms.The
libuvcall that powersreaddironly returns the types of files and their names, not any other metadata. (See thelibuvdocumentation for that returned type.) There are usually no additionalstatcalls happening. You may see code doing this in #22020, but it's only there to handle the cases where the file type information could not be retrieved in the samelibuvcall asreaddir. This only seems to happen on some specific systems/platforms (in testing, it seemed to happen on AIX, if I remember correctly).As @silverwind has mentioned, there are cases where most
statdata is not required, and file types are enough information to proceed. Indeed, the feature was added to support these use cases. In these cases, on systems wherestatcalls are unnecessary, an extrastatcall per file would negatively impact performance, especially when that data is already available from the originalreaddir/scandircall.That being said, as @silverwind also mentioned, there's certainly some utility in having full
Statobjects returned viareaddir, so perhaps it might seem that this warrants an additional option,withFileStats. A problem I see with that is that there's nonameproperty onStatsobjects, and it's rather necessary when listing directory entries. This could be added, though. On the other hand, this can easily be implemented as a library on npm, so it might not be worth doing in Node.js core. I don't really have a strong opinion here yet.Reacted by coderaiser and Vas SudanaguntaJust published fs-readdir-sync-with-file-types and fs-readdir-with-file-types
ponyfillto have ability to benefit of file types onnode < v10.10:).It seems like everything has been answered here, so I'm going to go ahead and close this. Please feel free to reopen/comment if you have additional related questions or if you think something has been left unanswered.
Just my 2c, but it doesn't seem that this was indeed answered (resolved) as indicated by @bengl. I personally would be in favor of an option to add additional stat properties, as the OP requested. It would be great to see this reopened (forgive me if this was tabled or discussed elsewhere).
Reacted by Govind Rai, Roman Dubrov, Jannis Leifeld, Chris Fairbanks, chag, Samuel Carswell, Adam Lyrén, Irelynx, Vas Sudanagunta, mvladimir7 and 4 moreOld but interesting. FWIW,
withFileTypes: true, at least on modern linux, is callingstatx()syscall to get the file type data. (withoutwithFileTypes: truethere are nostatx()calls for each). In addition, thestatx()result is returned with each directory entry, but is inaccessible -- hidden behind a Symbol.If one insists not to call
fs.stat()and attempts to use readdir data, it is possible to extract the Symbol(stats) from the first entry and reuse it:let statsSym; let dirData = require('fs').readdirSync('.', { withFileTypes: true }); if (!statsSym) Object.getOwnPropertySymbols(dirData[0]).forEach(s => { if (s.toString() == 'Symbol(stats)') statsSym = s; }); dirData.forEach(dentry => console.log(dentry.name, dentry[statsSym].size));
In node v10.10.0 with merge request #22020 was added a new flag
withFileTypesto fs.readdir:It is really great addition and I have a couple ideas to share.
Would be great if fs.Dirent contain all information fs.Stat has. I can see no reason to create new entity for such simple data structure which is half duplicated.
Lets compare them:
fs.Dirent
dirent.isBlockDevice()dirent.isCharacterDevice()dirent.isDirectory()dirent.isFIFO()dirent.isFile()dirent.isSocket()dirent.isSymbolicLink()dirent.namefs.Stats
stats.isBlockDevice()stats.isCharacterDevice()stats.isDirectory()stats.isFIFO()stats.isFile()stats.isSocket()stats.isSymbolicLink()stats.devstats.inostats.modestats.nlinkstats.uidstats.gidstats.rdevstats.sizestats.blksizestats.blocksstats.atimeMsstats.mtimeMsstats.ctimeMsstats.birthtimeMsstats.atimestats.mtimestats.ctimestats.birthtimeThe only thing that missing in
fs.Statsit's name.Is your feature request related to a problem? Please describe.
I'm working on file manager for the web Cloud Commander, so I do such things all the time: read directory content and then get stat of every file. Now this all done in readify. And I would like to use the new approach of reading files with stats, it can simplify my code a lot, and make node
APIcleaner.Right now if I use a flag
withFileTypesI will need to callfs.statanyways because I need not only divide files and directories but also get theirsize,mode,uidandmtime.Describe the solution you'd like
I suggest to change
withFileTypesbehavior to get realstatinformation: likesize,mtime,uidin method call:And remove this
fs.Dirent.Describe alternatives you've considered
Also we can use something like
withFileStatsto get full stats (maybe with names).