[Packager] Switch file watcher to Chokidar

Hey. You are currently using sane for file watching. How about using chokidar?

Community:
- Sane was downloaded 98K times last month.
- Chokidar was downloaded 58K times just yesterday. It is also used by socketstream and stuff like Gulp is switching to it right now.
Bugs:
- Chokidar is obviously more tested in real-world applications - we have a solid track of 24m+ stable releases. We are not aware of any "node watch bugs" you mention on the web site — with correct implementation. Chokidar also has huge test coverage.
Speed:
- With Chokidar, you may use our fastest available polling for OSX with FSEvents. Sane also uses FSevents, but through child_process. This is NOT fast and also error-prone. Chokidar uses FSevents with native calls.

I know that Sane is made by facebook dude, but how about trying more stable & robust solution?

@vkurchatkin

Additionally chokidar has so many problems but it's suffice to say it doesn't work on large codebases like ours.

They fixed a lot of issues in 1.0. I used to find it unreliable for months, but I suggest you give it another try unless you're absolutely sure.

He has the point. If chokidar has serious problems, why doesn't anyone report them? Are you absolutely sure that sane (which uses child_process) would handle large codebases without issues?

Hey. You are currently using sane for file watching. How about using chokidar?

We're primarily interested using watchman. Sane is just use for a fallback. Watchman is battle tested on facebook infrastructure, developer laptops and many open source users.

I'm still open to exploring that option but I believe that watchman is a much better technology.

Chokidar was downloaded 58K times just yesterday. It is also used by socketstream and stuff like Gulp is switching to it right now.

That's great. But by the same standards watchman is a much popular open source project than chokidar, it has 2.5k github stars while chokidar has 700 :)

Chokidar is obviously more tested in real-world applications - we have a solid track of 24m+ stable releases.

Again, watchman is also battle tested 36 releases :)

We are not aware of any "node watch bugs" you mention on the web site — with correct implementation. Chokidar also has huge test coverage.

Yes, because you use a native fsevents module.

He has the point. If we have serious problems, why doesn't anyone report them? Are you absolutely sure that sane (which uses child_process) would handle large codebases without issues?

I did report them in the past. It was incredibly slow to use on our codebase, I needed a ready event which I sent a pull request for that wasn't accepted. Even with the ready event it was no match to watchman's instant daemon model.

Also "child_process" is dishonest. We only exec to show an error message that watchman isn't installed.

@amasad

Would be great to go full NPM without homebrew though — opened the issue facebook/watchman#91

Also "child_process" is dishonest.

Right. Sorry for that, the import confused me.

@amasad what about that?

  // We need to ask the client binary where to find it.
  // This will cause the service to start for us if it isn't
  // already running.
  var watchmanCommand = this.watchmanBinaryPath + ' get-sockname';
  childProcess.exec(watchmanCommand,

@amasad

@amasad Do you perceive that there would be value in adding a watchman option to chokidar for users willing to do the separate non-npm install?

If we were to move in that direction, is there potential for consolidating efforts toward a single library, or are there any other concerns/shortcomings with the recent versions of chokidar that would prevent you from considering this?

@wez

Would be great to go full NPM without homebrew though — opened the issue facebook/watchman#91

Maybe @wez can think of a way this could work. But the way watchman works -- and what makes it fast and reliable -- is that it's a daemon.

@amasad is that the exec check?

So this shells out to get the socket name and then just uses unix sockets to connect to watchman.

@amasad Do you perceive that there would be value in adding a watchman option to chokidar for users willing to do the separate non-npm install?

If we were to move in that direction, is there potential for consolidating efforts toward a single library, or are there any other concerns/shortcomings with the recent versions of chokidar that would prevent you from considering this?

Yes, this sounds great. If chokidar has all the backends/fallbacks that sane has then there is no reason for sane to exist.

Yes, this sounds great. If chokidar has all the backends/fallbacks that sane has then there is no reason for sane to exist.

Well let me just finish pushing 1.0.0 out the door (should be next week), but after that I'd be happy to collaborate on giving this idea a try.

Here is the watching backends that would be great to support in this consolidated effort:

watchman -- used for high performance servers and large codebases
fsevents (what chokidar has) -- used for the common case where the user is on OS X and can install native module
node fs.watch (still better than polling) -- used for common linux and windows users
polling (currently done via the watch module on sane) -- used for environments where no native fs events exists, e.g. vagrant

Thoughts?

By the way, one of the remaining pain points remaining on using chokidar with extremely large file trees isn't the actual watching (unless using polling) - it's the need to scan the entire tree at instantiation as a basis for normalizing events. Currently use readdirp for this, have been thinking about submitting a patch there to create a "fast mode" option that avoids getting a stat for each file.

@amasad

@amasad sounds great

Well yeah - chokidar already has great support for 2-4 on your list. The effort would be to add watchman, and preferably detect its presence to have it kick in by default when available - otherwise fall back to existing defaults.

Also, regarding polling, see paulmillr/chokidar#242. Been mulling over how to solve this as well. Want to cover as many cases as possible where we can get it to "just work" without any environment-specific settings needed.

I'd be happy to help you figure out how to consume watchman from chokidar. For large trees and multiple tools wanting to consume file notifications, using watchman as a consolidation point will reduce overall resource utilization and avoid node-process startup time overheads (particularly if you restart your node process).

@es128

@es128 cool! It seems like chokidar has really improved in the year that I hadn't looked. Let's make this happen.

cc @stefanpenner any concerns?

Ya the editor induced churn is painful. Maybe watchman itself we can push through the ecosystem to help facilitate better experiences.

I suspect it fairly unlikely our os vendors will be likely to improve in the short term, but maybe it's worth dreaming?

This long running chokidar issue was brought to my attention. I'm going to leave a mention of it here for posterity paulmillr/chokidar#219

In my experience, the watchman process is the most stable form of watching I have yet to encounter.

I re-visited the c extension problem and the issue persists. New users are extremely confused, often believing the entire attempt failed or at best assuming a quality issue in the tool they use.
In hopes of dealing with this issue at the source, I have opened the following issue with NPM: npm/npm#7892 I believe this is primarily a UX issue, hopefully it can be sorted out.

One downside though, is that if the c-extension fails, the resulting watching experience may be as broken as the c-extension failing implied :(

2015 and we have to resort to hacks to get meaningful information change events form the FS. What a sad state of affairs.

@stefanpenner

@stefanpenner the npm UX issue is different from the one @amasad is referring to.

There is an issue in fsevents when it's bombarded with simultaneous events in some specific circumstances that sometimes causes processes to crash with a segfault - notably doing rm -rf node_modules && npm i with a big enough dep tree. @bnoordhuis has been putting some time in toward trying to resolve it, although that effort is still in progress. Unfortunately I don't have the C chops to do something about it myself, and I hope we're eventually able to sort it out.

Nonetheless, as has been discussed in this thread, looking forward to adding a watchman interface to chokidar which would provide another option for mediating this and other problems that still exist with the lower-level watch methods currently being relied upon.

if the c-extension fails, the resulting watching experience may be as broken as the c-extension failing implied

I disagree with this. In this case it falls back to the same code that works very well on Linux and Windows. It's not as good as it would be with fsevents, but it's not flat-out broken. The test suite is run using the fs.watch and fs.watchFile modes on all three platforms.

Yes. I am raising an additional concern.

For the most part, this impacts Mac users who haven't installed XCode (or the command-line tools). Is that who you're focusing on by raising the concern? Or is there another persona I'm unaware of that struggles with this?

At least as much savvy is required to go out and install watchman as to resolve the XCode thing. Which is why facebook/watchman#91 would be nice.

Also, fsevents/fsevents#59 is in progress which may largely alleviate this problem as it relates to chokidar/fsevents.

There is another cohort, one with Xcode installed but with a broken compiler tool chain. This is sadly common with those with enough context with the tech to be dangerous but insufficient context or time to resolve. This can easily occur when upgrading An OS, or botching something as simple as env $PATH.

Although it would be nice to re-delegate this issue back to the user in question. Our goal should be to reduce complexity/ friction/ noise not to compound it unless absolutely needed.

I realize I am likely coming off as a pain, but my motivations are really just to pave the pain points I observe from my users.

Our goal should be to reduce complexity/ friction/ noise not to compound it.

Absolutely, to whatever extent is possible

Going to close because it's not really actionable (we can continue to discuss). Let's revisit if once we have a plan to address the issues above.

Navigation Menu

Repository navigation

[Packager] Switch file watcher to Chokidar #628

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Packager] Switch file watcher to Chokidar #628

Description

Activity

vkurchatkin commented on Apr 2, 2015

paulmillr commented on Apr 2, 2015

amasad commented on Apr 2, 2015

amasad commented on Apr 2, 2015

paulmillr commented on Apr 2, 2015

es128 commented on Apr 2, 2015

amasad commented on Apr 2, 2015

es128 commented on Apr 2, 2015

amasad commented on Apr 2, 2015

es128 commented on Apr 2, 2015

paulmillr commented on Apr 2, 2015

es128 commented on Apr 2, 2015

es128 commented on Apr 2, 2015

wez commented on Apr 2, 2015

amasad commented on Apr 2, 2015

13 remaining items

stefanpenner commented on Apr 3, 2015

amasad commented on Apr 13, 2015

stefanpenner commented on Apr 13, 2015

es128 commented on Apr 13, 2015

stefanpenner commented on Apr 13, 2015

es128 commented on Apr 13, 2015

stefanpenner commented on Apr 13, 2015

es128 commented on Apr 13, 2015

amasad commented on Jun 2, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions