You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would like to open a discussion on our strategy for applying benchmarks across various LoopBack repos as discussed during our meeting @raymondfeng and @ritch.
I'll comment here, wiki is really bad for discussions (Google Docs works better for that IMO).
I would personally start with a full end-to-end benchmark covering loopback, strong-remoting, loopback-datasource-juggler and and a single connector. The connector under test should be configurable via the environment or command-line arguments, no code changes needed.
This should give us a good overview of which parts work well and which needs most improvement.
For the beginning, I would focus on the following scenarios:
Detect memory leaks
Benchmark requests per second, measure how much time is needed by individual parts: strong-remoting, loopback+juggler, connector
The made-up numbers above show that we spend most time talking to the database, so that's probably the first thing to check when looking for places to optimize. The second biggest contributor is strong-remoting, especially when the load gets higher, so that would be the second place to investigate.
Instead we should have automated benchmarks run per module. We should be able to see if there is a decrease on a per PR basis. Otherwise I'm not sure its worth the effort.
Create a new project named `loopback-leak-detection
Individual package that can be run on demand
Detect leaks for new functions added over time on an as-needed basis
Integration style tests (ie. maybe a dir for each functional area we want to check for leaks)
The reason for this is to do leak detection on every function would be an exponential problem as each leak detection test needs to be written in a a fashion similar to unit testing. Running the tests longer also do a better job of detecting leaks. So my suggestion for the entire solution is:
New project to consolidate tests for all loopback projects (this project will do leak detection for various parts of LB, strong-remoting, connectors, etc)
Pushing this project to GitHub should trigger the CI to run tests (which can take hours depending on the number of iterations we decide on)
The benefits of doing this way is that we can just add tests as need and the CI can run tests as it normally does without major infrastructure changes. This CI server should be separate from the regular CI server because these long running jobs would block other jobs from executing on the normal CI server.
benchmarking
Tests run and issues are printed to results.md at the moment. Thats only for the functions we decide to test. From @bajtos suggestions, I feel we need to isolate which parts of LoopBack are slow (and compared to what).
Express speed is the benchmark (because this is probably as fast as possible)
Compare our times against Express (or similar middleware using express)
The problem is to get such a precise benchmark as suggested by @bajtos, we need to measure various parts of LB to get a number and compare it something. For example, if loopback-connector-mongodb db has a speed of 5 and mongoose has a 4 and mongodb native driver has a 3. We'd have to benchmark against all 3 to make a fair comparison (this is also 3 times more work since we're testing 3 modules now, instead of just ours). The nice thing is once the test is written once, we should be able to rerun them and compare the values over time.
The second problem I see is we need to persist this data somewhere. I mean text files work fine for now, but it would make more sense to have some sort of data store hold these values. Maybe a LoopBack app that accepts some log values and simply displays it?
As for CI, if we create a project named loopback-benchmarks (replace the other one with the same name), with the same dir/tree structure as the leak detection example above, we can add benchmarks to specific functions over time too. And also run this on the same CI server as the leak detection (these two can share processing time since they both have long runnings jobs that don't need to be run all the time).
The alternative to gather everything in one repo is to have a separate benchmarks and leak-detection dir inside each individual project (like we current do in https://github.com/strongloop/loopback-connector-mongodb), but we will be clogging up CI if we ran this on a regular basis.
From my POV, I'd like to see a benchmark that will tell us which part of LoopBack is the slowest one (see #1291 (comment)). For example, start with the method "find with options" and gather timing data split across module boundaries: express, strong-remoting, loopback, loopback-datasource-juggler, connector, SQL backend.
I agree with this approach as we're not just benchmarking randomly. Maybe start with benchmarking each connector vs it's alternative (ie. loopback-connector-mongodb vs native vs mongoose). Do the same for all 5 connectors. Then we can perform the same comparisons with juggler + connectors. Then once we have this, we can add in strong remoting, etc. This way, we can split this giant task into smaller stories and flush it out over time.
Story 1 - mongo connector vs mongoose - 2 points
Story 2 - mongo connector vs mongo native - 2 points
Story 3 - juggler + mongo connector vs mongoose - 2 points
Story 4 - juggler + mongo connector vs mongo native - 2 points
Story 5 - remoting + juggler + mongo connector - 2 points
Story 6 - express + mongoose - 2 points
Sotry 7 - express + mongo native - 2 points
And so forth. We still have to decide where to persist the results to make it easy to compare the results.
If I am reading the benchmark results correctly, then our MongoDB connector is performing roughly as well as mongoose/mongodb and there isn't much space for optimization at our side. Is my reasoning correct?
if you guys think its worth continuing (so you can compare full request split-times).
I think it would be good to create a similar benchmark for one of our SQL connectors too. Then I would personally like to see how much overhead is added by juggler and strong-remoting. So yes, I think it makes sense to continue.
@bajtos Yes, you are reasoning correctly from the subset of data provided. I wanted to make sure this spike is providing the data you want to see before diving too deeply.
The plan is to do the same for the the other connectors, but I would like to finish up the MongoDB spike first to flush out any issues or questions that may need addressing. Adding the last two scenarios would make this a fairer comparision because express-mongoose is a common strategy and of course loopback-loopback-connector-mongodb would be a better real world baseline.
After this, we need to figure out a strategy for versioning. The way I set it up, I'm thinking we can use git tags for updates, so we can jump back and forth between updates and keep a history based on major comparisons we've decided on over time.
Activity
superkhau commentedon Apr 8, 2015
Actionable items:
mochafor benchmarks)superkhau commentedon Apr 8, 2015
@bajtos @raymondfeng @ritch Please edit the wiki and add your input. I will clean up the draft again once you guys add your parts.
raymondfeng commentedon Apr 9, 2015
@superkhau Do you want to check out https://github.com/bestiejs/benchmark.js?
superkhau commentedon Apr 9, 2015
Sure, I'll give it a shot tomorrow. This lib seems more geared towards performance testing.
bajtos commentedon Apr 9, 2015
I'll comment here, wiki is really bad for discussions (Google Docs works better for that IMO).
I would personally start with a full end-to-end benchmark covering loopback, strong-remoting, loopback-datasource-juggler and and a single connector. The connector under test should be configurable via the environment or command-line arguments, no code changes needed.
This should give us a good overview of which parts work well and which needs most improvement.
For the beginning, I would focus on the following scenarios:
Example usage:
The made-up numbers above show that we spend most time talking to the database, so that's probably the first thing to check when looking for places to optimize. The second biggest contributor is strong-remoting, especially when the load gets higher, so that would be the second place to investigate.
ritch commentedon Apr 9, 2015
+9999
ritch commentedon Apr 9, 2015
Overall I would love to have the tooling suggested by @bajtos. But I'm afraid that it will lead to scenarios like this fairly often:
ritch commentedon Apr 9, 2015
Instead we should have automated benchmarks run per module. We should be able to see if there is a decrease on a per PR basis. Otherwise I'm not sure its worth the effort.
This shouldn't be that hard to accomplish IMO.
34 remaining items
superkhau commentedon Jul 14, 2015
@raymondfeng @bajtos I feel we need to break this down into feasible tasks:
Leak detectionThe reason for this is to do leak detection on every function would be an exponential problem as each leak detection test needs to be written in a a fashion similar to unit testing. Running the tests longer also do a better job of detecting leaks. So my suggestion for the entire solution is:
The benefits of doing this way is that we can just add tests as need and the CI can run tests as it normally does without major infrastructure changes. This CI server should be separate from the regular CI server because these long running jobs would block other jobs from executing on the normal CI server.
benchmarkingTests run and issues are printed to
results.mdat the moment. Thats only for the functions we decide to test. From @bajtos suggestions, I feel we need to isolate which parts of LoopBack are slow (and compared to what).The problem is to get such a precise benchmark as suggested by @bajtos, we need to measure various parts of LB to get a number and compare it something. For example, if loopback-connector-mongodb db has a speed of 5 and mongoose has a 4 and mongodb native driver has a 3. We'd have to benchmark against all 3 to make a fair comparison (this is also 3 times more work since we're testing 3 modules now, instead of just ours). The nice thing is once the test is written once, we should be able to rerun them and compare the values over time.
The second problem I see is we need to persist this data somewhere. I mean text files work fine for now, but it would make more sense to have some sort of data store hold these values. Maybe a LoopBack app that accepts some log values and simply displays it?
As for CI, if we create a project named
loopback-benchmarks(replace the other one with the same name), with the same dir/tree structure as the leak detection example above, we can add benchmarks to specific functions over time too. And also run this on the same CI server as the leak detection (these two can share processing time since they both have long runnings jobs that don't need to be run all the time).The alternative to gather everything in one repo is to have a separate
benchmarksandleak-detectiondir inside each individual project (like we current do in https://github.com/strongloop/loopback-connector-mongodb), but we will be clogging up CI if we ran this on a regular basis.superkhau commentedon Jul 14, 2015
I agree with this approach as we're not just benchmarking randomly. Maybe start with benchmarking each connector vs it's alternative (ie.
loopback-connector-mongodbvsnativevsmongoose). Do the same for all 5 connectors. Then we can perform the same comparisons with juggler + connectors. Then once we have this, we can add in strong remoting, etc. This way, we can split this giant task into smaller stories and flush it out over time.Story 1 - mongo connector vs mongoose - 2 points
Story 2 - mongo connector vs mongo native - 2 points
Story 3 - juggler + mongo connector vs mongoose - 2 points
Story 4 - juggler + mongo connector vs mongo native - 2 points
Story 5 - remoting + juggler + mongo connector - 2 points
Story 6 - express + mongoose - 2 points
Sotry 7 - express + mongo native - 2 points
And so forth. We still have to decide where to persist the results to make it easy to compare the results.
superkhau commentedon Jul 14, 2015
Spike at https://github.com/strongloop/loopback-benchmark-mongodb
superkhau commentedon Jul 15, 2015
@bajtos @raymondfeng Please review https://github.com/strongloop/loopback-benchmark-mongodb/blob/master/results.md and debate whether its worth continuing. I only completed the time-split data for express-mongodb, will add express-mongoose and loopback-loopback-connector-mongodb if you guys think its worth continuing (so you can compare full request split-times).
bajtos commentedon Aug 22, 2015
If I am reading the benchmark results correctly, then our MongoDB connector is performing roughly as well as mongoose/mongodb and there isn't much space for optimization at our side. Is my reasoning correct?
I think it would be good to create a similar benchmark for one of our SQL connectors too. Then I would personally like to see how much overhead is added by juggler and strong-remoting. So yes, I think it makes sense to continue.
superkhau commentedon Aug 24, 2015
@bajtos Yes, you are reasoning correctly from the subset of data provided. I wanted to make sure this spike is providing the data you want to see before diving too deeply.
The plan is to do the same for the the other connectors, but I would like to finish up the MongoDB spike first to flush out any issues or questions that may need addressing. Adding the last two scenarios would make this a fairer comparision because
express-mongooseis a common strategy and of courseloopback-loopback-connector-mongodbwould be a better real world baseline.After this, we need to figure out a strategy for versioning. The way I set it up, I'm thinking we can use git tags for updates, so we can jump back and forth between updates and keep a history based on major comparisons we've decided on over time.
superkhau commentedon Apr 19, 2016
Closing this until we decide there is a need to work on benchmarking again.