Skip to content

Benchmarking #1291

Closed
Closed

Description

@superkhau

I would like to open a discussion on our strategy for applying benchmarks across various LoopBack repos as discussed during our meeting @raymondfeng and @ritch.

/cc @bajtos

Activity

  1. self-assigned this
    on Apr 8, 2015
  2. superkhau commented on Apr 8, 2015

    @superkhau
    ContributorAuthor

    Actionable items:

    • Create Wiki to gather discussion ideas
    • Create a draft of the benchmark wiki with the contents from the meeting with @raymondfeng and @ritch
    • Get draft feedback from
    • Spike on possible tooling to automate benchmarking (something like mocha for benchmarks)


    • Create leak detection prototype using MongoDB connector
    • Create benchmark prototype using MongoDB connector
  3. superkhau commented on Apr 8, 2015

    @superkhau
    ContributorAuthor

    @bajtos @raymondfeng @ritch Please edit the wiki and add your input. I will clean up the draft again once you guys add your parts.

  4. raymondfeng commented on Apr 9, 2015

    @raymondfeng
    Member
  5. superkhau commented on Apr 9, 2015

    @superkhau
    ContributorAuthor

    Sure, I'll give it a shot tomorrow. This lib seems more geared towards performance testing.

  6. bajtos commented on Apr 9, 2015

    @bajtos
    Member

    I'll comment here, wiki is really bad for discussions (Google Docs works better for that IMO).

    I would personally start with a full end-to-end benchmark covering loopback, strong-remoting, loopback-datasource-juggler and and a single connector. The connector under test should be configurable via the environment or command-line arguments, no code changes needed.

    This should give us a good overview of which parts work well and which needs most improvement.

    For the beginning, I would focus on the following scenarios:

    1. Detect memory leaks
    2. Benchmark requests per second, measure how much time is needed by individual parts: strong-remoting, loopback+juggler, connector

    Example usage:

    $ gulp memtest
    (...runnning...)
    No memory leaks detected
    
    $ gulp benchmark mysql
    (...running...)
    Requests per second:
      median: 1700
      stddev: 300
    
    Times per request (wall time in microseconds)
                 total      strong-remoting      loopback     connector
    median        588                  200            80            308       
    90p           812                  300           200            312
    99p           932                  350           240            342

    The made-up numbers above show that we spend most time talking to the database, so that's probably the first thing to check when looking for places to optimize. The second biggest contributor is strong-remoting, especially when the load gets higher, so that would be the second place to investigate.

  7. ritch commented on Apr 9, 2015

    @ritch
    Member

    Google Docs works better for that IMO

    +9999

  8. ritch commented on Apr 9, 2015

    @ritch
    Member

    Overall I would love to have the tooling suggested by @bajtos. But I'm afraid that it will lead to scenarios like this fairly often:

    1. foo submits a PR to strong-remoting that decreases the benchmark numbers
    2. the benchmarks are not run (since its takes a bit of time to do and it isnt automatic)... so impact in performance is completely missed
    3. the PR is released
    4. someone notices the performance decrease by running the lb benchmark with the new version of remoting
    5. there is a discussion on how to revert the new feature added by foobar
    6. we can't since it would be a breaking change
    7. :(
  9. ritch commented on Apr 9, 2015

    @ritch
    Member

    Instead we should have automated benchmarks run per module. We should be able to see if there is a decrease on a per PR basis. Otherwise I'm not sure its worth the effort.

    This shouldn't be that hard to accomplish IMO.

  10. 34 remaining items

  11. superkhau commented on Jul 14, 2015

    @superkhau
    ContributorAuthor

    @raymondfeng @bajtos I feel we need to break this down into feasible tasks:

    Leak detection

    • Create a new project named `loopback-leak-detection
    • Individual package that can be run on demand
    • Detect leaks for new functions added over time on an as-needed basis
    • Integration style tests (ie. maybe a dir for each functional area we want to check for leaks)

    The reason for this is to do leak detection on every function would be an exponential problem as each leak detection test needs to be written in a a fashion similar to unit testing. Running the tests longer also do a better job of detecting leaks. So my suggestion for the entire solution is:

    1. New project to consolidate tests for all loopback projects (this project will do leak detection for various parts of LB, strong-remoting, connectors, etc)
    2. Pushing this project to GitHub should trigger the CI to run tests (which can take hours depending on the number of iterations we decide on)
    loopback-leak-detection
    ├─loopback
    ├─loopback-connector-mongo
    ├─strong-remoting
    └─...
    

    The benefits of doing this way is that we can just add tests as need and the CI can run tests as it normally does without major infrastructure changes. This CI server should be separate from the regular CI server because these long running jobs would block other jobs from executing on the normal CI server.

    benchmarking

    Tests run and issues are printed to results.md at the moment. Thats only for the functions we decide to test. From @bajtos suggestions, I feel we need to isolate which parts of LoopBack are slow (and compared to what).

    1. Express speed is the benchmark (because this is probably as fast as possible)
    2. Compare our times against Express (or similar middleware using express)

    The problem is to get such a precise benchmark as suggested by @bajtos, we need to measure various parts of LB to get a number and compare it something. For example, if loopback-connector-mongodb db has a speed of 5 and mongoose has a 4 and mongodb native driver has a 3. We'd have to benchmark against all 3 to make a fair comparison (this is also 3 times more work since we're testing 3 modules now, instead of just ours). The nice thing is once the test is written once, we should be able to rerun them and compare the values over time.

    The second problem I see is we need to persist this data somewhere. I mean text files work fine for now, but it would make more sense to have some sort of data store hold these values. Maybe a LoopBack app that accepts some log values and simply displays it?

    As for CI, if we create a project named loopback-benchmarks (replace the other one with the same name), with the same dir/tree structure as the leak detection example above, we can add benchmarks to specific functions over time too. And also run this on the same CI server as the leak detection (these two can share processing time since they both have long runnings jobs that don't need to be run all the time).


    The alternative to gather everything in one repo is to have a separate benchmarks and leak-detection dir inside each individual project (like we current do in https://github.com/strongloop/loopback-connector-mongodb), but we will be clogging up CI if we ran this on a regular basis.

  12. superkhau commented on Jul 14, 2015

    @superkhau
    ContributorAuthor

    From my POV, I'd like to see a benchmark that will tell us which part of LoopBack is the slowest one (see #1291 (comment)). For example, start with the method "find with options" and gather timing data split across module boundaries: express, strong-remoting, loopback, loopback-datasource-juggler, connector, SQL backend.

    I agree with this approach as we're not just benchmarking randomly. Maybe start with benchmarking each connector vs it's alternative (ie. loopback-connector-mongodb vs native vs mongoose). Do the same for all 5 connectors. Then we can perform the same comparisons with juggler + connectors. Then once we have this, we can add in strong remoting, etc. This way, we can split this giant task into smaller stories and flush it out over time.

    Story 1 - mongo connector vs mongoose - 2 points
    Story 2 - mongo connector vs mongo native - 2 points
    Story 3 - juggler + mongo connector vs mongoose - 2 points
    Story 4 - juggler + mongo connector vs mongo native - 2 points
    Story 5 - remoting + juggler + mongo connector - 2 points
    Story 6 - express + mongoose - 2 points
    Sotry 7 - express + mongo native - 2 points

    And so forth. We still have to decide where to persist the results to make it easy to compare the results.

  13. superkhau commented on Jul 14, 2015

    @superkhau
    ContributorAuthor
  14. ghost removed on Jul 14, 2015
  15. ghost added on Jul 14, 2015
  16. superkhau commented on Jul 15, 2015

    @superkhau
    ContributorAuthor

    @bajtos @raymondfeng Please review https://github.com/strongloop/loopback-benchmark-mongodb/blob/master/results.md and debate whether its worth continuing. I only completed the time-split data for express-mongodb, will add express-mongoose and loopback-loopback-connector-mongodb if you guys think its worth continuing (so you can compare full request split-times).

  17. bajtos commented on Aug 22, 2015

    @bajtos
    Member

    If I am reading the benchmark results correctly, then our MongoDB connector is performing roughly as well as mongoose/mongodb and there isn't much space for optimization at our side. Is my reasoning correct?

    if you guys think its worth continuing (so you can compare full request split-times).

    I think it would be good to create a similar benchmark for one of our SQL connectors too. Then I would personally like to see how much overhead is added by juggler and strong-remoting. So yes, I think it makes sense to continue.

  18. superkhau commented on Aug 24, 2015

    @superkhau
    ContributorAuthor

    @bajtos Yes, you are reasoning correctly from the subset of data provided. I wanted to make sure this spike is providing the data you want to see before diving too deeply.

    The plan is to do the same for the the other connectors, but I would like to finish up the MongoDB spike first to flush out any issues or questions that may need addressing. Adding the last two scenarios would make this a fairer comparision because express-mongoose is a common strategy and of course loopback-loopback-connector-mongodb would be a better real world baseline.

    After this, we need to figure out a strategy for versioning. The way I set it up, I'm thinking we can use git tags for updates, so we can jump back and forth between updates and keep a history based on major comparisons we've decided on over time.

  19. superkhau commented on Apr 19, 2016

    @superkhau
    ContributorAuthor

    Closing this until we decide there is a need to work on benchmarking again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions