Skip to content

Data Importing #80

Closed
Closed

Description

@ritch

There are several use cases for importing data into a loopback data source. I've listed the ones that I can think of, feel free to add any.

  1. As a developer working on a new project, I need test data to work with. This could be for integration tests, prototyping, or even modeling data.
  2. As a developer working on an existing project with exsiting data, I need to constantly update my local development data with that which mirrors, closely mirrors, or simulates prodution data.
  3. LoopBack internals require data to be imported during app startup. Although it is important to note that the core could be modified in a way that does not depend on bootstrapped data.

Here is a basic example of an API and JSON format that could make implementing these use cases fairly straightforward and simple.

var app = loopback();
var db = app.dataSource('db', {connector: loopback.Memory});
var Car = app.model('Car', {dataSource: 'db'});
var Dealership = app.model('Dealership', {dataSource: 'db'});
var Option = app.model('Option', {dataSource: 'db'});
var Make = app.model('Make', {dataSource: 'db'});

Dealership.hasMany(Car);
Car.hasMany(Option);
Make.hasMany(Car);

// data.json
{
  data: {
    dealerships: [
      {
        name: 'acme cars',
        $cars: [
          {model: '500', $make: '#fiat', $options: ['#leather', '#ac']},
          {model: 's4', $make: '#audi', $options: '.audi-options'},
          {model: 'm3', $make: '#bmw'},
        ]
      }
    ],
    makes: [
      {name: 'audi', $$id: 'audi'}
      {name: 'fiat', $$id: 'fiat'}
      {name: 'bmw', $$id: 'bmw'}
    ],
    options: [
      {name: 'leather', $$id: 'leather'},
      {name: 'ac', $$id: 'ac'},
      {name: 'audi option 1', $$class: 'audi-options'},
      {name: 'audi option 2', $$class: 'audi-options'}
    ]
  }
}

var testData = app.createDataSet({
  file: 'data.json', //or data: someDataObj
});

testData.import({drop: true}, function (err) {
  console.log(err || 'Data imported');
});

A bit of reference for the JSON format:

  • $$ - indicates a reference only for use by the importer, allowing references / relations to be created before the actual object is inserted and id generated
  • $$id - for referencing a single instance
  • $$class - for referencing several instances
  • $ - indicates a special property. For now, this is used only for relations
  • $myRleationProperty - indicates the value for this key is a string or array of strings referencing classes or ids (see above).
  • .my-class-name - similar to how classes work in CSS: allows you to reference a set of objects tagged with a $$class: 'my-class-name.
  • #my-id - similar to how ids work in CSS: allows you to reference a single object by id

Activity

  1. bajtos commented on Dec 4, 2013

    @bajtos
    Member

    @ritch Could you please elaborate a little bit more on the meaning of dolare and double-dolar keys?

    It seems that $make is depending on name values, does your design allow to configure the lookup (e.g. $owner should look up a user by login instead of owner by name)?

    Perhaps we can auto-detect string (path to JSON file) vs. object (in-memory data), so that app.createDataSet takes a single argument?

    There are two functions needed to perform the import (app.createDataSet and testData.import), why not a single function app.importDataSet(options, callback)?

  2. ritch commented on Dec 4, 2013

    @ritch
    MemberAuthor

    I'd like to have room to create more methods on the data set itself. It would also be nice if the DataSet was an event emitter (perhaps a writeable stream?). This is just a best practice / idiom I see in node core. Eg. server = http.createServer().

  3. ritch commented on Dec 4, 2013

    @ritch
    MemberAuthor

    @bajtos - updated the original post with a brief reference

  4. raymondfeng commented on Dec 4, 2013

    @raymondfeng
    Member

    Importing data set for individual models seems to be straightforward. The connected/related data set are tricky, for example, a customer and a list of orders placed by the customer.

  5. bajtos commented on Dec 4, 2013

    @bajtos
    Member

    The connected/related data set are tricky, for example, a customer and a list of orders placed by the customer.

    I believe Ritchie's proposal solves this problem in rather elegant way. "orders placed by a customer" is the same relationship as "cars made by a manufacturer", you can find the latter in the sample code.

  6. Schoonology commented on Dec 4, 2013

    @Schoonology
    Contributor

    @ritch I agree that a separate data type for importable data makes sense, but I would invert the relationship somewhat (if practical - thinking out loud here). I would look more to:

    var set = loopback.DataSet.loadFromFile('data.json'); // Static method to load, could use load(stringOrObj), etc.
    app.importDataSet(set, { drop: true }, function callback(err) { ... });
    

    The DataSet, then, is just a decorated representation of importable data, able to be (de)serialized and passed around. The application knows how to import that decorated version into a set of models.

    This grants all kinds of cool powers, too. Replication, anyone? What about sync? Fun times.

  7. added this to the milestone on Feb 28, 2014
  8. modified the milestone: #Rel lb 2.0.0 on Sep 30, 2014
  9. changed the title [-]RFC - Data Importing[/-] [+]Data Importing[/+] on Feb 4, 2016
  10. bajtos commented on Apr 7, 2016

    @bajtos
    Member

    As part of our issue triage process, I am closing items that have been inactive for an extended period of time. If anybody thinks the feature proposed here is still relevant and should be prioritised, then please leave a comment to let us know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions