Skip to content

Support nested values in config overrides #37

Closed

Description

@bajtos

At the moment, loopback-boot does not support non-value types like arrays or objects in .local.json and .js config files.

The reason for that is with non-value types, there are multiple possible operations that one may want to perform:

  1. override the whole value, replace one object/array with another
  2. override the item at a given index in an array or replace the value of the given key in an object
  3. add items to an array or key/values to an object
  4. remove items from an array or delete key/values from an object

We should come up with a solution flexible enough to support most of these use-cases. Note that they are recursive, e.g. one may want to override the object value of the ssl key in object stored under the transport key.

Activity

  1. raymondfeng commented on Aug 18, 2014

    @raymondfeng
    Member

    In a plain json configuration file, it's impossible to tell if the developers' intention is 1, 2, 3, or 4 unless we introduce extra meta, in which case, the config file will become a diff/patch. IMO, overriding the whole value is a reasonable solution as long as we document it.

  2. bajtos commented on Aug 18, 2014

    @bajtos
    MemberAuthor

    IMO, overriding the whole value is a reasonable solution as long as we document it.

    In my opinion, the most frequent use-case for complex values is customising datasource configuration. In which case it makes more sense to add/override key-values instead of replacing the whole object.

    Example: to specify different MySQL settings for the production environment, one wants to add or change host, port, username and password. Other options (e.g. connnector: mysql) remain without change.

    If we are going to support only one of 1-4, then I would definitely like to get more feedback from the community before making the decision.

  3. raymondfeng commented on Aug 18, 2014

    @raymondfeng
    Member

    I'm fine with merging too.

    See https://github.com/strongloop/loopback-datasource-juggler/blob/master/lib/utils.js#L138. We use it to merge model settings.

  4. britztopher commented on Aug 18, 2014

    @britztopher

    Can we reuse this merge function from datasource-juggler to be used in loopback-boot for nested objects? If so, I can try to make this work, just need to know where to start looking for where in the loopback code this should go. Im guessing its somewhere in the executor or config-loader.js files.

  5. matejc commented on Aug 25, 2014

    @matejc

    I am using this one-level merging trick, it's not ideal but works for my usecase
    matejc@4c88a5d

  6. raymondfeng commented on Aug 25, 2014

    @raymondfeng
    Member

    @bajtos What's your take? We need a solution.

  7. bajtos commented on Aug 25, 2014

    @bajtos
    MemberAuthor

    @bajtos What's your take? We need a solution.

    How about supporting two flavours of datasources.*.js files:

    • a file exporting a config object (i.e. a dynamic version of *.json), only value-type values are supported (no change from the current state)

    • a file exporting a function that will be called to modify the config object (a new feature)

      module.exports = function(config) {
        config.mail.transports = [{
          // configure SMTP
        }];
      
        switch (config.db.connector) {
          case 'mysql':
            config.db.host = process.env.MYSQL_HOST;
            config.db.port = process.env.MYSQL_PORT;
            break;
        }
        // etc.
      });

    That way we don't have to worry whether the JSON format is expressing the intention (merge/replace) clearly, and the users can apply whatever changes they need.

  8. self-assigned this
    on Sep 5, 2014
  9. glesage commented on Sep 16, 2014

    @glesage

    So far when I attempt to change some variables (for ex, the password) like this I am having to do:

    config.datasources.db.connector.settings.password = process.env.MY_PASS;
    config.datasources.db.settings.password = process.env.MY_PASS;

    Is this correct? It does not seem to work for me /: It is the correct spot where the password is stored but when using the DB, loopback fails with:

    {
      "error": {
        "name": "Error",
        "status": 500,
        "message": "ER_ACCESS_DENIED_ERROR: Access denied for user 'starchup'@'localhost' (using password: YES)",
        "stack": "Error: ER_ACCESS_DENIED_ERROR: Access denied for user 'starchup'@'localhost' (using password: YES)\n    at Handshake.Sequence._packetToError (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/sequences/Sequence.js:48:14)\n    at Handshake.ErrorPacket (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/sequences/Handshake.js:101:18)\n    at Protocol._parsePacket (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:271:23)\n    at Parser.write (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Parser.js:77:12)\n    at Protocol.write (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:39:16)\n    at Socket.<anonymous> (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Connection.js:82:28)\n    at Socket.EventEmitter.emit (events.js:107:17)\n    at readableAddChunk (_stream_readable.js:159:16)\n    at Socket.Readable.push (_stream_readable.js:126:10)\n    at TCP.onread (net.js:514:20)\n    --------------------\n    at Protocol._enqueue (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:135:48)\n    at Protocol.handshake (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:52:41)\n    at PoolConnection.connect (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Connection.js:109:18)\n    at Pool.getConnection (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Pool.js:42:23)\n    at MySQL.query (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/lib/mysql.js:167:10)\n    at MySQL.all (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-connector-mysql/lib/mysql.js:580:8)\n    at Function.find (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback-datasource-juggler/lib/dao.js:735:34)\n    at Function.ACL.checkAccessForContext (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback/lib/models/acl.js:397:8)\n    at Function.Model.checkAccess (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback/lib/models/model.js:281:12)\n    at Function.<anonymous> (/home/jeff/Dropbox/Starchup/dev/starchup-central/node_modules/loopback/lib/application.js:319:13)",
        "code": "ER_ACCESS_DENIED_ERROR",
        "errno": 1045,
        "sqlState": "28000",
        "fatal": true,
        "INVALID_EXPRESSION_ERR": 51,
        "TYPE_ERR": 52
      }
    }
    

    Please advise,
    THanks!

  10. bajtos commented on Sep 16, 2014

    @bajtos
    MemberAuthor

    @glesage connector gets the settings object from datasource settings.

    // server/datasources.local.js
    module.exports = {
      "db" : {
        "password": process.env.MY_PASS
      }
    }
    
  11. glesage commented on Sep 16, 2014

    @glesage

    oh alright thks, works perfect indeed (:

  12. frankcarey commented on Oct 1, 2014

    @frankcarey

    Have you guys considered using https://github.com/lorenwest/node-config instead of rolling your own, or at least as a base? It supports hierarchical configurations (something I needed to have loopback-passport working properly in multiple environments.)

  13. bajtos commented on Oct 2, 2014

    @bajtos
    MemberAuthor

    Have you guys considered using https://github.com/lorenwest/node-config instead of rolling your own, or at least as a base?

    We did not investigate this option thoroughly. At the first glance, node-config does provide a fine-grained control over how the configs from different sources are merged.

    It supports hierarchical configurations (something I needed to have loopback-passport working properly in multiple environments.)

    loopback-boot supports hierarchical configurations too. Can you provide an example of what you are not able to achieve with our current solution?

  14. bajtos commented on Oct 8, 2014

    @bajtos
    MemberAuthor

    #52 implements a solution where both Object and Array values from .json files are recursively merged together.

  15. bajtos commented on Oct 16, 2014

    @bajtos
    MemberAuthor

    The feature is available as of v2.1.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions