Skip to content

Missing documentation on how to migrate codebase to support AccessToken scopes ? #3395

Closed

Description

@PaddyMann

After upgrading to Loopback 3.7, we've seen a load of errors relating to scopes.

Having searched the documentation, release notes, etc. I've failed to find anything mentioning how to address this breaking change. Am I missing something?

After many hours swearing and debugging, I've discovered that adding scopes as a VARCHAR(255) default '["DEFAULT"]' seems to address the issue, but I'm not confident that this is the best solution. I'd appreciate any advice on this!

Thanks :)

Activity

  1. ebarault commented on May 13, 2017

    @ebarault
    Contributor

    cc: @bajtos

    hi @PaddyMann,

    Support of AccessToken scopes was meant to be backward compatible, as this "DEFAULT" is added by default when missing (see here).

    would you have a sample to share to ease debugging?

    related:
    #3313
    #3299
    #3350

  2. PaddyMann commented on May 14, 2017

    @PaddyMann
    Author

    I'm not able to share the codebase or take the time out to diagnose as we're approaching a major deadline :|

    Here's the error I get without the scopes field added to the AccessToken table:

    Unhandled error for request /api/organizations/10/account?access_token=2arTPoLbV2Lx6RHPrsVaRcDIYzmfHVcZljeK526VFB23y9wHtxrOz9NfPjROMYh8 { Error: ER_BAD_FIELD_ERROR: Unknown column 'scopes' in 'field list'
        at Query.Sequence._packetToError (/node_modules/mysql/lib/protocol/sequences/Sequence.js:52:14)
        at Query.ErrorPacket (/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
        at Protocol._parsePacket (/node_modules/mysql/lib/protocol/Protocol.js:280:23)
        at Parser.write (/node_modules/mysql/lib/protocol/Parser.js:75:12)
        at Protocol.write (/node_modules/mysql/lib/protocol/Protocol.js:39:16)
        at Socket.<anonymous> (/node_modules/mysql/lib/Connection.js:103:28)
        at emitOne (events.js:96:13)
        at Socket.emit (events.js:189:7)
        at readableAddChunk (_stream_readable.js:176:18)
        at Socket.Readable.push (_stream_readable.js:134:10)
        at TCP.onread (net.js:551:20)
        --------------------
        at Protocol._enqueue (/node_modules/mysql/lib/protocol/Protocol.js:141:48)
        at PoolConnection.query (/node_modules/mysql/lib/Connection.js:208:25)
        at runQuery (/node_modules/loopback-connector-mysql/lib/mysql.js:186:16)
        at executeWithConnection (/node_modules/loopback-connector-mysql/lib/mysql.js:228:7)
        at Ping.onOperationComplete [as _callback] (/node_modules/mysql/lib/Pool.js:110:5)
        at Ping.Sequence.end (/node_modules/mysql/lib/protocol/sequences/Sequence.js:86:24)
        at Ping.Sequence.OkPacket (/node_modules/mysql/lib/protocol/sequences/Sequence.js:95:8)
        at Protocol._parsePacket (/node_modules/mysql/lib/protocol/Protocol.js:280:23)
        at Parser.write (/node_modules/mysql/lib/protocol/Parser.js:75:12)
        at Protocol.write (/node_modules/mysql/lib/protocol/Protocol.js:39:16)
        at Socket.<anonymous> (/node_modules/mysql/lib/Connection.js:103:28)
        at emitOne (events.js:96:13)
        at Socket.emit (events.js:189:7)
        at readableAddChunk (_stream_readable.js:176:18)
        at Socket.Readable.push (_stream_readable.js:134:10)
        at TCP.onread (net.js:551:20)
      code: 'ER_BAD_FIELD_ERROR',
      errno: 1054,
      sqlState: '42S22',
      index: 0 }
    

    Here's the error I get when I add a scopes field to the database but don't give it a default value:

    Unhandled error for request /api/organizations/21/account?access_token=9Z1bVSz37LEFqOPXj3AnscV7Wc5t0MMh5lTLFMMpM8XY1vVvht4FgklUoIbRLqDK { Error: could not create List from JSON string: ""
        at new List (/node_modules/loopback-datasource-juggler/lib/list.js:24:19)
        at List (/node_modules/loopback-datasource-juggler/lib/list.js:17:12)
        at ModelConstructor.ModelBaseClass._initProperties (/node_modules/loopback-datasource-juggler/lib/model.js:353:28)
        at ModelConstructor.ModelBaseClass (/node_modules/loopback-datasource-juggler/lib/model.js:58:8)
        at ModelConstructor (/node_modules/loopback-datasource-juggler/lib/model-builder.js:185:22)
        at ModelConstructor (/node_modules/loopback-datasource-juggler/lib/model-builder.js:185:22)
        at new ModelConstructor (/node_modules/loopback-datasource-juggler/lib/model-builder.js:185:22)
        at buildResult (/node_modules/loopback-datasource-juggler/lib/dao.js:2023:19)
        at /node_modules/loopback-datasource-juggler/lib/dao.js:2073:13
        at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
        at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
        at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
        at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
        at Function.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:122:5)
        at Function.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
        at Function.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:120:15)
        at Function.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
        at Function.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:120:15)
        at Function.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
        at Function.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:120:15)
        at Function.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
        at withNotify (/node_modules/loopback-datasource-juggler/lib/dao.js:2071:17)
        at /node_modules/loopback-datasource-juggler/lib/dao.js:2012:11
        at /node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:1007:9
        at eachOfArrayLike (/node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:940:9)
        at eachOf (/node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:990:5)
        at _asyncMap (/node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:1005:5)
        at Object.map (/node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:995:16)
        at allCb (/node_modules/loopback-datasource-juggler/lib/dao.js:2007:13)
        at /node_modules/loopback-connector/lib/sql.js:1206:7
        at /node_modules/loopback-datasource-juggler/lib/observer.js:172:22
        at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
        at MySQL.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:122:5)
        at MySQL.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
        at cbForWork (/node_modules/loopback-datasource-juggler/lib/observer.js:162:14)
        at /node_modules/loopback-connector/lib/sql.js:428:7
        at handleResponse (/node_modules/loopback-connector-mysql/lib/mysql.js:182:17)
        at Query._callback (/node_modules/loopback-connector-mysql/lib/mysql.js:193:7)
        at Query.Sequence.end (/node_modules/mysql/lib/protocol/sequences/Sequence.js:86:24)
        at Query._handleFinalResultPacket (/node_modules/mysql/lib/protocol/sequences/Query.js:137:8)
        at Query.EofPacket (/node_modules/mysql/lib/protocol/sequences/Query.js:121:8)
        at Protocol._parsePacket (/node_modules/mysql/lib/protocol/Protocol.js:280:23)
        at Parser.write (/node_modules/mysql/lib/protocol/Parser.js:75:12)
        at Protocol.write (/node_modules/mysql/lib/protocol/Protocol.js:39:16)
        at Socket.<anonymous> (/node_modules/mysql/lib/Connection.js:103:28)
        at emitOne (events.js:96:13)
        at Socket.emit (events.js:189:7)
        at readableAddChunk (_stream_readable.js:176:18)
        at Socket.Readable.push (_stream_readable.js:134:10)
        at TCP.onread (net.js:551:20) statusCode: 400 }
    

    As mentioned, it appears to be fixed if I set the default value to ["DEFAULT"] (in the database).

    Having scanned the PR, it seems that this should fix the problem without any unexpected side effects. Can you confirm?

    Given that you expected this change to be backward compatible, my guess is that the issue is MySQL specific.

  3. ebarault commented on May 14, 2017

    @ebarault
    Contributor

    Thanks for taking the time to at least provide the error messages @PaddyMann

    Having scanned the PR, it seems that this should fix the problem without any unexpected side effects. Can you confirm?

    Yes

    Given that you expected this change to be backward compatible, my guess is that the issue is MySQL specific.

    I think it might either be MySQL specific or SQL DBs specific. Any SQL DB that would throw an error if the AccessToken Table has not the scopes field.

  4. PaddyMann commented on May 14, 2017

    @PaddyMann
    Author

    Thank you v much for the speedy confirmation @ebarault - appreciated! 👍

  5. bajtos commented on May 22, 2017

    @bajtos
    Member

    @PaddyMann sorry for the confusion created by this change. You need to update your (MySQL) database and add the new scopes field. This can be done e.g. by running dataSource.autoupdate() function. See Migration in "Controlling data access" docs:

    To upgrade an application from an earlier version of LoopBack 3.x, follow these steps:

    • As common with semver-minor updates, update database schemas
      to accomodate any newly-added properties. For more information, see
      Creating a database schema from models (Auto-update).

    • If the application was already using a custom AccessToken.scopes
      property with a type different from an array, then update the relevant code
      to work with the type "array of strings".

  6. bajtos commented on May 22, 2017

    @bajtos
    Member

    I am closing this issue as answered, please reopen if you disagree.

  7. PaddyMann commented on May 22, 2017

    @PaddyMann
    Author

    Hi @bajtos

    Ah ha - I hadn't realized auto-update was needed to get table updates.

    We disabled all the auto-update and auto-migrate stuff early on as we found it to be buggy, and as a result really frickin' scary to have anywhere near our production database.

    I'd like to avoid turning it on here. If I want to manually make the updates to my database, where should I look to find to see the changes?

    Paddy

  8. bajtos commented on May 22, 2017

    @bajtos
    Member

    We disabled all the auto-update and auto-migrate stuff early on as we found it to be buggy, and as a result really frickin' scary to have anywhere near our production database.

    I totally understand that, I would not want to run any automagic stuff on a production database either.

    If I want to manually make the updates to my database, where should I look to find to see the changes?

    Hmm, I am afraid this is something we haven't addressed yet. In theory, it should be reasonably easy to change the autoupdate method to build SQL statements and print them instead of executing. That way we can create a migration SQL script that can be executed on production database server while upgrading to a newer app version. (See Martin Fowler's All database changes are migrations.)

    Here is what I would do myself:

    1. Create a local copy of the database using the schema you have in production right now
    2. Run autoupdate on this local database
    3. Find out what has changed. One option is run autoupdate with DEBUG logs enabled, that way the connector should print all SQL statements it executes. Another option is to compare the updated schema with the original one. A quick search found MySQL Workbench and this StackExchange question.
  9. PaddyMann commented on May 25, 2017

    @PaddyMann
    Author

    Thanks @bajtos for the suggested workaround.

    Given the agreement that auto-magic shouldn't be relied on, it'd be great to have a clearer process here.

    e.g.: Rather than relying on end-users to make this change (or fail to, and lose hours to trying to understand and fix), you could make the above steps part of your release process: Run the scripts, identify the changes made on supported databases, and include them in a release note.

  10. bajtos commented on May 25, 2017

    @bajtos
    Member

    I see your point. My concern is that obtaining that SQL script for each supported database would be require too much effort on our side. We would basically run the auto-magic routine for each database, thus it won't remove the reliance on autoupdate method.

    I'd much rather spend that time writing a small command that any LoopBack user can run to get a migration SQL script for the changes made in the app recently, be it new columns added by LoopBack core or new models/properties added by the app. I am envisioning something like lb autoupdate db --dry.

    cc @raymondfeng @ritch @kjdelisle

  11. PaddyMann commented on May 26, 2017

    @PaddyMann
    Author

    Fair enough! If there is such a tool, it would be great to reference in the release note when it needs to be called (or even with every release note).

    Can we reopen the issue?

  12. added this to the Sprint 36 milestone on May 26, 2017
  13. bajtos commented on May 26, 2017

    @bajtos
    Member

    Let's discuss the new tool in a new issue to keep the discussion more focused: #3422

  14. shaheero commented on May 28, 2017

    @shaheero

    TL;DR
    For SQL, add two new columns in your custom access tokens table:

    1. "scopes" with type VARCHAR and length 255
    2. "principalType" with type VARCHAR and length 255

    By the way, in the docs it says "Note: Alternatively, you can put these lines in the server.js". I have tried that, but unless the code is put into "auth" in middleware.json, the code in server.js on its own will give this error:
    "Cannot call AccessToken.findById(). The findById method has not been setup. The PersistedModel has not been correctly attached to a DataSource!"

  15. acrodrig commented on May 30, 2017

    @acrodrig

    @shaheero following this issue as it bit as I was upgrading Loopback to version 3.8 from 3.0. Looking through access-token.json I did not find a principalType column. Can you confirm that this is needed? Thanks much.

  16. shaheero commented on May 30, 2017

    @shaheero

    @acrodrig It is required by user.json, which access-token.json has a belongTo relationship with.
    In short, you have to use the polymorphic relations (including the discriminator principalType) in your custom access token model and all your extended user models if you have several models extending the built-in User model. In this case, you have to add principalType column to your database.

    As per the docs, if several models are extending the built-in User model, " you must modify the relations between the users models and the AccessToken models to allow a single AccessToken model to host access tokens for multiple types of users while at the same time allowing each user model instance to be linked to unique related access tokens."

    Otherwise, you will get this error:

    Error: polymorphic belongsTo relation: user requires param polymorphic.discriminator when param polymorphic.foreignKey is provided

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions