You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Missing documentation on how to migrate codebase to support AccessToken scopes ? #3395
After upgrading to Loopback 3.7, we've seen a load of errors relating to scopes.
Having searched the documentation, release notes, etc. I've failed to find anything mentioning how to address this breaking change. Am I missing something?
After many hours swearing and debugging, I've discovered that adding scopes as a VARCHAR(255) default '["DEFAULT"]' seems to address the issue, but I'm not confident that this is the best solution. I'd appreciate any advice on this!
I'm not able to share the codebase or take the time out to diagnose as we're approaching a major deadline :|
Here's the error I get without the scopes field added to the AccessToken table:
Unhandled error for request /api/organizations/10/account?access_token=2arTPoLbV2Lx6RHPrsVaRcDIYzmfHVcZljeK526VFB23y9wHtxrOz9NfPjROMYh8 { Error: ER_BAD_FIELD_ERROR: Unknown column 'scopes' in 'field list'
at Query.Sequence._packetToError (/node_modules/mysql/lib/protocol/sequences/Sequence.js:52:14)
at Query.ErrorPacket (/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
at Protocol._parsePacket (/node_modules/mysql/lib/protocol/Protocol.js:280:23)
at Parser.write (/node_modules/mysql/lib/protocol/Parser.js:75:12)
at Protocol.write (/node_modules/mysql/lib/protocol/Protocol.js:39:16)
at Socket.<anonymous> (/node_modules/mysql/lib/Connection.js:103:28)
at emitOne (events.js:96:13)
at Socket.emit (events.js:189:7)
at readableAddChunk (_stream_readable.js:176:18)
at Socket.Readable.push (_stream_readable.js:134:10)
at TCP.onread (net.js:551:20)
--------------------
at Protocol._enqueue (/node_modules/mysql/lib/protocol/Protocol.js:141:48)
at PoolConnection.query (/node_modules/mysql/lib/Connection.js:208:25)
at runQuery (/node_modules/loopback-connector-mysql/lib/mysql.js:186:16)
at executeWithConnection (/node_modules/loopback-connector-mysql/lib/mysql.js:228:7)
at Ping.onOperationComplete [as _callback] (/node_modules/mysql/lib/Pool.js:110:5)
at Ping.Sequence.end (/node_modules/mysql/lib/protocol/sequences/Sequence.js:86:24)
at Ping.Sequence.OkPacket (/node_modules/mysql/lib/protocol/sequences/Sequence.js:95:8)
at Protocol._parsePacket (/node_modules/mysql/lib/protocol/Protocol.js:280:23)
at Parser.write (/node_modules/mysql/lib/protocol/Parser.js:75:12)
at Protocol.write (/node_modules/mysql/lib/protocol/Protocol.js:39:16)
at Socket.<anonymous> (/node_modules/mysql/lib/Connection.js:103:28)
at emitOne (events.js:96:13)
at Socket.emit (events.js:189:7)
at readableAddChunk (_stream_readable.js:176:18)
at Socket.Readable.push (_stream_readable.js:134:10)
at TCP.onread (net.js:551:20)
code: 'ER_BAD_FIELD_ERROR',
errno: 1054,
sqlState: '42S22',
index: 0 }
Here's the error I get when I add a scopes field to the database but don't give it a default value:
Unhandled error for request /api/organizations/21/account?access_token=9Z1bVSz37LEFqOPXj3AnscV7Wc5t0MMh5lTLFMMpM8XY1vVvht4FgklUoIbRLqDK { Error: could not create List from JSON string: ""
at new List (/node_modules/loopback-datasource-juggler/lib/list.js:24:19)
at List (/node_modules/loopback-datasource-juggler/lib/list.js:17:12)
at ModelConstructor.ModelBaseClass._initProperties (/node_modules/loopback-datasource-juggler/lib/model.js:353:28)
at ModelConstructor.ModelBaseClass (/node_modules/loopback-datasource-juggler/lib/model.js:58:8)
at ModelConstructor (/node_modules/loopback-datasource-juggler/lib/model-builder.js:185:22)
at ModelConstructor (/node_modules/loopback-datasource-juggler/lib/model-builder.js:185:22)
at new ModelConstructor (/node_modules/loopback-datasource-juggler/lib/model-builder.js:185:22)
at buildResult (/node_modules/loopback-datasource-juggler/lib/dao.js:2023:19)
at /node_modules/loopback-datasource-juggler/lib/dao.js:2073:13
at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
at Function.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:122:5)
at Function.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
at Function.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:120:15)
at Function.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
at Function.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:120:15)
at Function.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
at Function.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:120:15)
at Function.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
at withNotify (/node_modules/loopback-datasource-juggler/lib/dao.js:2071:17)
at /node_modules/loopback-datasource-juggler/lib/dao.js:2012:11
at /node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:1007:9
at eachOfArrayLike (/node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:940:9)
at eachOf (/node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:990:5)
at _asyncMap (/node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:1005:5)
at Object.map (/node_modules/loopback-datasource-juggler/node_modules/async/dist/async.js:995:16)
at allCb (/node_modules/loopback-datasource-juggler/lib/dao.js:2007:13)
at /node_modules/loopback-connector/lib/sql.js:1206:7
at /node_modules/loopback-datasource-juggler/lib/observer.js:172:22
at doNotify (/node_modules/loopback-datasource-juggler/lib/observer.js:99:49)
at MySQL.ObserverMixin._notifyBaseObservers (/node_modules/loopback-datasource-juggler/lib/observer.js:122:5)
at MySQL.ObserverMixin.notifyObserversOf (/node_modules/loopback-datasource-juggler/lib/observer.js:97:8)
at cbForWork (/node_modules/loopback-datasource-juggler/lib/observer.js:162:14)
at /node_modules/loopback-connector/lib/sql.js:428:7
at handleResponse (/node_modules/loopback-connector-mysql/lib/mysql.js:182:17)
at Query._callback (/node_modules/loopback-connector-mysql/lib/mysql.js:193:7)
at Query.Sequence.end (/node_modules/mysql/lib/protocol/sequences/Sequence.js:86:24)
at Query._handleFinalResultPacket (/node_modules/mysql/lib/protocol/sequences/Query.js:137:8)
at Query.EofPacket (/node_modules/mysql/lib/protocol/sequences/Query.js:121:8)
at Protocol._parsePacket (/node_modules/mysql/lib/protocol/Protocol.js:280:23)
at Parser.write (/node_modules/mysql/lib/protocol/Parser.js:75:12)
at Protocol.write (/node_modules/mysql/lib/protocol/Protocol.js:39:16)
at Socket.<anonymous> (/node_modules/mysql/lib/Connection.js:103:28)
at emitOne (events.js:96:13)
at Socket.emit (events.js:189:7)
at readableAddChunk (_stream_readable.js:176:18)
at Socket.Readable.push (_stream_readable.js:134:10)
at TCP.onread (net.js:551:20) statusCode: 400 }
As mentioned, it appears to be fixed if I set the default value to ["DEFAULT"] (in the database).
Having scanned the PR, it seems that this should fix the problem without any unexpected side effects. Can you confirm?
Given that you expected this change to be backward compatible, my guess is that the issue is MySQL specific.
@PaddyMann sorry for the confusion created by this change. You need to update your (MySQL) database and add the new scopes field. This can be done e.g. by running dataSource.autoupdate() function. See Migration in "Controlling data access" docs:
To upgrade an application from an earlier version of LoopBack 3.x, follow these steps:
If the application was already using a custom AccessToken.scopes
property with a type different from an array, then update the relevant code
to work with the type "array of strings".
Ah ha - I hadn't realized auto-update was needed to get table updates.
We disabled all the auto-update and auto-migrate stuff early on as we found it to be buggy, and as a result really frickin' scary to have anywhere near our production database.
I'd like to avoid turning it on here. If I want to manually make the updates to my database, where should I look to find to see the changes?
We disabled all the auto-update and auto-migrate stuff early on as we found it to be buggy, and as a result really frickin' scary to have anywhere near our production database.
I totally understand that, I would not want to run any automagic stuff on a production database either.
If I want to manually make the updates to my database, where should I look to find to see the changes?
Hmm, I am afraid this is something we haven't addressed yet. In theory, it should be reasonably easy to change the autoupdate method to build SQL statements and print them instead of executing. That way we can create a migration SQL script that can be executed on production database server while upgrading to a newer app version. (See Martin Fowler's All database changes are migrations.)
Here is what I would do myself:
Create a local copy of the database using the schema you have in production right now
Run autoupdate on this local database
Find out what has changed. One option is run autoupdate with DEBUG logs enabled, that way the connector should print all SQL statements it executes. Another option is to compare the updated schema with the original one. A quick search found MySQL Workbench and this StackExchange question.
Given the agreement that auto-magic shouldn't be relied on, it'd be great to have a clearer process here.
e.g.: Rather than relying on end-users to make this change (or fail to, and lose hours to trying to understand and fix), you could make the above steps part of your release process: Run the scripts, identify the changes made on supported databases, and include them in a release note.
I see your point. My concern is that obtaining that SQL script for each supported database would be require too much effort on our side. We would basically run the auto-magic routine for each database, thus it won't remove the reliance on autoupdate method.
I'd much rather spend that time writing a small command that any LoopBack user can run to get a migration SQL script for the changes made in the app recently, be it new columns added by LoopBack core or new models/properties added by the app. I am envisioning something like lb autoupdate db --dry.
TL;DR
For SQL, add two new columns in your custom access tokens table:
"scopes" with type VARCHAR and length 255
"principalType" with type VARCHAR and length 255
By the way, in the docs it says "Note: Alternatively, you can put these lines in the server.js". I have tried that, but unless the code is put into "auth" in middleware.json, the code in server.js on its own will give this error:
"Cannot call AccessToken.findById(). The findById method has not been setup. The PersistedModel has not been correctly attached to a DataSource!"
@shaheero following this issue as it bit as I was upgrading Loopback to version 3.8 from 3.0. Looking through access-token.json I did not find a principalType column. Can you confirm that this is needed? Thanks much.
@acrodrig It is required by user.json, which access-token.json has a belongTo relationship with.
In short, you have to use the polymorphic relations (including the discriminator principalType) in your custom access token model and all your extended user models if you have several models extending the built-in User model. In this case, you have to add principalType column to your database.
As per the docs, if several models are extending the built-in User model, " you must modify the relations between the users models and the AccessToken models to allow a single AccessToken model to host access tokens for multiple types of users while at the same time allowing each user model instance to be linked to unique related access tokens."
Otherwise, you will get this error:
Error: polymorphic belongsTo relation: user requires param polymorphic.discriminator when param polymorphic.foreignKey is provided
Activity
ebarault commentedon May 13, 2017
cc: @bajtos
hi @PaddyMann,
Support of AccessToken scopes was meant to be backward compatible, as this
"DEFAULT"is added by default when missing (see here).would you have a sample to share to ease debugging?
related:
#3313
#3299
#3350
PaddyMann commentedon May 14, 2017
I'm not able to share the codebase or take the time out to diagnose as we're approaching a major deadline :|
Here's the error I get without the
scopesfield added to theAccessTokentable:Here's the error I get when I add a
scopesfield to the database but don't give it a default value:As mentioned, it appears to be fixed if I set the default value to
["DEFAULT"](in the database).Having scanned the PR, it seems that this should fix the problem without any unexpected side effects. Can you confirm?
Given that you expected this change to be backward compatible, my guess is that the issue is MySQL specific.
ebarault commentedon May 14, 2017
Thanks for taking the time to at least provide the error messages @PaddyMann
Yes
I think it might either be MySQL specific or SQL DBs specific. Any SQL DB that would throw an error if the AccessToken Table has not the scopes field.
PaddyMann commentedon May 14, 2017
Thank you v much for the speedy confirmation @ebarault - appreciated! 👍
bajtos commentedon May 22, 2017
@PaddyMann sorry for the confusion created by this change. You need to update your (MySQL) database and add the new
scopesfield. This can be done e.g. by running dataSource.autoupdate() function. See Migration in "Controlling data access" docs:bajtos commentedon May 22, 2017
I am closing this issue as answered, please reopen if you disagree.
PaddyMann commentedon May 22, 2017
Hi @bajtos
Ah ha - I hadn't realized
auto-updatewas needed to get table updates.We disabled all the
auto-updateandauto-migratestuff early on as we found it to be buggy, and as a result really frickin' scary to have anywhere near our production database.I'd like to avoid turning it on here. If I want to manually make the updates to my database, where should I look to find to see the changes?
Paddy
bajtos commentedon May 22, 2017
I totally understand that, I would not want to run any automagic stuff on a production database either.
Hmm, I am afraid this is something we haven't addressed yet. In theory, it should be reasonably easy to change the
autoupdatemethod to build SQL statements and print them instead of executing. That way we can create a migration SQL script that can be executed on production database server while upgrading to a newer app version. (See Martin Fowler's All database changes are migrations.)Here is what I would do myself:
autoupdateon this local databaseautoupdatewith DEBUG logs enabled, that way the connector should print all SQL statements it executes. Another option is to compare the updated schema with the original one. A quick search found MySQL Workbench and this StackExchange question.PaddyMann commentedon May 25, 2017
Thanks @bajtos for the suggested workaround.
Given the agreement that auto-magic shouldn't be relied on, it'd be great to have a clearer process here.
e.g.: Rather than relying on end-users to make this change (or fail to, and lose hours to trying to understand and fix), you could make the above steps part of your release process: Run the scripts, identify the changes made on supported databases, and include them in a release note.
bajtos commentedon May 25, 2017
I see your point. My concern is that obtaining that SQL script for each supported database would be require too much effort on our side. We would basically run the auto-magic routine for each database, thus it won't remove the reliance on
autoupdatemethod.I'd much rather spend that time writing a small command that any LoopBack user can run to get a migration SQL script for the changes made in the app recently, be it new columns added by LoopBack core or new models/properties added by the app. I am envisioning something like
lb autoupdate db --dry.cc @raymondfeng @ritch @kjdelisle
PaddyMann commentedon May 26, 2017
Fair enough! If there is such a tool, it would be great to reference in the release note when it needs to be called (or even with every release note).
Can we reopen the issue?
bajtos commentedon May 26, 2017
Let's discuss the new tool in a new issue to keep the discussion more focused: #3422
shaheero commentedon May 28, 2017
TL;DR
For SQL, add two new columns in your custom access tokens table:
By the way, in the docs it says "Note: Alternatively, you can put these lines in the server.js". I have tried that, but unless the code is put into "auth" in middleware.json, the code in server.js on its own will give this error:
"Cannot call AccessToken.findById(). The findById method has not been setup. The PersistedModel has not been correctly attached to a DataSource!"
acrodrig commentedon May 30, 2017
@shaheero following this issue as it bit as I was upgrading Loopback to version 3.8 from 3.0. Looking through
access-token.jsonI did not find aprincipalTypecolumn. Can you confirm that this is needed? Thanks much.shaheero commentedon May 30, 2017
@acrodrig It is required by
user.json, whichaccess-token.jsonhas a belongTo relationship with.In short, you have to use the polymorphic relations (including the discriminator
principalType) in your custom access token model and all your extended user models if you have several models extending the built-inUsermodel. In this case, you have to addprincipalTypecolumn to your database.As per the docs, if several models are extending the built-in
Usermodel, " you must modify the relations between the users models and the AccessToken models to allow a single AccessToken model to host access tokens for multiple types of users while at the same time allowing each user model instance to be linked to unique related access tokens."Otherwise, you will get this error: