Repository navigation
DynamoDBRecord throws decimal.Rounded for large numbers (over 38 digits) #4845
Description
Expected Behaviour
When encountering a DynamoDB Streams event with a large number in the new/old image such as {"N": "110111111111111110000000000000000000000"}, Powertools' DynamoDB Streams data classes should be able to deserialize it like any other event. That is currently not the case if the number string is longer than 38 characters.
According to DynamoDB docs, numbers may have up to 38 digits of precision. However, the trailing zeros do not count towards the precision and creating a document with the number above works fine.
(Note: if you try to create such document using AWS Console, the editor there blocks it unless you use the JSON view. The form editor seems to use a simpler check for the number validity.)
Current Behaviour
A decimal.Rounded exception gets thrown with a stacktrace like this when you try to access record.dynamodb.new_image when using DynamoDBRecord/DynamoDBStreamEvent from DynamoDB Streams data classes to process a DynamoDB Streams events:
Rounded: [<class 'decimal.Rounded'>]
File "aws_lambda_powertools/middleware_factory/factory.py", line 135, in wrapper
response = middleware()
File "aws_lambda_powertools/utilities/data_classes/event_source.py", line 39, in event_source
return handler(data_class(event), context)
File "handler.py", line 61, in handler
xxx
File "utils.py", line 88, in xxx
item = record.dynamodb.new_image
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 155, in new_image
return self._deserialize_dynamodb_dict("NewImage")
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 139, in _deserialize_dynamodb_dict
return {k: self._deserializer.deserialize(v) for k, v in dynamodb_dict.items()}
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 139, in <dictcomp>
return {k: self._deserializer.deserialize(v) for k, v in dynamodb_dict.items()}
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
return deserializer(value[dynamodb_type])
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in _deserialize_m
return {k: self.deserialize(v) for k, v in value.items()}
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in <dictcomp>
return {k: self.deserialize(v) for k, v in value.items()}
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
return deserializer(value[dynamodb_type])
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in _deserialize_m
return {k: self.deserialize(v) for k, v in value.items()}
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in <dictcomp>
return {k: self.deserialize(v) for k, v in value.items()}
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
return deserializer(value[dynamodb_type])
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 94, in _deserialize_l
return [self.deserialize(v) for v in value]
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 94, in <listcomp>
return [self.deserialize(v) for v in value]
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
return deserializer(value[dynamodb_type])
File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 76, in _deserialize_n
return DYNAMODB_CONTEXT.create_decimal(value)
Code snippet
from aws_lambda_powertools.utilities.data_classes.dynamo_db_stream_event import DynamoDBRecord
data = {
"awsRegion": "eu-central-1",
"dynamodb": {
"ApproximateCreationDateTime": 1722318027.0,
"Keys": {"id": {"S": "test"}},
"NewImage": {
"id": {"S": "test"},
"number": {"N": "110111111111111110000000000000000000000"},
},
"OldImage": {
"id": {"S": "test"},
}
}
}
DynamoDBRecord(data).dynamodb.new_imagePossible Solution
No response
Steps to Reproduce
- Create a lambda that processes DynamoDB Streams events with the Powertools data classes or the batch processing support and call
record.dynamodb.new_imagefor a record in the event. - Create a new item in the DynamoDB table with an attribute that has type "number" and value
110111111111111110000000000000000000000
Powertools for AWS Lambda (Python) version
2.40.1
AWS Lambda function runtime
3.11
Packaging format used
Lambda Layers
Debugging logs
No response
Activity
boring-cyborg commentedon Jul 30, 2024
Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link
sthulb commentedon Jul 30, 2024
Hi @miikka I’ll look into this tomorrow. Thanks for reporting it.
sthulb commentedon Jul 31, 2024
Hi @miikka I've prepped a PR #4863 to handle this, it'll strip off tailing 0s after the 38th char (as per the docs).
Let me know if this would be sufficient.
github-actions commentedon Aug 2, 2024
This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
github-actions commentedon Aug 8, 2024
This is now released under 2.43.0 version!