'Table.insert_data' does not support Datetime values

Seems like the python api library does not support native python datetime objects for inserting BigQuery DateTime-values.

INFO:root:Sending load request
[(1L, 'as', datetime.datetime(2014, 2, 1, 0, 0), 'asdasd'),
 (2L, 'ds', datetime.datetime(2014, 2, 2, 0, 0), 'asdsad')]
Traceback (most recent call last):
...
  File "/home/chris/.local/lib/python2.7/site-packages/google/cloud/bigquery/table.py", line 770, in insert_data
    data=data)
  File "/home/chris/.local/lib/python2.7/site-packages/google/cloud/_http.py", line 326, in api_request
    data = json.dumps(data)
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2014, 2, 1, 0, 0) is not JSON serializable

@inkrement

Thanks for reporting @inkrement!

@inkrement

@inkrement I haven't been able to reproduce this locally. Are you on the latest version of the library?

I installed it directly from the pip repository. But I don't use the library anymore so I don't know :/

@daspecter Make sure there is a test for this case. If there already is, point to it and close this. If there is not, make one and post a PR.

I believe this is covered in this system test.

@daspecster

@daspecster Perhaps your environment is different? I'm still seeing the issue, and it's exceedingly easy to reproduce.

Create a Python2.7 virtualenv. Activate it, obviously.
pip install google-cloud-bigquery (currently resolves to the latest version on pip: google_cloud_bigquery-0.23.0-py2.py3-none-any)
Make sure you're authenticated and create a test project. Do this in python interactvely:

>>> from google.cloud import bigquery
>>> import datetime
>>> client = bigquery.Client(project='myproject-1803')
>>> dataset = client.dataset('testset')
>>> dataset.create()
>>> table = dataset.table('test_datefield')
>>> table.schema = (bigquery.SchemaField('name', 'STRING'), bigquery.SchemaField('birthday', 'DATETIME'))
>>> table.create()
>>> table.insert_data(rows=[('Ken', datetime.datetime.now())])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kkinder/bigquerytest/testenv/lib/python2.7/site-packages/google/cloud/bigquery/table.py", line 770, in insert_data
    data=data)
  File "/Users/kkinder/bigquerytest/testenv/lib/python2.7/site-packages/google/cloud/_http.py", line 294, in api_request
    data = json.dumps(data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2017, 3, 30, 14, 23, 50, 699725) is not JSON serializable

It's also a pretty common thing with Python's json library. Any time you pass a datetime object, this happens. The question is, what is the other end expecting for formatting?

@tswast

@tswast I'm running up against the difference in the JSON representation of datetime for row values (i.e., floating point seconds to microsecond precisions) vs. query parameters (ISO strings). I'm assuming that is a known issue (or design choice): are there other types where the row value representation is known to differ from query parameter representation?

Marshal row data correctly in 'Table.insert_data()' (#3426)

Navigation Menu

Repository navigation

'Table.insert_data' does not support Datetime values #2957

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

'Table.insert_data' does not support Datetime values #2957

Description

Activity

daspecster commented on Jan 23, 2017

daspecster commented on Mar 3, 2017

inkrement commented on Mar 4, 2017

lukesneeringer commented on Mar 4, 2017

daspecster commented on Mar 6, 2017

kkinder commented on Mar 30, 2017

tseaver commented on May 16, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions