Attribute error thrown in gcloud/exceptions.py

If I use an incorrect dataset_id I get an AttributeError when I try to make an API call, because payload in the exception is set to b'Backend Error'.

gcloud/exceptions.py in make_exception(response, content, use_json)
186
--> 187 message = payload.get('message', message)
188 errors = payload.get('error', {}).get('errors', ())
189

AttributeError: 'bytes' object has no attribute 'get'

This is with gcloud==0.4.1 on Python 3.4.2.

Code to reproduce:

from gcloud import datastore
datastore.set_defaults(dataset_id=123456789)
query = datastore.Query(kind="SomeKind")
list(query.fetch())

Expected behaviour would be for a helpfully worded exception to be thrown.

Thanks for the report. I just "fixed" make_exception and it seems I broke it in some other way. Looking into it.

Ahh I see now, it's not make_exception that's broken, you just expect better errors when using the "wrong" dataset ID.

>>> list(query.fetch())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "gcloud/datastore/query.py", line 420, in __iter__
    self.next_page()
  File "gcloud/datastore/query.py", line 387, in next_page
    transaction_id=transaction and transaction.id,
  File "gcloud/datastore/connection.py", line 246, in run_query
    datastore_pb.RunQueryResponse)
  File "gcloud/datastore/connection.py", line 93, in _rpc
    data=request_pb.SerializeToString())
  File "gcloud/datastore/connection.py", line 70, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error

The answer is that your dataset ID is not your numeric project ID, but is instead a word or phrase. The following are actually equivalent

datastore.set_defaults(dataset_id=123456789)
datastore.set_defaults(dataset_id='123456789')

Using your project ID is just like using FOOBARAZ as your dataset ID, and the backend returns a 503.

Maybe we should file a Cloud Datastore issue?

I'm closing this out since the issue is not the library, but either a) the incorrect dataset ID or b) the insufficient information returned in the backend response.

Please re-open if you'd like to discuss more. I'm happy to brainstorm on this.

@dhermes

@dhermes It is a 'real' AttributeError that I am seeing. If you're not able to reproduce it, it is probably down to my environment.

I had to jump and hack through some hoops to use a Python 3 only library I needed for a prototype.

Can you send the stack trace? Also, maybe the output of env from the command line (at least what you are willing to send)?

@craigloftus

@craigloftus I have been trying to reproduce and got

AccessTokenRefreshError: invalid_grant

which is due to the issue fixed in googleapis/oauth2client#126

After installed from HEAD in oauth2client and using protobuf==3.0.0-alpha-1:

>>> from gcloud import datastore
>>> datastore.set_defaults(dataset_id=123456789)
>>> k = datastore.Key('Kind', 10)
>>> datastore.get([k])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/some/path/gcloud-python/gcloud/datastore/api.py", line 211, in get
    key_pbs=[k.to_protobuf() for k in keys],
  File "/some/path/gcloud-python/gcloud/datastore/api.py", line 211, in <listcomp>
    key_pbs=[k.to_protobuf() for k in keys],
  File "/some/path/gcloud-python/gcloud/datastore/key.py", line 250, in to_protobuf
    key.partition_id.dataset_id = self.dataset_id
  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 488, in field_setter
    self._fields[field] = type_checker.CheckValue(new_value)
  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/type_checkers.py", line 155, in CheckValue
    raise TypeError(message)
TypeError: 123456789 has type <class 'int'>, but expected one of: (<class 'bytes'>, <class 'str'>)

From HEAD in oauth2client, using a valid ID:

>>> from gcloud import datastore
>>> datastore.set_defaults()
>>> k = datastore.Key('Kind', 10)
>>> datastore.get([k])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/some/path/gcloud/datastore/api.py", line 214, in get
    transaction_id=transaction and transaction.id,
  File "/some/path/gcloud/datastore/api.py", line 144, in _extended_lookup
    transaction_id=transaction_id,
  File "/some/path/gcloud/datastore/connection.py", line 189, in lookup
    datastore_pb.LookupResponse)
  File "/some/path/gcloud/datastore/connection.py", line 109, in _rpc
    data=request_pb.SerializeToString())
  File "/some/path/gcloud/datastore/connection.py", line 82, in _request
    method='POST', headers=headers, body=data)
  File "/usr/local/lib/python3.4/dist-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/oauth2client/client.py", line 551, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 1139, in request
    headers = self._normalize_headers(headers)
  File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 1107, in _normalize_headers
    return _normalize_headers(headers)
  File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 195, in _normalize_headers
    return dict([ (key.lower(), NORMALIZE_SPACE.sub(value, ' ').strip())  for (key, value) in headers.items()])
  File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 195, in <listcomp>
    return dict([ (key.lower(), NORMALIZE_SPACE.sub(value, ' ').strip())  for (key, value) in headers.items()])
TypeError: sequence item 0: expected str instance, bytes found