Skip to content

Spanner: Error message for non-existent spanner instance is hard to debug. #8695

Closed
Closed
Spanner: Error message for non-existent spanner instance is hard to debug.#8695
Assignees
tseaver
Labels
api: spannerIssues related to the Spanner API.backendexternalThis issue is blocked on a bug with the actual product.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Description

@crwilcox
crwilcox
opened on Jul 17, 2019

When an invalid (non-existent) instance id is specified the error you receive isn't intuitive.

spanner_client = spanner.Client()

# Get an instance, but provide a non-existent instance id
instance = spanner_client.instance('non_existent_spanner_instance')

database = instance.database('real_database_id_on_different_instance')
with database.snapshot() as snapshot:
  results = snapshot.execute_sql('SELECT 1')

  for row in results:
      print(row)

Expected

An error indicating my instance doesn't exist.

Actual

google.api_core.exceptions.InvalidArgument: 400 Invalid CreateSession request.

 File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/cloud/spanner_v1/database.py", line 499, in __enter__
    session = self._session = self._database._pool.get()
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/cloud/spanner_v1/pool.py", line 268, in get
    session.create()
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/cloud/spanner_v1/session.py", line 116, in create
    session_pb = api.create_session(self._database.name, metadata=metadata, **kw)
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/cloud/spanner_v1/gapic/spanner_client.py", line 290, in create_session
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "/usr/lib/python2.7/dist-packages/six.py", line 737, in raise_from
    raise value
google.api_core.exceptions.InvalidArgument: 400 Invalid CreateSession request.

Activity

  1. crwilcox
    added
    api: spannerIssues related to the Spanner API.
    on Jul 17, 2019
  2. tseaver
    added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Jul 17, 2019

  3. tseaver commented on Jul 17, 2019

    @tseaver
    tseaver
    on Jul 17, 2019
    Contributor

    @crwilcox I'm not sure how we can fix that: is there a more useful message in the details of the exception we can surface? We don't want to implicitly generated an 'instance.get' call early to enforce instance existence.

  4. mf2199 commented on Jul 18, 2019

    @mf2199
    mf2199
    on Jul 18, 2019
    Contributor

    A simple check whether the instance exists results in another error:
    InvalidArgument: 400 Invalid GetInstance request.

    This shouldn't happen, as Instance.exists() method is supposed to simply return False if the instance does not exists. Instead, it throws an exception. Could it be a matter for another issue?

  5. tseaver commented on Jul 19, 2019

    @tseaver
    tseaver
    on Jul 19, 2019
    Contributor

    @mf2199 I can't reproduce that with a "legal" instance ID:

    Python 3.7.1 (default, Oct 22 2018, 11:31:38) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud.spanner_v1 import Client
>>> client = Client()
>>> instance = client.instance('nonesuch-instance-exists')
>>> instance.exists()
False

    Hyphens are allowed in instance names, but underscores are not, per the Instance message API docs.

  6. mf2199 commented on Jul 25, 2019

    @mf2199
    mf2199
    on Jul 25, 2019
    Contributor

    @tseaver Got it, thanks! The issue can be circumvented by adding some checkpoints inside the Database class' batch() and snapshot() methods. However, as was noted in #220, this can also be readdressed as a feature request and solved at the server side. What do you think?

  7. tseaver commented on Jul 26, 2019

    @tseaver
    tseaver
    on Jul 26, 2019
    Contributor

    Making an unasked-for API call to validate existence of the instance / database before returning a session has undesirable performance / billing consequences.

    Trying to reproduce with a valid (hyphens, not underscores) but non-existent instance id:

    >>> from google.cloud import spanner
>>> client = spanner.Client()
>>> instance = client.instance('nonesuch-instance-exists')
>>> instance.exists()
False
>>> database = instance.database('valid-database-name')
>>> database.exists()
False
>>> with database.snapshot() as snapshot:
...     results = snapshot.execute_sql('SELECT 1')
...     for row in results:
...         print(row)
... 
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/pool.py", line 264, in get
    session = self._sessions.get_nowait()
  File "/opt/Python-3.7.1/lib/python3.7/queue.py", line 198, in get_nowait
    return self.get(block=False)
  File "/opt/Python-3.7.1/lib/python3.7/queue.py", line 167, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/.nox/system-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/.nox/system-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Instance not found: projects/citric-celerity-697/instances/nonesuch-instance-exists"
	debug_error_string = "{"created":"@1564156732.715693901","description":"Error received from peer ipv6:[2607:f8b0:4004:801::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Instance not found: projects/citric-celerity-697/instances/nonesuch-instance-exists","grpc_status":5}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/database.py", line 499, in __enter__
    session = self._session = self._database._pool.get()
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/pool.py", line 267, in get
    session.create()
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/session.py", line 117, in create
    session_pb = api.create_session(self._database.name, metadata=metadata, **kw)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/gapic/spanner_client.py", line 307, in create_session
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.NotFound: 404 Instance not found: projects/citric-celerity-697/instances/nonesuch-instance-exists

    Which seems like a pretty helpful messsage. I think making the message for and invalid instance ID more useful is an issue for the back-end API, not for google-cloud-spanner. It is true for other RPCs, as well, e.g.:

    >>> invalid_instance.exists()
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/.nox/system-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/.nox/system-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Invalid GetInstance request."
	debug_error_string = "{"created":"@1564156898.102875705","description":"Error received from peer ipv6:[2607:f8b0:4004:801::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Invalid GetInstance request.","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/instance.py", line 236, in exists
    api.get_instance(self.name, metadata=metadata)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_admin_instance_v1/gapic/instance_admin_client.py", line 609, in get_instance
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Invalid GetInstance request.
  8. tseaver
    added
    backend
    externalThis issue is blocked on a bug with the actual product.
    on Jul 26, 2019

  9. mf2199 commented on Jul 26, 2019

    @mf2199
    mf2199
    on Jul 26, 2019
    Contributor

    @tseaver, I suspected so, thanks for the informative response. I'd suggest this issue be redirected to the back-end team(s), given all the above argumentation.

  10. tseaver commented on Jul 29, 2019

    @tseaver
    tseaver
    on Jul 29, 2019
    Contributor

    @crwilcox Can you raise the issue of the error messages for invalid (not missing) instance IDs to the Spanner team?

  11. tseaver
    closed this as completedon Sep 27, 2019
  12. JustinBeckwith
    assigned
    tseaver
    on Feb 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: spannerIssues related to the Spanner API.backendexternalThis issue is blocked on a bug with the actual product.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions