Firestore: what possible cause of this exception: InternalServerError: 500 Received RST_STREAM with error code 0

[READ] Step 1: Are you in the right place?

For issues or feature requests related to the code in this repository
file a Github issue.
- If this is a feature request make sure the issue title starts with "FR:".
For general technical questions, post a question on StackOverflow
with the firebase tag.
For general Firebase discussion, use the firebase-talk
google group.
For help troubleshooting your application that does not fall under one
of the above categories, reach out to the personalized
Firebase support channel.

[REQUIRED] Step 2: Describe your environment

Operating System version: Ubuntu 18.04 + openerp 7___
Firebase SDK version: _____
Library version: _from pip freeze:
firebase-admin==2.16.0 funcsigs==1.0.2 future==0.16.0 futures==3.2.0 gax-google-logging-v2==0.8.3 gax-google-pubsub-v1==0.8.3 gcloud==0.18.3 gdata==2.0.18 google-api-core==1.9.0 google-auth==1.6.3 google-cloud==0.34.0 google-cloud-core==0.29.1 google-cloud-firestore==0.32.1 google-cloud-storage==1.15.0 google-gax==0.16.0 google-resumable-media==0.3.2 googleapis-common-protos==1.5.9 grpc-google-logging-v2==0.11.1 grpc-google-pubsub-v1==0.11.1
Firebase Product: firestore___ (auth, database, storage, etc)

[REQUIRED] Step 3: Describe the problem

I'm developing a system to store orders into openerp that came from a mobile app.
So I've integrated a firestore listener on a query snapshot on "orders" collection,
BookingAppFirestore.uniqdb.collection('orders') .where('new_order', '==', True) .where('nxscm_order_id', '==', None) .on_snapshot(self.new_orders_on_snapshot)

Sometimes during the executions of my proofs I get this exception when an order is wrote on firestore and is received from the listener.

Steps to reproduce:

What happened? How can we make the problem occur?
This could be a description, log/console output, etc.

the exception trace from openerp log system

Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/cloud/firestore_v1/watch.py", line 290, in close
    raise reason
InternalServerError: 500 Received RST_STREAM with error code 0

2019-04-26 10:00:50,011 9102 INFO ? google.cloud.firestore_v1.watch: RPC termination has signaled manager shutdown.
2019-04-26 10:00:50,011 9102 ERROR db_dev google.api_core.bidi: Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "Received RST_STREAM with error code 0"
	debug_error_string = "{"created":"@1556272850.010768238","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1095,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
> and will exit.
Traceback (most recent call last):
  File "/home/nix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 543, in _thread_main
    response = self._bidi_rpc.recv()
  File "/home/nix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 454, in recv
    return self._recoverable(self._recv)
  File "/home/nix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 413, in _recoverable
    raise exc
_Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "Received RST_STREAM with error code 0"
	debug_error_string = "{"created":"@1556272850.010768238","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1095,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
>
2019-04-26 10:00:50,011 9102 INFO db_dev google.api_core.bidi: Thread-ConsumeBidirectionalStream exiting

// TODO(you): code here to reproduce the problem

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

@tseaver

@tseaver @crwilcox do you have any thoughts?

I'm also experiencing this issue both on my Kubuntu developer machine and my target platform Raspberry Pi. The problem is that when it happens I don't receive any updates from Firestore anymore.

What would be the proper way of detecting this and recover from it?

@BenWhitehead

Per the HTTP/2 spec, an RST_STREAM frame with error code NO_ERROR (0x0) is a signal from the peer to close the stream gracefully. We should loop in the Firestore back-end folks (summoning @BenWhitehead) to figure out why the server is closing the stream.

@olejl77

We're also experiencing this issue on several platforms. Are there any hooks to handle the close stream message that the client library users can leverage to attempt a reconnect?

Like @olejl77, we receive no further updates once this error has been thrown.

@gbonline

@gbonline @olejl77 @toby Can you provide any details about the environment you're running firebase-admin-python from? Are you connecting from a GCE/GKE node or from outside GCP?

Hi, my environment is outside GCP: I'm developing a module for Odoo/Openerp v 7.0 which use pyhton 2.7, in a parallels vm where guest so is ubuntu18.04, the vm have a bridged network.
So in ubuntu I installed the firebase-admin-python and build code to manage the connection with the json configuaration file.
then I've a class that it keeps the some query listeners, below is the initialization part.
Usually i start the vm ,start odoo, so my module is running, then I move my attention to develop the mobile app: this app read data from firestore, then write a order to firestore 'orders' collection and this order is received in odoo by the listener.
Sometimes I've the exception, but I don't know when it happens. I read the exception in the odoo log.

class BookingAppFirestore:

    uniqfireapp = None
    uniqdb = None
    users_ref = None
    orders_ref = None

    def __init__(self, cr, uid, pool, appname, context=None):
        self.cr = cr
        self.uid = uid
        self.pool = pool
        self.context = context
        self.appname = appname
        self.fireapp = None
        self.db = None
        self._initialize()

    def _initialize(self):
        configGSA = self.pool.get('nx.gsa.repository').browse(self.cr, self.uid, self.pool.get('nx.gsa.repository').search(self.cr, self.uid, [('code','=','SALEAPP')]), self.context)
        configGSA = configGSA and configGSA[0]
        if configGSA and BookingAppFirestore.uniqfireapp is None:
            # connect to firestore
            cred = credentials.Certificate(cert=json.loads(base64.decodestring(configGSA.serviceAccount)))

            BookingAppFirestore.uniqfireapp = firebase_admin.initialize_app(cred, name=self.appname)

            BookingAppFirestore.uniqdb = firestore.client(BookingAppFirestore.uniqfireapp)

            # listener x new users
            BookingAppFirestore.users_ref = BookingAppFirestore.uniqdb.collection('users').where('newuser','==',True).on_snapshot(self.new_users_on_snapshot)
            # listener x updated users
            # BookingAppFirestore.users_ref = BookingAppFirestore.uniqdb.collection('users').where('updateuser','==',True).on_snapshot(self.update_user_on_snapshot)
            # self.auth = self.fireapp.auth()
            # self.storage = self.fireapp.storage()
            BookingAppFirestore.orders_ref = BookingAppFirestore.uniqdb.collection('orders')\
                .where('new_order', '==', True)\
                .on_snapshot(self.new_orders_on_snapshot)

the last exception occurred now without any connections

2019-05-10 08:37:36,980 7101 INFO ? google.cloud.firestore_v1.watch: RPC termination has signaled manager shutdown.
2019-05-10 08:37:36,981 7101 ERROR ? google.api_core.bidi: Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "Received RST_STREAM with error code 0"
	debug_error_string = "{"created":"@1557477456.979956810","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1095,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
> and will exit.
Traceback (most recent call last):
  File "/home/nemesix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 543, in _thread_main
    response = self._bidi_rpc.recv()
  File "/home/nemesix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 454, in recv
    return self._recoverable(self._recv)
  File "/home/nemesix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 413, in _recoverable
    raise exc
_Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "Received RST_STREAM with error code 0"
	debug_error_string = "{"created":"@1557477456.979956810","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1095,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
>
2019-05-10 08:37:36,984 7101 INFO ? google.api_core.bidi: Thread-ConsumeBidirectionalStream exiting
Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nemesix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/cloud/firestore_v1/watch.py", line 290, in close
    raise reason
InternalServerError: 500 Received RST_STREAM with error code 0

@BenWhitehead

Hi @BenWhitehead,

We're using firebase-admin-python inside of Docker and experiencing this issue in the Google Cloud Compute Engine (on a container VM) and on our local development machines.

Python version: 3.7.3

pip freeze:

CacheControl==0.12.5
cachetools==3.0.0
certifi==2018.11.29
chardet==3.0.4
Django==2.2.1
django-phone-field==1.7.0
django-storages==1.7.1
django-timezone-field==3.0
firebase-admin==2.14.0
google-api-core==1.7.0
google-auth==1.6.2
google-cloud-core==0.29.1
google-cloud-firestore==0.31.0
google-cloud-storage==1.13.2
google-resumable-media==0.3.2
googleapis-common-protos==1.5.5
grpcio==1.17.1
gunicorn==19.9.0
idna==2.8
msgpack==0.6.0
protobuf==3.6.1
psycopg2-binary==2.7.6.1
pyasn1==0.4.5
pyasn1-modules==0.2.3
python-dateutil==2.8.0
pytz==2019.1
requests==2.21.0
rsa==4.0
six==1.12.0
sqlparse==0.3.0
urllib3==1.24.1

The past three times I've tested, following error occurs exactly 60 minutes after registering for on_snapshot on a collection:

2019-05-10 17:05 google.cloud.firestore_v1beta1.watch INFO     RPC termination has signaled manager shutdown.
2019-05-10 17:05 google.api_core.bidi ERROR    Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
      status = StatusCode.INTERNAL
      details = "Received RST_STREAM with error code 0"
      debug_error_string = "{"created":"@1557507952.036789871","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
> and will exit.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 529, in _thread_main
    response = self._bidi_rpc.recv()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 441, in recv
    return self._recoverable(self._recv)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 400, in _recoverable
    raise exc
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 390, in _recoverable
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 438, in _recv
    return next(call)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 363, in __next__
    return self._next()
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 357, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
      status = StatusCode.INTERNAL
      details = "Received RST_STREAM with error code 0"
      debug_error_string = "{"created":"@1557507952.036789871","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
>
2019-05-10 17:05 google.api_core.bidi INFO     Thread-ConsumeBidirectionalStream exiting
Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/firestore_v1beta1/watch.py", line 289, in close
    raise reason
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 0

2019-05-10 17:05 google.cloud.firestore_v1beta1.watch INFO     RPC termination has signaled manager shutdown.
2019-05-10 17:05 google.api_core.bidi ERROR    Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
      status = StatusCode.INTERNAL
      details = "Received RST_STREAM with error code 0"
      debug_error_string = "{"created":"@1557507952.296414670","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
> and will exit.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 529, in _thread_main
    response = self._bidi_rpc.recv()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 441, in recv
    return self._recoverable(self._recv)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 400, in _recoverable
    raise exc
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 390, in _recoverable
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 438, in _recv
    return next(call)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 363, in __next__
    return self._next()
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 357, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
      status = StatusCode.INTERNAL
      details = "Received RST_STREAM with error code 0"
      debug_error_string = "{"created":"@1557507952.296414670","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
>
2019-05-10 17:05 google.api_core.bidi INFO     Thread-ConsumeBidirectionalStream exiting
Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/firestore_v1beta1/watch.py", line 289, in close
    raise reason
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 0

Hi
I confirm that the exception also happens to me 60 minutes after the subscription on_snapshot to collection

@gbonline

@gbonline @toby thanks for the details, I'll get back to you when I have more info from the backend team.

Okay, after checking a few things it looks like this may be related to the GRPC client being used. There is an update to the client that was made several weeks ago. Would it be possible for you to try out the newest version of google-cloud-firestore 1.1.0 which was released on May 1st? (Specifically you're wanting to have grpcio version 1.20.1 or newer.)

Updating from 0.31.0 you may also need to update some import paths to use the new versions:

google.cloud.firestore_v1beta1 -> google.cloud.firestore_v1

@BenWhitehead

@BenWhitehead thank you for looking into this. I've updated the versions of google-cloud-firestore, grpcio and firebase-admin (see attached pip freeze) but unforunately I'm still getting the error after 60 minutes. I've included the most recent stack trace:

2019-05-13 22:28 google.cloud.firestore_v1.watch INFO     RPC termination has signaled manager shutdown.
2019-05-13 22:28 google.api_core.bidi ERROR    Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with
:
      status = StatusCode.INTERNAL
      details = "Received RST_STREAM with error code 0"
      debug_error_string = "{"created":"@1557786509.084302918","description":"Error received from peer ipv4:216.58.194.106:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
> and will exit.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 543, in _thread_main
    response = self._bidi_rpc.recv()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 454, in recv
    return self._recoverable(self._recv)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 413, in _recoverable
    raise exc
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 403, in _recoverable
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 451, in _recv
    return next(call)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 363, in __next__
    return self._next()
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 357, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
      status = StatusCode.INTERNAL
      details = "Received RST_STREAM with error code 0"
      debug_error_string = "{"created":"@1557786509.084302918","description":"Error received from peer ipv4:216.58.194.106:443","file":"src/core/lib/
surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
>
2019-05-13 22:28 google.api_core.bidi INFO     Thread-ConsumeBidirectionalStream exiting
Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/firestore_v1/watch.py", line 290, in close
    raise reason
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 0

CacheControl==0.12.5
cachetools==3.0.0
certifi==2018.11.29
chardet==3.0.4
Django==2.2.1
django-phone-field==1.7.0
django-storages==1.7.1
django-timezone-field==3.0
firebase-admin==2.16.0
google-api-core==1.10.0
google-auth==1.6.2
google-cloud-core==0.29.1
google-cloud-firestore==1.1.0
google-cloud-storage==1.13.2
google-resumable-media==0.3.2
googleapis-common-protos==1.5.5
grpcio==1.20.1
gunicorn==19.9.0
idna==2.8
msgpack==0.6.0
protobuf==3.6.1
psycopg2-binary==2.7.6.1
pyasn1==0.4.5
pyasn1-modules==0.2.3
python-dateutil==2.8.0
pytz==2019.1
requests==2.21.0
rsa==4.0
six==1.12.0
sqlparse==0.3.0
urllib3==1.24.1

Investigative work is being done internally to track down why this is happening and what will be a safe fix to it.

As of right now I believe we have been able to reproduce the behavior with a python code base and are now determining the appropriate fix.

same here. the connection dropped after 60 mins

here is my log also.

requirement

-i https://pypi.org/simple
cachecontrol==0.12.5
cachetools==3.1.1
certifi==2019.9.11
chardet==3.0.4
firebase-admin==3.0.0
google-api-core[grpc]==1.14.3 ; platform_python_implementation != 'PyPy'
google-api-python-client==1.7.11
google-auth-httplib2==0.0.3
google-auth==1.6.3
google-cloud-core==1.0.3
google-cloud-firestore==1.5.0 ; platform_python_implementation != 'PyPy'
google-cloud-storage==1.20.0
google-resumable-media==0.4.1
googleapis-common-protos==1.6.0
grpcio==1.24.1
httplib2==0.14.0
idna==2.8
msgpack==0.6.2
protobuf==3.10.0
pyasn1-modules==0.2.7
pyasn1==0.4.7
pytz==2019.3
requests==2.22.0
rsa==4.0
ruamel-yaml==0.16.5
six==1.12.0
uritemplate==3.0.0
urllib3==1.25.6
bottle==0.12.17

I can confirm we're seeing the exact same issue, in our prototype there are 4 docs being watched which is why there are 4 error streams posted below. We are outside of GCP, running under Python 3.5.3 with the following relevant content from pip freeze

firebase-admin==3.2.0
gcp-devrel-py-tools==0.0.15
google-api-core==1.14.3
google-api-python-client==1.7.8
google-auth==1.6.3
google-auth-httplib2==0.0.3
google-cloud-core==1.0.3
google-cloud-firestore==1.6.0
google-cloud-pubsub==0.40.0
google-cloud-storage==1.23.0
google-resumable-media==0.5.0
googleapis-common-protos==1.6.0
grpc-google-iam-v1==0.11.4
grpcio==1.20.1

Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "{"created":"@1576297721.599474431","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"

and will exit.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 655, in _thread_main
response = self._bidi_rpc.recv()
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 562, in recv
return self._recoverable(self._recv)
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 521, in _recoverable
raise exc
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 505, in _recoverable
return method(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 559, in _recv
return next(call)
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 363, in next
return self._next()
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 357, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "{"created":"@1576297721.599474431","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"

Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/cloud/firestore_v1/watch.py", line 290, in close
raise reason
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 0

Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "{"created":"@1576297721.645791309","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"

and will exit.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 655, in _thread_main
response = self._bidi_rpc.recv()
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 562, in recv
return self._recoverable(self._recv)
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 521, in _recoverable
raise exc
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 505, in _recoverable
return method(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 559, in _recv
return next(call)
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 363, in next
return self._next()
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 357, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "{"created":"@1576297721.645791309","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"

Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/cloud/firestore_v1/watch.py", line 290, in close
raise reason
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 0

Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "{"created":"@1576297721.674135559","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"

and will exit.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 655, in _thread_main
response = self._bidi_rpc.recv()
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 562, in recv
return self._recoverable(self._recv)
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 521, in _recoverable
raise exc
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 505, in _recoverable
return method(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 559, in _recv
return next(call)
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 363, in next
return self._next()
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 357, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "{"created":"@1576297721.674135559","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"

Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/cloud/firestore_v1/watch.py", line 290, in close
raise reason
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 0

Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "{"created":"@1576299333.815859727","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"

and will exit.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 655, in _thread_main
response = self._bidi_rpc.recv()
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 562, in recv
return self._recoverable(self._recv)
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 521, in _recoverable
raise exc
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 505, in _recoverable
return method(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/api_core/bidi.py", line 559, in _recv
return next(call)
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 363, in next
return self._next()
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 357, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "{"created":"@1576299333.815859727","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"

Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/cloud/firestore_v1/watch.py", line 290, in close
raise reason
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 0

+1

@BenWhitehead

@BenWhitehead Hi Ben - is there an official bug ID or other case for this issue and a site where we're able to see latest status or is this issue logged on github and github the only place for us to keep track of status?

thx - sean

@seanpmaddox

@seanpmaddox Hello, this issue (and the related issues linked to here) are the publicly accessible issues that are available at this time.

Unfortunately we don't yet have a tested & verified solution we're ready to release. We do have some of my Python expert colleagues as well as gRPC experts and folks from the backend team working on coming up with a solution we'll be happy with. There are a few options we're looking at, but due to the length of time it takes to reproduce progress getting the solution to users is slow.

When we have a proposed solution available, it should be linked to this issue so this is a good place to keep an eye on things.

@BenWhitehead

@BenWhitehead thank you Ben. I don't suppose any of the folks will be at CES in a few weeks and will/able to discuss?

Hopeful a solution is identified, tested & verified over the next 3 months so we can avoid going into production with a known issue like this. Our system will have as many as 48 document snapshots on each controller device which is running this API...if each watch has to be restarted every hour that is 1100+ reads per device and it starts to add up quickly when you're talking thousands of devices.

thx - sean

Proposed fix: googleapis/google-cloud-python#9995

@seanpmaddox

@seanpmaddox I won't be at CES, and I'm not sure as to other folks schedules. Per the linked PR above, it seems this has been tracked down to retry config on error codes.

https://pypi.org/project/google-cloud-firestore/1.6.1/ is released and contains the change to widen retries.

Closing this based on the fixes that have been made to the google-cloud-firestore package.

[READ] Step 1: Are you in the right place?

For issues or feature requests related to the code in this repository
file a Github issue.

If this is a feature request make sure the issue title starts with "FR:".

For general technical questions, post a question on StackOverflow
with the firebase tag.

For general Firebase discussion, use the firebase-talk
google group.

For help troubleshooting your application that does not fall under one
of the above categories, reach out to the personalized
Firebase support channel.

[REQUIRED] Step 2: Describe your environment

Operating System version: Ubuntu 18.04 + openerp 7___

Firebase SDK version: _____

Library version: _from pip freeze:
firebase-admin==2.16.0 funcsigs==1.0.2 future==0.16.0 futures==3.2.0 gax-google-logging-v2==0.8.3 gax-google-pubsub-v1==0.8.3 gcloud==0.18.3 gdata==2.0.18 google-api-core==1.9.0 google-auth==1.6.3 google-cloud==0.34.0 google-cloud-core==0.29.1 google-cloud-firestore==0.32.1 google-cloud-storage==1.15.0 google-gax==0.16.0 google-resumable-media==0.3.2 googleapis-common-protos==1.5.9 grpc-google-logging-v2==0.11.1 grpc-google-pubsub-v1==0.11.1

Firebase Product: firestore___ (auth, database, storage, etc)

[REQUIRED] Step 3: Describe the problem

I'm developing a system to store orders into openerp that came from a mobile app.
So I've integrated a firestore listener on a query snapshot on "orders" collection,
BookingAppFirestore.uniqdb.collection('orders') .where('new_order', '==', True) .where('nxscm_order_id', '==', None) .on_snapshot(self.new_orders_on_snapshot)

Sometimes during the executions of my proofs I get this exception when an order is wrote on firestore and is received from the listener.

Steps to reproduce:

What happened? How can we make the problem occur?
This could be a description, log/console output, etc.

the exception trace from openerp log system
Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/cloud/firestore_v1/watch.py", line 290, in close
    raise reason
InternalServerError: 500 Received RST_STREAM with error code 0

2019-04-26 10:00:50,011 9102 INFO ? google.cloud.firestore_v1.watch: RPC termination has signaled manager shutdown.
2019-04-26 10:00:50,011 9102 ERROR db_dev google.api_core.bidi: Thread-ConsumeBidirectionalStream caught unexpected exception <_Rendezvous of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "Received RST_STREAM with error code 0"
	debug_error_string = "{"created":"@1556272850.010768238","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1095,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
> and will exit.
Traceback (most recent call last):
  File "/home/nix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 543, in _thread_main
    response = self._bidi_rpc.recv()
  File "/home/nix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 454, in recv
    return self._recoverable(self._recv)
  File "/home/nix/Sviluppo/venvs/odoo-dev/local/lib/python2.7/site-packages/google/api_core/bidi.py", line 413, in _recoverable
    raise exc
_Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "Received RST_STREAM with error code 0"
	debug_error_string = "{"created":"@1556272850.010768238","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1095,"grpc_message":"Received RST_STREAM with error code 0","grpc_status":13}"
>
2019-04-26 10:00:50,011 9102 INFO db_dev google.api_core.bidi: Thread-ConsumeBidirectionalStream exiting
// TODO(you): code here to reproduce the problem

i have facing same problem , jot in to android studio and sign in android studio with same email account . and give the permissions and refresh the database

Navigation Menu

Repository navigation

Firestore: what possible cause of this exception: InternalServerError: 500 Received RST_STREAM with error code 0 #282

[READ] Step 1: Are you in the right place?

[REQUIRED] Step 2: Describe your environment

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

12 remaining items

requirement

[READ] Step 1: Are you in the right place?

[REQUIRED] Step 2: Describe your environment

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Firestore: what possible cause of this exception: InternalServerError: 500 Received RST_STREAM with error code 0 #282

Description

[READ] Step 1: Are you in the right place?

[REQUIRED] Step 2: Describe your environment

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Activity

google-oss-bot commented on Apr 26, 2019

hiranya911 commented on Apr 26, 2019

olejl77 commented on May 1, 2019

tseaver commented on May 1, 2019

toby commented on May 9, 2019

BenWhitehead commented on May 9, 2019

gbonline commented on May 10, 2019

toby commented on May 10, 2019

gbonline commented on May 13, 2019

BenWhitehead commented on May 13, 2019

BenWhitehead commented on May 13, 2019

toby commented on May 13, 2019

12 remaining items

BenWhitehead commented on Dec 6, 2019

AlphaWong commented on Dec 16, 2019

AlphaWong commented on Dec 16, 2019

requirement

seanpmaddox commented on Dec 16, 2019

seongkim0228 commented on Dec 24, 2019

seanpmaddox commented on Dec 24, 2019

BenWhitehead commented on Dec 26, 2019

seanpmaddox commented on Dec 27, 2019

BenWhitehead commented on Dec 30, 2019

BenWhitehead commented on Dec 30, 2019

crwilcox commented on Jan 2, 2020

hiranya911 commented on Jul 31, 2020

ShreyashAnpat commented on May 27, 2021

[READ] Step 1: Are you in the right place?

[REQUIRED] Step 2: Describe your environment

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions