Test failure in dataflow/gpu-workers #5247

Activity

1. tmatsuo

   added

   priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.

   type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

   api: dataflowIssues related to the Dataflow API.Issues related to the Dataflow API.

   samplesIssues that are directly related to samples.Issues that are directly related to samples.

   on Jan 15, 2021
2. tmatsuo

   self-assigned this

   on Jan 15, 2021
3. tmatsuo commented on Jan 15, 2021

   

   tmatsuo

   on Jan 15, 2021

   ContributorAuthor

   More actions

   It seems like it always fails.

   @davidcavazos Do you have any idea why it fails?
4. tmatsuo commented on Jan 15, 2021

   

   tmatsuo

   on Jan 15, 2021

   ContributorAuthor

   More actions

   I see a dlerror in the log:

   2021-01-15 22:29:10.524176: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
   2021-01-15 22:29:10.524235: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
   
5. tmatsuo commented on Jan 15, 2021

   

   tmatsuo

   on Jan 15, 2021

   ContributorAuthor

   More actions

   I found a more important error:

   Startup of the worker pool in zone us-central1-a failed to bring up any of the desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone 'projects/python-docs-samples-tests/zones/us-central1-a' does not have enough resources available to fulfill the request.  '(resource type:compute)'.
   
6. tmatsuo commented on Jan 15, 2021

   

   tmatsuo

   on Jan 15, 2021

   ContributorAuthor

   More actions

   There were leaked GCE instances. I deleted them and I think we're good to go.
7. tmatsuo commented on Jan 16, 2021

   

   tmatsuo

   on Jan 16, 2021

   ContributorAuthor

   More actions

   Now it's constantly passing.
8. tmatsuo

   closed this as completedon Jan 16, 2021
9. davidcavazos commented on Jan 25, 2021

   

   davidcavazos

   on Jan 25, 2021

   More actions

   Thanks. Sorry, I missed these notifications. The dynamic library warning is normal. It's trying to load GPUs, but the VM that launches the job doesn't have GPUs, but when the workers run the pipeline they should work as expected.

   About the quota issue, Valentyn also mentioned that we could change to us-central1-f and that could solve that. I applied that in #5275