Repository navigation
dataflow.gpu-workers.e2e_test: test_end_to_end failed #5369
Description
Note: #5319 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.
commit: b45fe08
buildURL: Build Status, Sponge
status: failed
Test output
Traceback (most recent call last):
File "/workspace/dataflow/gpu-workers/e2e_test.py", line 129, in test_end_to_end
check=True,
File "/usr/local/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['python', 'landsat_view.py', '--output-path-prefix=gs://dataflow-gpu-test-4d29db/outputs/', '--runner=DataflowRunner', '--project=python-docs-samples-tests', '--region=us-central1', '--temp_location=gs://dataflow-gpu-test-4d29db/temp', '--worker_machine_type=custom-1-13312-ext', '--worker_harness_container_image=gcr.io/python-docs-samples-tests/dataflow/gpu-workers/test-4d29db:latest', '--worker_zone=us-central1-a', '--experiments=worker_accelerator=type=nvidia-tesla-t4,count=1,install-nvidia-driver', '--experiments=use_runner_v2']' returned non-zero exit status 1.Metadata
Metadata
Assignees
Labels
This issue needs some love.Issues related to the Dataflow API.Tells the Flaky Bot not to close or comment on this issue.An issue filed by the Flaky Bot. Should not be added manually.Important issue which blocks shipping the next release. Will be fixed prior to next release.Issues that are directly related to samples.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Activity
flaky-bot commentedon Feb 17, 2021
commit: 4484d37
buildURL: Build Status, Sponge
status: failed
Test output
Traceback (most recent call last): File "/workspace/dataflow/gpu-workers/e2e_test.py", line 129, in test_end_to_end check=True, File "/usr/local/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['python', 'landsat_view.py', '--output-path-prefix=gs://dataflow-gpu-test-7a95c6/outputs/', '--runner=DataflowRunner', '--project=python-docs-samples-tests', '--region=us-central1', '--temp_location=gs://dataflow-gpu-test-7a95c6/temp', '--worker_machine_type=custom-1-13312-ext', '--worker_harness_container_image=gcr.io/python-docs-samples-tests/dataflow/gpu-workers/test-7a95c6:latest', '--worker_zone=us-central1-a', '--experiments=worker_accelerator=type=nvidia-tesla-t4,count=1,install-nvidia-driver', '--experiments=use_runner_v2']' returned non-zero exit status 1.flaky-bot commentedon Feb 18, 2021
commit: 6634b6a
buildURL: Build Status, Sponge
status: failed
Test output
Traceback (most recent call last): File "/workspace/dataflow/gpu-workers/e2e_test.py", line 129, in test_end_to_end check=True, File "/usr/local/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['python', 'landsat_view.py', '--output-path-prefix=gs://dataflow-gpu-test-bc1155/outputs/', '--runner=DataflowRunner', '--project=python-docs-samples-tests', '--region=us-central1', '--temp_location=gs://dataflow-gpu-test-bc1155/temp', '--worker_machine_type=custom-1-13312-ext', '--worker_harness_container_image=gcr.io/python-docs-samples-tests/dataflow/gpu-workers/test-bc1155:latest', '--worker_zone=us-central1-a', '--experiments=worker_accelerator=type=nvidia-tesla-t4,count=1,install-nvidia-driver', '--experiments=use_runner_v2']' returned non-zero exit status 1.flaky-bot commentedon Feb 19, 2021
commit: c04e2ad
buildURL: Build Status, Sponge
status: failed
Test output
Traceback (most recent call last): File "/workspace/dataflow/gpu-workers/e2e_test.py", line 129, in test_end_to_end check=True, File "/usr/local/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['python', 'landsat_view.py', '--output-path-prefix=gs://dataflow-gpu-test-2cd078/outputs/', '--runner=DataflowRunner', '--project=python-docs-samples-tests', '--region=us-central1', '--temp_location=gs://dataflow-gpu-test-2cd078/temp', '--worker_machine_type=custom-1-13312-ext', '--worker_harness_container_image=gcr.io/python-docs-samples-tests/dataflow/gpu-workers/test-2cd078:latest', '--worker_zone=us-central1-a', '--experiments=worker_accelerator=type=nvidia-tesla-t4,count=1,install-nvidia-driver', '--experiments=use_runner_v2']' returned non-zero exit status 1.flaky-bot commentedon Feb 20, 2021
commit: d384d10
buildURL: Build Status, Sponge
status: failed
Test output
Traceback (most recent call last): File "/workspace/dataflow/gpu-workers/e2e_test.py", line 129, in test_end_to_end check=True, File "/usr/local/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['python', 'landsat_view.py', '--output-path-prefix=gs://dataflow-gpu-test-837955/outputs/', '--runner=DataflowRunner', '--project=python-docs-samples-tests', '--region=us-central1', '--temp_location=gs://dataflow-gpu-test-837955/temp', '--worker_machine_type=custom-1-13312-ext', '--worker_harness_container_image=gcr.io/python-docs-samples-tests/dataflow/gpu-workers/test-837955:latest', '--worker_zone=us-central1-a', '--experiments=worker_accelerator=type=nvidia-tesla-t4,count=1,install-nvidia-driver', '--experiments=use_runner_v2']' returned non-zero exit status 1.flaky-bot commentedon Feb 22, 2021
Looks like this issue is flaky. 😟
I'm going to leave this open and stop commenting.
A human should fix and close this.
When run at the same commit (d384d10), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).
tmatsuo commentedon Mar 2, 2021
I think the most important line in the error log is this:
tmatsuo commentedon Mar 2, 2021
I think it's mitigated by #5413.
Maybe we should close this.
davidcavazos commentedon Mar 2, 2021
Yes, I think that should fix it.