Skip to content

people-and-planet-ai.image-classification.e2e_test: test_predict failed #6463

New issue
New issue
Closed
#6668
Closed
people-and-planet-ai.image-classification.e2e_test: test_predict failed#6463
#6668
Assignees
davidcavazos
Labels
api: dataflowIssues related to the Dataflow API.flakybot: flakyTells the Flaky Bot not to close or comment on this issue.flakybot: issueAn issue filed by the Flaky Bot. Should not be added manually.priority: p2Moderately-important priority. Fix may not be included in next release.samplesIssues that are directly related to samples.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Description

@flaky-bot
flaky-bot
bot
opened on Jul 22, 2021

Note: #6157 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

commit: 3b5a4bb
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.InternalServerError: 500 INTERNAL

Activity

  1. flaky-bot
    added
    flakybot: issueAn issue filed by the Flaky Bot. Should not be added manually.
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Jul 22, 2021
  2. product-auto-label
    added
    samplesIssues that are directly related to samples.
    on Jul 22, 2021
  3. blunderbuss-gcf
    assigned
    engelke
    on Jul 22, 2021
  4. flaky-bot
    added
    flakybot: flakyTells the Flaky Bot not to close or comment on this issue.
    on Jul 22, 2021

  5. flaky-bot commented on Jul 22, 2021

    @flaky-bot
    flaky-botBot
    on Jul 22, 2021
    Author

    Looks like this issue is flaky. 😟

    I'm going to leave this open and stop commenting.

    A human should fix and close this.

    When run at the same commit (3b5a4bb), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

  6. dandhlee commented on Jul 22, 2021

    @dandhlee
    dandhlee
    on Jul 22, 2021
    Collaborator

    Oh no it's back D:

  7. dandhlee commented on Jul 22, 2021

    @dandhlee
    dandhlee
    on Jul 22, 2021
    Collaborator

    Closing this for now, this is the same issue we've had from the last time this test was flaky, hoping that it's a one time flake. If it happens again I'll reach back to the product team.

  8. dandhlee
    closed this as completedon Jul 22, 2021

  9. flaky-bot commented on Jul 30, 2021

    @flaky-bot
    flaky-botBot
    on Jul 30, 2021
    Author

    Oops! Looks like this issue is still flaky. It failed again. 😬

    I reopened the issue, but a human will need to close it again.

    commit: 73fa296
    buildURL: Build Status, Sponge
    status: failed

    Test output
    Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800` is not found."
	debug_error_string = "{"created":"@1627637689.682554879","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800` is not found.","grpc_status":5}"
>

    The above exception was the direct cause of the following exception:
    

    Traceback (most recent call last):
    
File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    
PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
    
File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 77, in deploy_model
    
traffic_split={"0": 100},
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 905, in deploy_model
    
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
    
return wrapped_func(*args, **kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
    
six.raise_from(exceptions.from_grpc_error(exc), exc)
    
File "", line 3, in raise_from
    
google.api_core.exceptions.NotFound: 404 Model projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800 is not found.

  10. flaky-bot
    reopened this on Jul 30, 2021

  11. parthea commented on Jul 30, 2021

    @parthea
    parthea
    on Jul 30, 2021
    Collaborator

    This error google.api_core.exceptions.NotFound: 404 Model is different than the OP, which is an internal 500 error. I'm going to close this issue and see if it comes back.

  12. parthea
    closed this as completedon Jul 30, 2021

  13. flaky-bot commented on Jul 30, 2021

    @flaky-bot
    flaky-botBot
    on Jul 30, 2021
    Author

    Oops! Looks like this issue is still flaky. It failed again. 😬

    I reopened the issue, but a human will need to close it again.

    commit: 73fa296
    buildURL: Build Status, Sponge
    status: failed

    Test output
    Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800` is not found."
	debug_error_string = "{"created":"@1627645645.653339393","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800` is not found.","grpc_status":5}"
>

    The above exception was the direct cause of the following exception:
    

    Traceback (most recent call last):
    
File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 111, in model_endpoint_id
    
deployed_model_id = deploy_model.deploy_model(
    
File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 65, in deploy_model
    
response = client.deploy_model(
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 905, in deploy_model
    
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
    
return wrapped_func(*args, **kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
    
six.raise_from(exceptions.from_grpc_error(exc), exc)
    
File "", line 3, in raise_from
    
google.api_core.exceptions.NotFound: 404 Model projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800 is not found.

  14. 5 remaining items

  15. busunkim96
    assigned
    davidcavazos
    and unassigned
    engelke
    on Aug 3, 2021
  16. leahecole
    closed this as completedon Aug 9, 2021
  17. flaky-bot
    reopened this on Aug 15, 2021

  18. flaky-bot commented on Aug 15, 2021

    @flaky-bot
    flaky-botBot
    on Aug 15, 2021
    Author

    Oops! Looks like this issue is still flaky. It failed again. 😬

    I reopened the issue, but a human will need to close it again.

    commit: 7fa18f1
    buildURL: Build Status, Sponge
    status: failed

    Test output
    Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "", line 3, in raise_from
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.6/http/client.py", line 1379, in getresponse
    response.begin()
  File "/usr/local/lib/python3.6/http/client.py", line 311, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.6/http/client.py", line 272, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.6/ssl.py", line 1012, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.6/ssl.py", line 874, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

    During handling of the above exception, another exception occurred:
    

    Traceback (most recent call last):
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    
timeout=timeout
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
    
raise six.reraise(type(error), error, _stacktrace)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/packages/six.py", line 769, in reraise
    
raise value.with_traceback(tb)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    
chunked=chunked,
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    
six.raise_from(e, None)
    
File "", line 3, in raise_from
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    
httplib_response = conn.getresponse()
    
File "/usr/local/lib/python3.6/http/client.py", line 1379, in getresponse
    
response.begin()
    
File "/usr/local/lib/python3.6/http/client.py", line 311, in begin
    
version, status, reason = self._read_status()
    
File "/usr/local/lib/python3.6/http/client.py", line 272, in _read_status
    
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    
return self._sock.recv_into(b)
    
File "/usr/local/lib/python3.6/ssl.py", line 1012, in recv_into
    
return self.read(nbytes, buffer)
    
File "/usr/local/lib/python3.6/ssl.py", line 874, in read
    
return self._sslobj.read(len, buffer)
    
File "/usr/local/lib/python3.6/ssl.py", line 631, in read
    
v = self._sslobj.read(len, buffer)
    
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
    

    During handling of the above exception, another exception occurred:
    

    Traceback (most recent call last):
    
File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 177, in test_predict
    
image_file="animals/0036/0072.jpg",  # tapirus indicus
    
File "/workspace/people-and-planet-ai/image-classification/predict.py", line 46, in run
    
image_bytes = requests.get(f"{base_url}/{image_file}").content
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/requests/api.py", line 75, in get
    
return request('get', url, params=params, **kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/requests/api.py", line 61, in request
    
return session.request(method=method, url=url, **kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    
resp = self.send(prep, **send_kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    
r = adapter.send(request, **kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    
raise ConnectionError(err, request=request)
    
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  19. busunkim96 commented on Aug 16, 2021

    @busunkim96
    busunkim96
    on Aug 16, 2021
    Contributor

    Connection reset by peer is a transient error, closing.

  20. busunkim96
    closed this as completedon Aug 16, 2021

  21. flaky-bot commented on Aug 23, 2021

    @flaky-bot
    flaky-botBot
    on Aug 23, 2021
    Author

    Oops! Looks like this issue is still flaky. It failed again. 😬

    I reopened the issue, but a human will need to close it again.

    commit: 2e6131a
    buildURL: Build Status, Sponge
    status: failed

    Test output
    Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "", line 3, in raise_from
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1369, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 310, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 271, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

    During handling of the above exception, another exception occurred:
    

    Traceback (most recent call last):
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    
timeout=timeout
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/util/retry.py", line 532, in increment
    
raise six.reraise(type(error), error, _stacktrace)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise
    
raise value.with_traceback(tb)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    
chunked=chunked,
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    
six.raise_from(e, None)
    
File "", line 3, in raise_from
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    
httplib_response = conn.getresponse()
    
File "/usr/local/lib/python3.7/http/client.py", line 1369, in getresponse
    
response.begin()
    
File "/usr/local/lib/python3.7/http/client.py", line 310, in begin
    
version, status, reason = self._read_status()
    
File "/usr/local/lib/python3.7/http/client.py", line 271, in _read_status
    
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    
File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    
return self._sock.recv_into(b)
    
File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
    
return self.read(nbytes, buffer)
    
File "/usr/local/lib/python3.7/ssl.py", line 929, in read
    
return self._sslobj.read(len, buffer)
    
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
    

    During handling of the above exception, another exception occurred:
    

    Traceback (most recent call last):
    
File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 177, in test_predict
    
image_file="animals/0036/0072.jpg",  # tapirus indicus
    
File "/workspace/people-and-planet-ai/image-classification/predict.py", line 46, in run
    
image_bytes = requests.get(f"{base_url}/{image_file}").content
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/requests/api.py", line 75, in get
    
return request('get', url, params=params, **kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/requests/api.py", line 61, in request
    
return session.request(method=method, url=url, **kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    
resp = self.send(prep, **send_kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    
r = adapter.send(request, **kwargs)
    
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    
raise ConnectionError(err, request=request)
    
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  22. flaky-bot
    reopened this on Aug 23, 2021
  23. meredithslota
    added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    and removed
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    🚨This issue needs some love.
    on Aug 28, 2021

  24. meredithslota commented on Aug 28, 2021

    @meredithslota
    meredithslota
    on Aug 28, 2021
    Contributor

    Ok. These seems real enough now, but David has been OOO. I'm dropping this to P2 since it's not (currently) blocking anything, and will ask David to look at it when he's back.

  25. gcf-merge-on-green
    closed this as completedin #6668on Sep 8, 2021
  26. gcf-merge-on-green
    added a commit that references this issue on Sep 8, 2021

    ppai/image-classification: add exponential retries to download file (#…

    a5c2f3e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: dataflowIssues related to the Dataflow API.flakybot: flakyTells the Flaky Bot not to close or comment on this issue.flakybot: issueAn issue filed by the Flaky Bot. Should not be added manually.priority: p2Moderately-important priority. Fix may not be included in next release.samplesIssues that are directly related to samples.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions