Skip to content
This repository was archived by the owner on May 21, 2025. It is now read-only.
This repository was archived by the owner on May 21, 2025. It is now read-only.

Retry uploads on (connection) errors #158

Open
Open
Retry uploads on (connection) errors#158

Description

@blueyed
blueyed
opened on Sep 7, 2018

Example: https://ci.appveyor.com/project/pytestbot/pytest/build/1.0.5141/job/ycpyxb9yvrckho21#L402 (RemoteDisconnected)

codecov-bash tries 3 times already, sleeping for 30s inbetween.

Activity

  1. blueyed commented on Nov 17, 2018

    @blueyed
    blueyed
    on Nov 17, 2018
    ContributorAuthor

    Another example:

          _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.0.15
==> Detecting CI provider
    AppVeyor Detected
    Fixing merge commit SHA
==> Preparing upload
    Error running `cd 'C:\projects\pytest' && git ls-files`: None
XX> Skip processing gcov
==> Collecting reports
XX> Searching for reports disabled
    Targeting specific files
    + coverage.xml bytes=1071184
==> Appending environment variables
    + TOXENV
==> Uploading
    .url https://codecov.io
    .query commit=3acfa3abdc7577f97bb40b9790c10d43c0e486e4&branch=master&job=pytestbot%2Fpytest%2F1.0.5975&pr=4409&service=appveyor&build=vku9wlsxtbjwwk0j&slug=pytest-dev%2Fpytest&flags=py27%2Cwindows&package=py2.0.15
    Pinging Codecov...
Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
Tip: See all example repositories: https://github.com/codecov?query=example
Support channels:
  Email:   hello@codecov.io
  IRC:     #codecov
  Gitter:  https://gitter.im/codecov/support
  Twitter: @codecov

  2. blueyed commented on Feb 18, 2019

    @blueyed
    blueyed
    on Feb 18, 2019
    ContributorAuthor

    Any update on this?

    Another example (from AppVeyor):

    ==> Uploading
    .url https://codecov.io
    .query commit=7e9f35e6cba84c386d29fc45e28d60b64927f753&branch=master&job=pytestbot%2Fpytest%2F1.0.6650&pr=4799&service=appveyor&build=r1cx4er7tmmd98ul&slug=pytest-dev%2Fpytest&package=py2.0.15
    Pinging Codecov...
Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /upload/v4?commit=7e9f35e6cba84c386d29fc45e28d60b64927f753&branch=master&job=pytestbot%2Fpytest%2F1.0.6650&pr=4799&service=appveyor&build=r1cx4er7tmmd98ul&slug=pytest-dev%2Fpytest&package=py2.0.15 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x02DFC030>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))

Tip: See all example repositories: https://github.com/codecov?query=example
Support channels:
  Email:   hello@codecov.io
  IRC:     #codecov
  Gitter:  https://gitter.im/codecov/support
  Twitter: @codecov

Sorry, we tried running command for 3 times and all attempts were unsuccessful!

  3. quantus commented on Jul 23, 2019

    @quantus
    quantus
    on Jul 23, 2019

    I've also seen quite a few similar connection errors from our CircleCI builds. My current solution for the issue has been running codecov like this:

    codecov --required || (sleep 5 && codecov --required) || (sleep 5 && codecov --required) || (sleep 5 && codecov --required) || (sleep 5 && codecov --required)

    I know it isn't pretty but I haven't seen it failing yet.

  4. timandy commented on Aug 20, 2019

    @timandy
    timandy
    on Aug 20, 2019

    I got the same issue
    Could add a parameter to specify the number of retries?

    1748      _____          _
1749     / ____|        | |
1750    | |     ___   __| | ___  ___ _____   __
1751    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
1752    | |___| (_) | (_| |  __/ (_| (_) \ V /
1753     \_____\___/ \____|\___|\___\___/ \_/
1754                                    v2.0.15
1755
1756==> Detecting CI provider
1757    Travis Detected
1758==> Preparing upload
1759==> Processing gcov (disable by -X gcov)
1760    Executing gcov (find /home/travis/build/timandy/linq -not -path './bower_components/**' -not -path './node_modules/**' -not -path './vendor/**' -type f -name '*.gcno'  -exec gcov -pb  {} +)
1761==> Collecting reports
1762    + /home/travis/build/timandy/linq/target/site/jacoco/jacoco.xml bytes=1354762
1763==> Appending environment variables
1764    + TRAVIS_JDK_VERSION
1765    + TRAVIS_OS_NAME
1766==> Uploading
1767    .url https://codecov.io
1768    .query pr=false&service=travis&package=py2.0.15&job=573661452&build=230.1&branch=master&commit=f768b4f9b4b1cd2acdf5152d0f7be269a22793c0&slug=timandy%2Flinq
1769    Pinging Codecov...
1770    Uploading to S3...
1771Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /codecov/v4/raw/2019-08-19/BD907C23C2C05F757F59BF33F38DF67D/f768b4f9b4b1cd2acdf5152d0f7be269a22793c0/7d85c4bc-4456-43cd-9289-81a22b1c23b1.txt?AWSAccessKeyId=AKIAIHLZSCQCS4WIHD4A&Expires=1566195120&Signature=Hj%2BcQ2c4lj7zLAMWUejjidNN6DU%3D (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb360244390>: Failed to establish a new connection: [Errno 110] Connection timed out',))
1772
1773Tip: See an example jdk repo: https://github.com/codecov/example-jdk
1774Support channels:
1775  Email:   hello@codecov.io
1776  IRC:     #codecov
1777  Gitter:  https://gitter.im/codecov/support
1778  Twitter: @codecov
1779
1780
1781
  5. Niccolum
    added a commit that references this issue on Sep 1, 2019

    add few changes to travisCI for not to error to request. Here more: c…

    b4642cd

  6. ikostan commented on Apr 20, 2020

    @ikostan
    ikostan
    on Apr 20, 2020

    Hi all.
    Same issue, using GitLab CI:

    Uploading to S3... Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /codecov/v4/raw/2020-04-20/102BD5852C808ECE77423D85935DC840/fc50b3d848de669df5679d7f36fbd3b14e460e2d/88c1c006-6219-41d0-8a6e-8dc3709b6235.txt?AWSAccessKeyId=AKIAIHLZSCQCS4WIHD4A&Expires=1587375167&Signature=ycNO0ZmrV%2FVhr2jmGd3dIi7Juc4%3D (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b00bfb6d8>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Tip: See all example repositories: https://github.com/codecov?query=example Support channels: Email: hello@codecov.io IRC: #codecov Gitter: https://gitter.im/codecov/support Twitter: @codecov

  7. jquast commented on May 29, 2020

    @jquast
    jquast
    on May 29, 2020

    Dear maintainer(s): Would a PR to use a 3rd party "retryable" decorator, https://tenacity.readthedocs.io/en/latest/ be accepted?

  8. nmoinvaz commented on May 29, 2020

    @nmoinvaz
    nmoinvaz
    on May 29, 2020
    Contributor

    I implemented a retry for requests here and retry number is configurable via args:

    https://github.com/nmoinvaz/codecov-python/commit/9632dc655ed378ece67bf54861a300dd1e732aef
    https://github.com/nmoinvaz/codecov-python/commit/30ff5e3140291b1a11bea9f2973efae72d8dd2aa

    If this is something that would be accepted I can submit a PR.

  9. xoxys commented on Jun 3, 2020

    @xoxys
    xoxys
    on Jun 3, 2020

    Looks like this is a long-running issue. What about the solution from @nmoinvaz? Could it be added or not? What is needed to finally get a solution into codecov-python?

  10. blueyed commented on Aug 31, 2020

    @blueyed
    blueyed
    on Aug 31, 2020
    ContributorAuthor

    Retries have been added in 07583f3 by @thomasrockhu, thanks!

    However I've still seen this now:

          _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.1.9
==> Detecting CI provider
    Gitlab CI Detected
==> Preparing upload
XX> Skip processing gcov
==> Collecting reports
    Targeting specific files
    + coverage.xml bytes=1272194
==> Appending environment variables
    + TOXENV
==> Uploading
    .url https://codecov.io
    .query commit=XXX
    Gzipping contents..
    Compressed contents to 86377 bytes
    Pinging Codecov...
Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /upload/v4?commit=XXX (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000026EC415FE48>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

    From this it is not clear if there have been any retries at all (should get logged probably), and it might have worked better with a delay between retries (#293, #295).

  11. nmoinvaz commented on Aug 31, 2020

    @nmoinvaz
    nmoinvaz
    on Aug 31, 2020
    Contributor

    codecov-bash tries 3 times already, sleeping for 30s inbetween.

    Looking at codecov bash uploader it appears to now try 5 times, sleeping for 2s in between.

  12. thomasrockhu
    self-assigned this
    on Nov 29, 2020

  13. nmoinvaz commented on Dec 2, 2020

    @nmoinvaz
    nmoinvaz
    on Dec 2, 2020
    Contributor

    I believe this issue is resolved now. It does delay between retries and also displays each time it retries.

  14. jquast
    added a commit that references this issue on Mar 6, 2021

    Bugfix for term.split_seqs('term.right(3)')

    0224922
  15. avylove
    added a commit that references this issue on Jun 10, 2021

    Bugfix for term.split_seqs('term.right(3)') #197

    2d75fdc
  16. thomasrockhu
    removed their assignment
    on Jul 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions