Skip to content

[Python][R] GcsFilesystem can appear to hang for non permanent errors #32332

Open

Description

@asfimport

GcsFileSystem will attempt to retry if it gets a non-permanent error (such as couldn't connect to server). That's fine, except: (1) the sleep call used by the retry doesn't seem to check for interrupts and (2) the default retry timeout is 15 minutes!

The following snippets will hang for 15 minutes if you run them and wait about 5 seconds before trying to do a keyboard interrupt (CTRL+C):

Rscript -e 'library(arrow); fs <- GcsFileSystem$create(endpoint_override="localhost:1234", anonymous=TRUE); fs$CreateDir("x")'

python -c 'from pyarrow.fs import GcsFileSystem; fs = GcsFileSystem(endpoint_override="localhost:1234", anonymous=True); fs.create_dir("x")'

 

Reporter: Will Jones / @wjones127

Related issues:

Note: This issue was originally created as ARROW-17020. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Jul 8, 2022

    @asfimport
    Author
  2. asfimport commented on Jul 22, 2022

    @asfimport
    Author

    Ian Cook / @ianmcook:
    As of the 9.0.0 release, R package docs contain a reference to this 15-minute default and an example showing how to override it (added in ARROW-16887) and the PyArrow docs contain an example showing how to override it (added in ARROW-16818). If we choose to implement a different default retry timeout to solve this issue, these docs should be modified accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions