Skip to content

[C++] Dataset APIs require s3:ListBucket Permissions #20147

Closed

Description

@asfimport

Hi team, first time posting an issue so I apologize if the format is lacking. My original comment is on ARROW-13685 Github Issue here. 

Long story short, our environment is super locked down, and while my application has permission to write data against an s3 prefix, I do not have the ListBucket permission nor can I add it. This does not prevent me from using the "individual" file APIs like pq.write_table but the bucket validation logic in the "dataset" APIs breaks when trying to test for the bucket's existence. 

pq.write_to_dataset(pa.Table.from_batches([data]), location, filesystem=s3fs)
OSError: When creating bucket '<my bucket>': AWS Error [code 15]: Access Denied

The same is true for the generic pyarrow.dataset APIs. My understanding is the bucket validation logic is part of the C++ code, not the Python API. As a Pythonista who knows nothing of C++ I am not sure how to resolve this problem.
 
Would it be possible to disable the bucket existence check with an optional key word argument? Thank you for your time!
 

Reporter: Jonny Fuller
Assignee: Sanjiban Sengupta / @sanjibansg

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-15892. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Mar 9, 2022

    @asfimport
    Author

    Antoine Pitrou / @pitrou:
    Which permissions do you have in that S3 environment exactly?

  2. asfimport commented on Mar 9, 2022

    @asfimport
    Author

    Will Jones / @wjones127:
    I think the ListBucket permission grants access to list objects in the bucket, which I think is required for a lot of dataset operations.

    From Amazon's docs: 

    For example, the s3:ListBucket permission allows the user to use the Amazon S3 GET Bucket (List Objects) operation.
    The error you have sounds more like it's from not having s3:CreateBucket? Which I would agree is something that shouldn't be necessary.

  3. asfimport commented on Mar 9, 2022

    @asfimport
    Author

    Jonny Fuller:
    My understanding is that the API tries to create a bucket to see if it exists as part of its standard workflow. The only permissions I have are PutObject within a prefix boundary (ie put object at /myapp/namespace/). I'm looking for a way to bypass the bucket checking step and simply write the data as if I was using the write_table API. 

    I should add, I cannot provide the detailed permissions because they are provided by a temporary STS session. To my knowledge there is no way to ask STS "what permissions does this assumed role have?" ie you have to actually make requests to see if they are denied. I know I can PutObject because write_table ie writing a single parquet file works, as do other single file operations. Only when trying to use the higher level dataset (aka multiple file partition APIs) does my operation fail due to access control issues as detailed above. 

  4. asfimport commented on Mar 9, 2022

    @asfimport
    Author

    Weston Pace / @westonpace:
    I can reproduce this and I agree that the permission in question is probably ListBucket. In this case it isn't the logic around checking bucket existence exactly that is causing the problem. In the datasets module we always call CreateDir since the function succeeds if the directory already exists.

    In s3fs, if the directory is a bucket (e.g. if it is foo and not foo/my_dataset), we call CreateBucket without doing any existence checking at all (then if we get a "bucket already exists" error we just return ok). That explains why the error you are getting is "OSError: When creating bucket" and not "OSError: When testing for existence of bucket"

    It should be straightforward to add an option to completely disable CreateDir calls in datasets. This would be a little bizarre from a local filesystem perspective but makes perfect sense in an object store where directories don't need to be created. You would also need to make sure you specify existing_data_behavior=overwrite_or_ignore

  5. asfimport commented on Mar 9, 2022

    @asfimport
    Author

    Weston Pace / @westonpace:
    Actually, I suppose this means that a user might need the CreateBucket permission (assuming they don't disable CreateDir calls entirely with the aforementioned flag) in order to use write_dataset if the base directory for their dataset is a bucket.

    In other words, today, the call write_dataset(tab, 'foo', filesystem=s3fs, format='parquet') requires both CreateBucket and ListBucket permission and the call write_dataset(tab, 'foo/my_dataset', filesystem=s3fs, format='parquet') only requires ListBucket permission (assuming the foo bucket exists). We could probably improve on that as well.

  6. asfimport commented on Mar 9, 2022

    @asfimport
    Author

    Will Jones / @wjones127:
    @westonpace based on my prior experience with S3, I'm actually surprise it would call CreateBucket at all. IMO it should probably error if the bucket doesn't exist by default; buckets are entities that usually have lots of governance around them (tags, permissions, etc), so making it easy for someone to accidentally create one via a typo seems bad. 

  7. asfimport commented on Mar 10, 2022

    @asfimport
    Author

    Jonny Fuller:
    Wow thanks for following up so quickly team. I'm definitely with @wjones127. I would want to be very specific about creating buckets.

  8. asfimport commented on Mar 10, 2022

    @asfimport
    Author

    Weston Pace / @westonpace:
    I think this is one of those things that is just a bit leaky when you have a filesystem abstraction that covers both local and remote filesystems. The flag I described above (don't create directories) ought to prevent any buckets from being created. We could also add a configuration parameter to the S3FileSystem itself (e.g. "safe mode") that would error out if it were asked to create a bucket. That would help if the user made a typo for example. I think that probably belongs in a different JIRA so feel free to open.

  9. asfimport commented on Mar 10, 2022

    @asfimport
    Author

    Antoine Pitrou / @pitrou:
    Hmm, I think adding a general S3 option is a good idea. I agree it may not make sense generally to allow implicit creation of buckets (but some S3-compatible backends may have different rules, so it should still be possible).

  10. asfimport commented on Apr 25, 2022

    @asfimport
    Author

    Antoine Pitrou / @pitrou:
    Issue resolved by pull request 12701
    #12701

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions