You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would propose creating a new datafusion-contrib crate, perhaps datafusion-catalog-glue, which communicates with an AWS Glue Data Catalog.
I'll leave the exact design for whoever picks this up, but I might expect something along the following lines.
Create a GlueCatalog with an optional catalog ID
Provide a async fn GlueCatalog::list_databases(&self) -> Vec<String> to list the databases
Provide a async fn GlueCatalog::get_database(&self, name: &str) -> Result<GlueDatabase> to get a database
Implement SchemaProvider for GlueDatabase
I think it should be possible to reuse the FileScanConfig structure used by ListingTable to simplify implementation of the TableProvider.
Describe alternatives you've considered
We could not support AWS Glue
Additional context
This will help with datafusion-contrib/datafusion-objectstore-s3#53 by alleviating the need to infer the schema from the files on every query, and only listing files in non-pruned partitions.
I created issue on the official aws-sdk-rust repo to add support for Glue. I believe they prioritize based on getting 👍 so anyone interested in this please up vote there if you get the chance.
I suppose this would mean using rusoto in the meantime. im unsure what this would mean if we tried to use rusoto for glue and aws-sdk-rust for s3 (via datafusion-objectstore-s3). If needed / any type of incompatibility perhaps we could add a rusoto feature to datafusion-objectstore-s3 that could be used until official support lands.
Reacted by Raphael Taylor-Davies, Ayush Bindlish and naseerscorpio
Activity
matthewmturner commentedon Apr 12, 2022
I created issue on the official aws-sdk-rust repo to add support for Glue. I believe they prioritize based on getting 👍 so anyone interested in this please up vote there if you get the chance.
I suppose this would mean using rusoto in the meantime. im unsure what this would mean if we tried to use rusoto for glue and aws-sdk-rust for s3 (via
datafusion-objectstore-s3). If needed / any type of incompatibility perhaps we could add a rusoto feature todatafusion-objectstore-s3that could be used until official support lands.timvw commentedon May 3, 2022
I have some sample code which uses the aws-sdk-glue client, iterates over all databases and tables, and registers them in a memorycatalogprovider
https://gist.github.com/timvw/84246389d9c79fc0bf07570c625fdaf4