Repository navigation
Allow listing tables to be created via TableFactories - #4112
Conversation
|
@andygrove and @alamb sorry for the immediate follow-on PR. I did not realize we could merge Ballista PRs pointed at git refs of Datafusion or I would have done these two as a single PR. |
There was a problem hiding this comment.
Register tables on startup.
There was a problem hiding this comment.
@alamb this is getting closer to something I think you asked for earlier: treating ListingTables and custom tables the same way and just registering factories for table types.
There was a problem hiding this comment.
Needed to add SessionState to allow ListingTables to load their schema.
There was a problem hiding this comment.
Deltatables are always folders, but .csvs should have their extension removed from their table name (and unfortunately the Path method to do this is marked unstable).
There was a problem hiding this comment.
Which method? Maybe we can contribute something back upstream to object_store 🤔
There was a problem hiding this comment.
https://doc.rust-lang.org/std/path/struct.PathBuf.html#method.file_prefix
For some reason the github UI didn't let me respond in thread until now :/
|
@andygrove I think you were requesting this feature ☝️ |
alamb
left a comment
There was a problem hiding this comment.
The code looks good to me -- is there any way we can write a test for this functionality?
There was a problem hiding this comment.
Which method? Maybe we can contribute something back upstream to object_store 🤔
There was a problem hiding this comment.
🤔 it would be really neat to somehow combine the logic in ListingTable and ListingTableFactory (or maybe datafusion-cli could just use the factory -- not sure)
There was a problem hiding this comment.
combine the logic
I think this is the duplicated fragment. I would love to combine those two and only have register_table instead of create_listing_table() and create_custom_table().
|
I know tests are somewhat painful to write, but if we don't have them I worry about breaking the functionality at some future time Maybe making a test as an example in datafusion-examples https://github.com/apache/arrow-datafusion/tree/master/datafusion-examples/examples would be a good way to
|
Done. I also filed this: #4114 . It's my own personal fault, so I understand if we don't want to inflict that on everyone. |
There was a problem hiding this comment.
Add TableFactorys for default formats.
There was a problem hiding this comment.
I refactored this test to have a better failure message because it was failing for me locally. The fact that CI passed makes me think it's not running there because datafusion-cli is excluded from the workspace - I don't know why this is, but would propose we include it.
There was a problem hiding this comment.
Could you please file a ticket to do so? Thank you!
There was a problem hiding this comment.
Moving this code to a common function on the context which we can use from datafusion-cli, tests, Ballista, etc.
There was a problem hiding this comment.
not found is a hard error message to ctrl-f for. Adding the word table will hopefully make this statistically more likely to be found.
There was a problem hiding this comment.
The proof is in the pudding. Can't select from a table without registering it first, so this must be auto-registered.
alamb
left a comment
There was a problem hiding this comment.
Looks great to me -- thank you @avantgardnerio
I also merged this PR to master locally and all the tests still pass for me
I left a few comments, but I don't think any of them are required to merge this PR
There was a problem hiding this comment.
Could you please file a ticket to do so? Thank you!
There was a problem hiding this comment.
| /// Finds any ListSchemaProviders and instructs them to reload tables from "disk" | |
| /// Invokes `ListingSchemaProvider::reload()` for all registered providers |
There was a problem hiding this comment.
You might also consider using assert_batches_eq here in this test
Then if it's alright with you, I'd love to get it merged rather than wait for another round of CI checks. Thanks for thoroughly reviewing it! |
|
Benchmark runs are scheduled for baseline = c1fc732 and contender = 7b5842b. 7b5842b is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |


Which issue does this PR close?
Closes #4111.
Rationale for this change
Described in issue.
What changes are included in this PR?
Described in issue.
Are there any user-facing changes?
They can register CSVs, parquets, jsons and avros from the datafusion-cli.