Faster ListingTable Listing #6182

Is your feature request related to a problem or challenge?

ListingTable currently has a very naive algorithm for finding files within a dataset that serially lists every file in the dataset, and then applies pruning to each returned file

Describe the solution you'd like

The partition pruning logic should instead list each partition separately, ideally in parallel, prune these partitions, and then return the list of contained files

Describe alternatives you've considered

No response

Additional context

No response