Closed
Copy link
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
I am trying to understand if/when the FileStatisticsCache cache is used
However, now there is no way to understand the contents of the cache.
@nuno-faria made a really nice feature to view the contents of the cache: metadata_cache()
For example:
> select * from metadata_cache();
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
| path | file_modified | file_size_bytes | e_tag | version | metadata_size_bytes | hits | extra |
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
| hits_compatible/athena_partitioned/hits_1.parquet | 2022-07-03T15:33:57 | 174965044 | "1f5da68e097309811a675c849491ac48-9" | NULL | 165128 | 0 | page_index=false |
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
1 row(s) fetched.
Elapsed 0.005 seconds.Describe the solution you'd like
I would like a table function similar to metadata_cache() for the statistics cache.
Someting like
select * from statistics_cache();| path | file_modified | file_size_bytes | e_tag | statistics_size_bytes |
|---|---|---|---|---|
| /foo/bar | 2022-07-03T15:33:57 | 1234 | ... | 132 |
| /foo/baz | 2022-07-03T15:33:57 | 5678 | ... | 3112 |
| ... | ... | ... | ... | ... |
Where statistics_size_bytes shows the size of the statistics, in bytes
Describe alternatives you've considered
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
[-]Add a way to show the contents of the FileStatisticsCache in `datafusio-cli`[/-][+]Add a way to show the contents of the FileStatisticsCache in `datafusion-cli`[/+]nuno-faria commentedon Nov 30, 2025
@alamb I can take a stab at this. Just want to get some feedback first:
FileStatisticsCachewill need to be a trait with alist_entriestype function to get the cached entries, just likeFileMetadataCache. SinceFileMetadataCacheis public, it will cause breaking changes, but it shouldn't be difficult to migrate.statistics_size_bytes, but I don't think that information is directly available inStatistics, unlike with the cache metadata entries. However, looking at theStatisticsstruct it appears to only store twoPrecision<usize>+ fivePrecision<usize | ScalarValue>per column stored, which should be relatively small.alamb commentedon Dec 1, 2025
Thank you @nuno-faria 🙏
I agree we'll need a real trait and your proposal makes sense to me
I think this can get potentially quite large for wide tables or with large numbers of columns. For example, a
ScalarValueis 64 bytes each, so that means a 20 column table with 10 row groups each takes 6KB (without accounting for the size of Vecs, etc): 16 + (20 * 5 * 64)I recommend:
Statisticsobjectsalamb commentedon Dec 2, 2025
I looked at the code a bit more, and I think the current setup means there is no global statistics cache (by default each table provider gets its own individual cache).
I filed a ticket to track limiting the cache size and making one session-level cache here (basically following your example from the statistics cache):
DefaultFileStatisticsCache#19052I suspect once we did that then showing the contents of the cache will be much simpler
alchemist51 commentedon Dec 3, 2025
Didn't understand this fully 😅 , we are creating the Statistics cache in the CacheManager right which is a Session object. How is it still constraint to a table?
alamb commentedon Dec 4, 2025
Somehow it seems like this is the statistics cache that gets created (not connected to the session)
https://github.com/apache/datafusion/blob/9f725d9c7064813cda0de0f87d115354b68d76e6/datafusion/catalog-listing/src/table.rs#L227-L226
So making a PR to show the statistics cache is great, but I suspect it won't show up for a while
feat: Implement the `statistics_cache` function (#19054)