Skip to content

datafusion: per-file side tables never shrink #84

Closed

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong. Two maps keyed by file only ever grow:

  • ParquetCacheMetadata.expression_hints (src/datafusion/src/io/mod.rs:13). Each column path is capped at 16 expressions, but the number of paths grows without bound.
  • LiquidCacheParquet.files (src/datafusion/src/cache/mod.rs:364).

Neither shrinks when the cache is emptied or a file's handles are dropped. DefaultCacheMetadata.lineages (src/core/src/cache/io_context.rs:38) is the same pattern but was not tested.

Evidence.

  • 1,000 column paths were still tracked after flush_data() and reset() left the cache with 0 entries.
  • 10,000 of 10,000 file identities remained after every handle was dropped.

Tests.

Test File Command Flags
audit_expression_hints_are_pruned_when_cache_is_emptied src/datafusion/src/cache/mod.rs:1125 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_expression_hints_are_pruned_when_cache_is_emptied -- --nocapture uses the test-only test_hint_counts and test_metadata
audit_file_registry_shrinks_once_files_hold_no_data src/datafusion/src/cache/mod.rs:1184 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_file_registry_shrinks_once_files_hold_no_data -- --nocapture
audit_expression_hint_history_is_capped_per_column src/datafusion/src/cache/mod.rs:1161 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_expression_hint_history_is_capped_per_column -- --nocapture C (passes; confirms the cap of 16)

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemoryMemory accounting or memory usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions