Skip to content

optimizers: admission gate ignores usage and under-counts #85

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • The opt-in admission gate (src/datafusion/src/optimizers/mod.rs:155-170) compares a scan's footprint with the maximum budgets. It does not consider how full the cache already is.
  • When statistics are missing, file_required_bytes (optimizers/mod.rs:575-591) falls back to the object size. That is a compressed on-disk size, while the statistics path is a decoded-Arrow estimate. So the "deliberate over-estimate" the doc comment describes is in fact an under-estimate, by a factor of 2,000 here.

Evidence.

  • A scan with an 8,926,215-byte footprint was admitted into a cache of the same size that already held 8,072,736 bytes (90.4% full).
  • The no-statistics fallback estimated 12,217 bytes, where the statistics path estimated 25,600,000 (0.00048x).
  • Whether the gate should count current usage is a design call, since entries can be evicted. The units mismatch is a bug either way.

Tests.

Test File Command Flags
audit_admission_gate_declines_a_scan_when_cache_is_already_full src/datafusion/src/optimizers/mod.rs:1629 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_admission_gate_declines_a_scan_when_cache_is_already_full -- --nocapture
audit_admission_estimate_without_statistics_is_not_smaller src/datafusion/src/optimizers/mod.rs:1705 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_admission_estimate_without_statistics_is_not_smaller -- --nocapture

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    performanceWasted work, over-eviction or excess IO
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformanceWasted work, over-eviction or excess IO

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions