Skip to content

server: execution plans and stats are never pruned #90

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • execution_plans (src/datafusion-server/src/service.rs:39) gets an insert at :163 but has no removal anywhere in the crate.
  • execution_stats (service.rs:40, :222-224) is push-only, and each item can carry a full flamegraph SVG string.

Evidence.

  • 25 of 25 fully drained plans were still held afterwards.
  • All 50,000 of 50,000 stats were retained, about 48 MiB of flamegraph strings.
  • The test drives execute_plan directly. That the Flight path also never prunes was established by reading the code.

Tests.

Test File Command Flags
audit_drained_plans_are_pruned src/datafusion-server/src/tests/audit_config.rs:134 cargo +1.96.0 test -p liquid-cache-datafusion-server --lib audit_drained_plans_are_pruned -- --nocapture uses the test-only execution_plans_len
audit_execution_stats_are_bounded src/datafusion-server/src/tests/audit_config.rs:172 cargo +1.96.0 test -p liquid-cache-datafusion-server --lib audit_execution_stats_are_bounded -- --nocapture uses the test-only execution_stats_len

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    memoryMemory accounting or memory usage
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemoryMemory accounting or memory usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions