You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Measure to_dataset peak memory + wall time on non-toy data #172
The lazy-backend memory claim (only the requested region is materialized) is verified today only against the conftest fixtures and one xarray tutorial dataset. Those are all small enough that a regression that quietly buffers the whole result would still fit in RAM and look fine.
What we want is a benchmark / smoke test that runs on a real-world-sized Dataset (a few GB on disk, multiple chunks) and asserts:
peak RSS for a single-region access stays bounded by chunk size, not result size,
streaming aggregation does not blow up while reducing a long axis,
compute() time scales linearly in materialized rows.
Likely lives under perf_tests/, gated behind a marker so CI does not always pay the download cost.
If you did put it in tests/, we have an "integration" test market that is typically ignored in CI. We could set something up to make these run daily, for eg.
Activity
alxmrs commentedon Jun 22, 2026
If you did put it in tests/, we have an "integration" test market that is typically ignored in CI. We could set something up to make these run daily, for eg.