Skip to content

Opening large Zarr datasets should be lazy (and fast) #17

Closed
#100
#19
Closed
Opening large Zarr datasets should be lazy (and fast)#17
#100
#19
Labels
bugSomething isn't working

Description

@alxmrs
alxmrs
opened on Feb 19, 2024

Found out when trying to open ARCO-ERA5. It looks like it eagerly converts data to a dask dataframe.

Instead, it should return immediately with deferred calls to convert the chunks to dfs.

Implementation Notes

Change from_map to from_delayed.

Activity

  1. alxmrs
    added
    bugSomething isn't working
    on Feb 19, 2024

  2. alxmrs commented on Feb 19, 2024

    @alxmrs
    alxmrs
    on Feb 19, 2024
    MemberAuthor

    https://docs.dask.org/en/latest/delayed-collections.html

  3. alxmrs
    closed this as completedin #19on Feb 19, 2024
  4. alxmrs
    added a commit that references this issue on Feb 19, 2024

    Added more metadata to Dask Dataframe creation (#19)

    7da2184
  5. alxmrs
    added a commit that references this issue on Feb 1, 2026

    Lazy datafusion registration via custom table provider (#100)

    a1cafbf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions