Skip to content

manipulating non-missing elements in a field #911

Open

Description

@JonathanGregory

In order to use a scikit regression module, I need to

  • Extract the non-missing elements from a 3D field fin(z,y,x) into a numpy array (z,p), where p is a 1D index over all the non-missing (y,x) locations, which are the same for every z.

  • Put a 1D array of results r(p) into the non-missing data elements of a 2D field fout(y,x).

I've done it as follows:

findata=fin.data.compressed().to_dask_array()
findata.compute_chunk_sizes()
findata=findata.reshape(fin.shape[0],fin[0,:].data.compressed().size)
# compute r from findata
foutdata=fout.get_data().flatten()
foutdata[~foutdata.mask]=r
fout.set_data(foutdata.reshape(fout.shape))

Are there better ways, or could you provide better ways? The following occur to me:

  • It would be handy if compressed() was a method of a CF field, to flatten it into a 1D array containing only the non-missing values.
  • I needed three lines to produce findata because of having to compute_chunk_sizes on the dask array. Without that step, I got errors as below.
  • Is there a way to avoid having to get the Data out of the field in order to assign to some elements of it using the mask? This was a bit clumsy.

In the Met O IDL lib, both input and output are easier:

findata=fin.data(where(fin(0).data ne fin(0).bmdi))
; compute r from findata
fout.data(where(fout.data ne fout.bmdi))=r

It would be good to have something equally convenient in cf-python. Perhaps there is?

Here's the dask problem:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/users/sws02jmg/.local/lib/python3.11/site-packages/cfdm/decorators.py", line 44, in inplace_wrapper
    processed_copy = operation_method(self, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/sws02jmg/.local/lib/python3.11/site-packages/cf/data/data.py", line 10489, in reshape
    dx = dx.reshape(*shape, merge_chunks=merge_chunks, limit=limit)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/applications/anaconda/2023.09-0/lib/python3.11/site-packages/dask/array/core.py", line 2220, in reshape
    return reshape(self, shape, merge_chunks=merge_chunks, limit=limit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/applications/anaconda/2023.09-0/lib/python3.11/site-packages/dask/array/reshape.py", line 222, in reshape
    raise ValueError(
ValueError: Array chunk size or shape is unknown. shape: (nan,)

Possible solution with x.compute_chunk_sizes()

Thanks. Jonathan

Activity

  1. sadielbartholomew commented on Jan 13, 2026

    @sadielbartholomew
    Member

    Hi @JonathanGregory, firstly sorry for the late response, both @davidhassell and I missed this in the busy run up to the Christmas break somehow.

    David and I discussed this last Friday and it turned out to be really quite interesting. I've tried to summarise our conclusions below (in my own words - David please jump up if you feel anything needs clarifying or expanding or otherwise!).

    The crux of this, we think, is orthogonality of indexing...

    In the Met O IDL lib, both input and output are easier

    The reason that you can do what you need in IDL with an elegant two-liner is that IDL uses non-orthogonal (pointwise/linear) indexing - or at least, it does when applying Boolean indexing such as the case of the linearised statement array[where(<condition>)] which you use (apparently it does also support orthogonal indexing but not in such contexts). But cf-python only supports orthogonal indexing, which means that in order to perform the operations you are seem to need (we assume the scikit regression module can't handle masked/missing data and hence why it's necessary despite it being a pain in cf-python/cfdm space to do) you need to do work to get around that.

    For example, cf-python has a where operation with similar aim to the IDL one, but because of orthogonal indexing ours applies broadcasting and therefore attempts at linear style indexing will be interpreted differently so won't work.

    Could we support non-orthogonal indexing, generally, in cf-python? Maybe one day, we thought, but it is quite a task to implement it so certainly not any time soon, or unless there's some significant/priority use case that crops up. (In case you're wondering whether we could make it easier for you!).

    Are there better ways, or could you provide better ways?

    We considered your snippet carefully and deduced that, actually what you have is pretty much as direct as it can possibly be given the above, regrettably, so have no suggestions of anything better (with ). We were very impressed that you have constructed such a way forward, by the way! Overall, it seems contrived on the face of it but it's all necessary given what you want to achieve with differences of indexing requirements and that the data is compressed such that relevant shape changes need to be accounted for too.

    Specifically:

    It would be handy if compressed() was a method of a CF field, to flatten it into a 1D array containing only the non-missing values.

    We appreciate it would seem convenient to have that method available on a Field object, but that would entail or else imply that any data arrays for metadata constructs, notably coordinates, would be compressed along with the core/variable data array, , and it wouldn't be useful to support compression of metadata too because, for one, then we wouldn't be able to provide a proper 'view' of a field rendering it not very useful unless/until uncompressed. (The way you flatten the data with Dask is perfectly fine despite clunky appearances.)

    I needed three lines to produce findata because of having to compute_chunk_sizes on the dask array. Without that step, I got errors as below.

    The reason that errors come up otherwise is that compression is a lazy (execution-deferring) operation so the shape after that is unknown (hence shape: (nan,)), but the reshape operation needs to know the shape of the array to do the re-shaping, so it needs to run some form of computing operation such as compute or compute_chunk_sizes which will result in an in-memory result of known shape.

    Is there a way to avoid having to get the Data out of the field in order to assign to some elements of it using the mask? This was a bit clumsy.

    No, sadly - we need to be in 'data space' to do that, so accessing it via get_data() as you have done, or .data, is necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    daskRelating to the use of DaskenhancementNew feature or requestquestionGeneral question

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions