Repository navigation
Open
Labels
daskRelating to the use of DaskRelating to the use of DaskenhancementNew feature or requestNew feature or requestquestionGeneral questionGeneral question
Description
In order to use a scikit regression module, I need to
-
Extract the non-missing elements from a 3D field
fin(z,y,x) into a numpy array (z,p), where p is a 1D index over all the non-missing (y,x) locations, which are the same for every z. -
Put a 1D array of results
r(p) into the non-missing data elements of a 2D fieldfout(y,x).
I've done it as follows:
findata=fin.data.compressed().to_dask_array()
findata.compute_chunk_sizes()
findata=findata.reshape(fin.shape[0],fin[0,:].data.compressed().size)
# compute r from findata
foutdata=fout.get_data().flatten()
foutdata[~foutdata.mask]=r
fout.set_data(foutdata.reshape(fout.shape))
Are there better ways, or could you provide better ways? The following occur to me:
- It would be handy if
compressed()was a method of a CF field, to flatten it into a 1D array containing only the non-missing values. - I needed three lines to produce
findatabecause of having tocompute_chunk_sizeson the dask array. Without that step, I got errors as below. - Is there a way to avoid having to get the
Dataout of the field in order to assign to some elements of it using the mask? This was a bit clumsy.
In the Met O IDL lib, both input and output are easier:
findata=fin.data(where(fin(0).data ne fin(0).bmdi))
; compute r from findata
fout.data(where(fout.data ne fout.bmdi))=r
It would be good to have something equally convenient in cf-python. Perhaps there is?
Here's the dask problem:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/users/sws02jmg/.local/lib/python3.11/site-packages/cfdm/decorators.py", line 44, in inplace_wrapper
processed_copy = operation_method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/users/sws02jmg/.local/lib/python3.11/site-packages/cf/data/data.py", line 10489, in reshape
dx = dx.reshape(*shape, merge_chunks=merge_chunks, limit=limit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/software/applications/anaconda/2023.09-0/lib/python3.11/site-packages/dask/array/core.py", line 2220, in reshape
return reshape(self, shape, merge_chunks=merge_chunks, limit=limit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/software/applications/anaconda/2023.09-0/lib/python3.11/site-packages/dask/array/reshape.py", line 222, in reshape
raise ValueError(
ValueError: Array chunk size or shape is unknown. shape: (nan,)
Possible solution with x.compute_chunk_sizes()
Thanks. Jonathan
Metadata
Metadata
Assignees
Labels
daskRelating to the use of DaskRelating to the use of DaskenhancementNew feature or requestNew feature or requestquestionGeneral questionGeneral question
Activity
sadielbartholomew commentedon Jan 13, 2026
Hi @JonathanGregory, firstly sorry for the late response, both @davidhassell and I missed this in the busy run up to the Christmas break somehow.
David and I discussed this last Friday and it turned out to be really quite interesting. I've tried to summarise our conclusions below (in my own words - David please jump up if you feel anything needs clarifying or expanding or otherwise!).
The crux of this, we think, is orthogonality of indexing...
The reason that you can do what you need in IDL with an elegant two-liner is that IDL uses non-orthogonal (pointwise/linear) indexing - or at least, it does when applying Boolean indexing such as the case of the linearised statement
array[where(<condition>)]which you use (apparently it does also support orthogonal indexing but not in such contexts). But cf-python only supports orthogonal indexing, which means that in order to perform the operations you are seem to need (we assume the scikit regression module can't handle masked/missing data and hence why it's necessary despite it being a pain in cf-python/cfdm space to do) you need to do work to get around that.For example, cf-python has a
whereoperation with similar aim to the IDL one, but because of orthogonal indexing ours applies broadcasting and therefore attempts at linear style indexing will be interpreted differently so won't work.Could we support non-orthogonal indexing, generally, in cf-python? Maybe one day, we thought, but it is quite a task to implement it so certainly not any time soon, or unless there's some significant/priority use case that crops up. (In case you're wondering whether we could make it easier for you!).
We considered your snippet carefully and deduced that, actually what you have is pretty much as direct as it can possibly be given the above, regrettably, so have no suggestions of anything better (with ). We were very impressed that you have constructed such a way forward, by the way! Overall, it seems contrived on the face of it but it's all necessary given what you want to achieve with differences of indexing requirements and that the data is compressed such that relevant shape changes need to be accounted for too.
Specifically:
We appreciate it would seem convenient to have that method available on a
Fieldobject, but that would entail or else imply that any data arrays for metadata constructs, notably coordinates, would be compressed along with the core/variable data array, , and it wouldn't be useful to support compression of metadata too because, for one, then we wouldn't be able to provide a proper 'view' of a field rendering it not very useful unless/until uncompressed. (The way you flatten the data with Dask is perfectly fine despite clunky appearances.)The reason that errors come up otherwise is that compression is a lazy (execution-deferring) operation so the shape after that is unknown (hence
shape: (nan,)), but thereshapeoperation needs to know the shape of the array to do the re-shaping, so it needs to run some form of computing operation such ascomputeorcompute_chunk_sizeswhich will result in an in-memory result of known shape.No, sadly - we need to be in 'data space' to do that, so accessing it via
get_data()as you have done, or.data, is necessary.