You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A coordinate on a dimension no data variable has breaks every query on the table #286
A Dataset with a coordinate on a dimension that no data variable has registers without error, but then every query on it fails.
importnumpyasnp, xarrayasxr, xarray_sqlasxqlds=xr.Dataset(
{"t": (("x",), np.arange(3.0))},
coords={"x": [0, 1, 2], "y": [10, 20]}, # no data variable has `y`
)
ctx=xql.Context()
ctx.from_dataset("ds", ds.chunk())
ctx.sql("SELECT x, t FROM ds ORDER BY x").to_pandas()
On main, unchunked: registration succeeds, and the query raises
DataFusion error: FFI error: Execution error: Failed to call stream factory: ValueError: Dimensions {'y'} do not exist. Expected one or more of FrozenMappingWarningOnValuesAccess({'x': 3})
Chunked (ds.chunk(), as above), from_dataset itself raises TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType' in _block_len.
The schema and row count come from every dimension of the Dataset (ds.dims), while the blocks come from the data variables' dimensions. y has no place in t's rows.
Expected: the table for t has columns x and t and three rows, as ds[["t"]] (which drops y) gives. A fix could register each dimension group as ds[var_names], as the mixed-dimension path already does.
#275 sends such a Dataset to the lazy reader rather than to an in-memory table, so it behaves as on main, with a TODO naming this issue.
Activity
Address review: batched partitions, a shorter docstring, two fallbacks