Repository navigation
Allow Arrow Capsule Interface #2680
Description
Apache Iceberg version
0.10.0 (latest release)
Please describe the bug 🐞
Due to how iceberg-python does checks in certain places, I can't use libraries such as arro3 or polars without having to do a conversion and include pyarrow as a dependency. Here is such a case in table/__init__.py:
def append(self, df: pa.Table, snapshot_properties: Dict[str, str] = EMPTY_DICT, branch: Optional[str] = MAIN_BRANCH) -> None:
"""
Shorthand API for appending a PyArrow table to a table transaction.
Args:
df: The Arrow dataframe that will be appended to overwrite the table
snapshot_properties: Custom properties to be added to the snapshot summary
branch: Branch Reference to run the append operation
"""
try:
import pyarrow as pa
except ModuleNotFoundError as e:
raise ModuleNotFoundError("For writes PyArrow needs to be installed") from e
from pyiceberg.io.pyarrow import _check_pyarrow_schema_compatible, _dataframe_to_data_files
if not isinstance(df, pa.Table):
raise ValueError(f"Expected PyArrow table, got: {df}")Can this be updated to use the capsule interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html ?
I can create a patch if this is something that will be accepted. Sorry for the new account, due to employer issues I can't use my "regular" one.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
kevinjqliu commentedon Nov 3, 2025
Thanks for raising this @DisturbedOcean
I believe this is similar to #1655
Please feel free to give it a try
DisturbedOcean commentedon Nov 3, 2025
@kevinjqliu Sorry - what am I supposed to try? The problem is the
isinstancechecks inpyicebergfrom what I can tell.kevinjqliu commentedon Nov 3, 2025
Feel free to submit a PR
pyiceberg is very much coupled with pyarrow right now. Would be good to decouple it and support the Arrow Capsule interface.
I see arro3 has the
Tableobject, whats the best way to abstract the isinstance check so that its not solely dependent on pyarrow?kylebarron commentedon Dec 1, 2025
The point of the PyCapsule Interface is to not have your API be stuck/tied to any one library implementation. So instead of taking in a
table: pyarrow.Table, you should take in anArrowStreamExportable, defined asSo for any input object that advertises an
__arrow_c_stream__method, you can import its data into your internal Arrow implementation of choice using, say,pyarrow.table(input_object)github-actions commentedon May 31, 2026
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.