[tae898]

Lands after #71. This PR is stacked on p04-port-fix (base branch of this PR); retarget it to the branch #71 merges into once #71 lands.

What it is

A port of humemai/arcadedb-embedded-python#514 (native: columnar bulk insert by address for strings, nulls and vectors, no JSON text). The native image gets one new entry point, arcadedb_n_insert_colp. Python lays whole columns out in memory (int64, float64, bool, int32, float32, UTF-8 strings as Arrow offsets plus bytes, float32 vector matrices, an Arrow validity bitmap per column) and hands the addresses over. The image builds the documents without a JSON parser and without a crossing per value.

• src/arcadedb/_native/columnar.py (new) decides the layout of each column. A column it cannot lay out (nested values, mixed types, date-times, decimals, integers past 64 bits) makes the whole call use the JSON path, so nothing is half sent.
• src/arcadedb/_native/bulk.py sends the columns. insert_columns and insert_many chunks use it when every column maps to a supported kind. ARCADEDB_NATIVE_BULK=0 turns it off.
• src/arcadedb/dataframe.py: the native document load keeps string columns and numeric columns with nulls as Arrow arrays and reads them in place.
• native/src/main/java/org/humemai/nativelib/{BulkOps,ArcadeNative}.java: the image side. This needs a rebuilt native library.
• New tests: tests/test_native_columnar_pack.py (needs no native library, backend-neutral, so it runs on both backends) and tests/test_native_columnar_insert.py (compares the stored records of the JSON path and the columnar path, native only, listed in collect_ignore on the JVM backend like the other native-only files).

Speed (measured in the humemai repo, not re-measured here)

The laptop A/B of #514 against the old native code passed its speed rule: DataFrame/Arrow insert took 0.398 of the old time (95% interval 0.395 to 0.399) and insert of dict rows took 0.437 (0.434 to 0.440). The run notes are in the private notes of the humemai repo (native-columnar-ab-laptop-20261025 in its bench repros). The A/B ran the original #514 code. This port differs from it in one rule that does not touch the columns the A/B used (see below).

Differences from #514

1. Only a float32 matrix, or a list of float32 arrays, is a column of vectors. #514 took any float matrix. The base branch already holds the rule from 0e266d7 (the JPype path stores a float64 array as the list it would have been, and tests/test_float_vector_bulk.py pins that), so columnar.py follows it and test_native_columnar_pack.py has one test changed (float32 input) plus one new test (float64 stays on the row path). Strings, nulls, ints, 1-D floats, bools and float32 vectors are as in #514.
2. The two new test files import numpy, pandas and pyarrow directly instead of pytest.importorskip (the repo's no-skips policy). The arcadedb_embedded names became arcadedb; scripts/check_no_embedded_name.py is clean.

Hunk-by-hunk port check

I diffed every hunk of #514 (the three commits, 11 files) against this branch. The earlier port 99a9dc2 dropped Java hunks silently, so the Java files were compared byte for byte.

#514 file	hunks	taken
native-spike/.../BulkOps.java	4	all 4. The file is identical to #514's head.
native-spike/.../ArcadeNative.java	2	both. Identical to #514's head except one doc-comment line that already said arcadedb/_native on the base.
_native/bulk.py	1 (rewrite)	taken as #514 has it.
_native/columnar.py	new file	taken, with the float32 rule above (2 code lines, 2 docstring lines).
_native/database.py	3	all 3 (the insert_many docstring, the call in _insert_chunk, the insert_rows_columnar import; the import was added by hand because the base orders imports differently).
_native/lib.py	1	taken (arcadedb_n_insert_colp signature).
dataframe.py	4	all 4.
tests/conftest.py	1	taken (collect_ignore of test_native_columnar_insert.py off the native backend).
tests/test_native_columnar_insert.py	new file	taken, imports as above.
tests/test_native_columnar_pack.py	new file	taken, one test changed to float32 and one added (see above), imports as above.
docs/development/bridge.md	1	not taken: it edits one row of the JPype bridge table, and this repository has no bridge page (docs/modules/ROOT/pages/development holds ci.adoc and release.adoc only).

Tests

Native image rebuilt once from this branch (GraalVM CE 25.4.4.1.1, engine 26.10.2, -march=compatibility, through the laptop job gate). A wheel was built against that image and native/wheel/parity_wheel.py ran these files, one process each, on the native backend:

file	passed	failed	NOTYET	NA-JVM
test_native_columnar_insert	17	0	0	0
test_native_columnar_pack	26	0	0	0
test_bulk_insert	46	0	0	0
test_bulk_json_paths	53	0	0	0
test_insert_columns	27	0	0	0
test_graph_batch	41	0	0	4
test_dataframe_ingest	69	0	0	0
test_resultset_arrow	10	0	0	0
test_core	39	0	0	0
test_concurrency	5	0	0	0
test_float_vector_bulk	25	0	13	1
test_vector	38	0	3	6
test_vector_delta_visibility	2	0	0	0
test_vector_params_verification	2	0	0	11
test_vector_product_quantization	12	0	1	0
test_vector_second_pass	1	0	0	0
test_vector_sql	39	0	5	2
total	452	0	22	24

No failures, errors, timeouts or other skips. The NOTYET and NA-JVM counts of the files that existed before are the same as in the earlier parity run. On the JVM backend (run from the source tree), test_native_columnar_pack.py passes (26) and a whole-directory collection leaves test_native_columnar_insert.py out. Not run here: the rest of the JVM-backend suite (the JVM code paths of dataframe.py are unchanged: the new arrow argument defaults to off) and the full native parity of all other files.

Lint: Bandit 1.9.4 (-c pyproject.toml -r src tests --severity-level low --confidence-level low) reports no issues, black 26.10.1 and isort 9.0.2 (--profile black) are clean on src and tests, and check_no_embedded_name.py passes.

🤖 Generated with Claude Code