Skip to content

Native: get_stats, get_metadata, PQ search and AsyncExecutor.transaction work; the NotOnNativeYet docs say what is true - #69

Merged
tae898 merged 4 commits into
native-two-distfrom
native-not-yet-gaps
Oct 25, 2026
Merged

tae898 merged 4 commits into
native-two-distfrom
native-not-yet-gaps

Conversation

@tae898

@tae898 tae898 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Closes four NotOnNativeYet gaps of the native build and rewrites the module docstring of src/arcadedb/_native/__init__.py to what is true today. Lands after #67: this branch is restacked on native-callbacks and the base of this PR is native-callbacks, so the diff shows only the changes of this PR. Retarget to main once #67 is merged.

What works now on native

  • VectorIndex.get_stats: the counters of the primary bucket index, from its schema:index:<name> row (a statement, no glue). The row's own columns (name, fileId, ...) are left out.
  • VectorIndex.get_metadata and get_quantization of an index this session did not create (a reopened database). The engine's configuration comes from one new glue entry point, arcadedb_n_vindex (op metadata), because no statement reports it. The dict has the same 20 keys as on the JVM build, engine defaults included; before, native returned only what the session had passed.
  • VectorIndex.find_nearest_approximate over PRODUCT quantization: findNeighborsFromVectorApproximate through the same entry point (op approximate). No SQL option runs the PQ search (vectorNeighbors takes efSearch, filter, where, groupBy, groupSize, maxDistance only).
  • warm_up on native was a no-op; it now runs the first search that loads a persisted graph (this test had been hidden behind the get_stats skip).
  • AsyncExecutor.transaction: a new op of arcadedb_n_async. The engine begins the transaction on a writer thread and Python runs the block in the upcall. A block that raises is rolled back and reaches the per-call and the global error callback, as on the JVM build.
  • get_java_database and get_java_document stay JVM-only by design. which-package.adoc and installation.adoc now say so, and no longer list the four calls above as missing.

Callback registration: AsyncExecutor.transaction registers up to three callbacks per call. They are kept in the executor's _cb_ids (declared in __init__, unused until now) and forgotten (_forget, from #67) when wait_completion has returned or the executor is closed. A retry calls the block again, so they cannot be released earlier. The other async operations still register callbacks that are never released; that leak is known and unchanged here.

Dropped when restacking on #67, and why: this PR used to carry two fixes that #67 solves in its own way, and #67 wins in both places.

Tests (written first, shared by both builds): tests/test_vector_index_introspection.py (14) and tests/test_async_transaction.py (4). They use no Java object, so they run on both backends unchanged, and neither file is collect-ignored.

  • Native, one image built from this restacked tree (XMX=12g, glue includes Native: Python functions in SQL and openCypher, record events, Ctrl-C, and the async writer crash #67 and this PR) and the tests run in the foreground: test_async_transaction 4, test_vector_index_introspection 14, test_async_executor 21, test_python_callbacks 40, test_python_function_interrupt 3, test_native_udf 8, test_percall_calls 30, test_concurrency 5, test_free_threading 4, test_cypher 30, test_core 39 and the other six test_vector*.py files (test_vector 41, test_vector_delta_visibility 2, test_vector_params_verification 2, test_vector_sql 39, test_vector_second_pass 1, test_vector_product_quantization 12): 295 passed, 0 failed, 40 skipped (the existing NOTYET and NA-JVM skips only).
  • Also run on the same image: test_sigint, test_interrupt_hygiene, test_transaction_context_interrupt, test_results_after_close, test_resultset, test_native_startup_runtime: 51 passed, 0 failed, 11 skipped.
  • JVM build, in the sibling repo with the same test files (humemai/arcadedb-embedded-python): 18 passed, three runs. Not re-run after the restack; the two test files are unchanged.
  • Before the change the two new files gave 16 NOTYET skips and 1 failure on native.
  • pre-commit, scripts/check_no_embedded_name.py and Bandit (--severity-level low --confidence-level low, 0 issues) pass.

Glue change: native/src/main/java/org/humemai/nativelib/NativeExtras.java only (the transaction case in async() and the new arcadedb_n_vindex entry point), so it should merge with the other glue PRs except for the import lines. Not run in CI yet: the native build and parity jobs of native.yml. test_free_threading ran on Python 3.12 (GIL enabled), not on a free-threaded build.

🤖 Generated with Claude Code

@tae898
tae898 marked this pull request as ready for review October 25, 2026 12:22
@tae898

tae898 commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto native-two-dist 56aab39 (the one conflict was get_metadata in schema_objects.py: the base added the optional keys rerank_oversample and fused_pq, which the JVM build reports when the engine has them; native reports them once the glue does, with the move to 26.11.1, and the shared test treats them as optional). The glue is unchanged by the rebase, so the image built from this branch still applies; after the rebase the same suites give 171 passed, 0 failed, 37 skipped (existing NOTYET and NA-JVM skips). The diff in NativeExtras.java and _native/lib.py stays minimal: one import block and the transaction case plus the vindex entry point in the glue, and one block in _thread_handle in lib.py. The Python-only fix in extras.py means this PR needs no second image build.

🤖 Generated with Claude Code

tae898 and others added 3 commits October 25, 2026 21:32
…nsaction runs its block; the module docstring says what is true

VectorIndex.get_stats reads the schema:index:<bucket> row (the counters, not the row's own columns). get_metadata and get_quantization ask the engine through one new glue entry point (arcadedb_n_vindex, op metadata), so an index this session did not create reports the same keys as on the JVM build, defaults included. find_nearest_approximate runs the engine's PQ search through the same entry point (op approximate): no statement runs it. warm_up runs the first search that loads a persisted graph.

AsyncExecutor.transaction is a new op of arcadedb_n_async: the engine begins the transaction on a writer thread, Python runs the block in the upcall, and a Python exception is thrown on the Java side so the writer reports it. The block runs in the upcall of #67, and a Python exception comes back through its out slot as {"error": ...}, which the Java side throws so the writer rolls back and reports it.

Tests first, shared by both builds: tests/test_vector_index_introspection.py and tests/test_async_transaction.py (18 tests on the JVM build, 17 of 18 on native, see the PR). The two get_java_* calls stay JVM-only and are documented as such. The module docstring of _native/__init__.py listed far more as missing than is true.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cutor has finished its work

AsyncExecutor.transaction registers up to three callbacks per call (block, ok, error). They are kept in the executor's _cb_ids (declared, never used so far) and forgotten when wait_completion has returned or the executor is closed, because a retry calls the block again. The other async operations still register callbacks that are never released; that is unchanged.

The reply of a failing block needs nothing new here: with #67 it is written into the out slot as library memory, which outlives the Python thread state of an async writer, so the Java side reads {"error": ...} and the writer rolls back and reaches the error callback.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, as on the JVM build

The base added these two keys to the JVM get_metadata when the engine has them (hasattr). The image links engine 26.10.2, which has not, so the native dict carries them only when the glue reports them; the shared test accepts them as optional. The glue reports them with the move to 26.11.1.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 force-pushed the native-not-yet-gaps branch from 488d003 to ae9cf58 Compare October 25, 2026 12:35
@tae898
tae898 changed the base branch from native-two-dist to native-callbacks October 25, 2026 12:36
…name, parity setuptools) into native-not-yet-gaps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from native-callbacks to native-two-dist October 25, 2026 13:28
@tae898
tae898 merged commit c79a1e3 into native-two-dist Oct 25, 2026
22 of 23 checks passed
@tae898
tae898 deleted the native-not-yet-gaps branch October 25, 2026 13:34
tae898 added a commit that referenced this pull request Oct 25, 2026
…) into native-ci-python-versions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898 added a commit that referenced this pull request Oct 25, 2026
…umnar-port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant