Repository navigation
Native: get_stats, get_metadata, PQ search and AsyncExecutor.transaction work; the NotOnNativeYet docs say what is true - #69
Conversation
|
Rebased onto native-two-dist 56aab39 (the one conflict was get_metadata in schema_objects.py: the base added the optional keys rerank_oversample and fused_pq, which the JVM build reports when the engine has them; native reports them once the glue does, with the move to 26.11.1, and the shared test treats them as optional). The glue is unchanged by the rebase, so the image built from this branch still applies; after the rebase the same suites give 171 passed, 0 failed, 37 skipped (existing NOTYET and NA-JVM skips). The diff in NativeExtras.java and _native/lib.py stays minimal: one import block and the transaction case plus the vindex entry point in the glue, and one block in _thread_handle in lib.py. The Python-only fix in extras.py means this PR needs no second image build. 🤖 Generated with Claude Code |
38fc164 to
488d003
Compare
…nsaction runs its block; the module docstring says what is true VectorIndex.get_stats reads the schema:index:<bucket> row (the counters, not the row's own columns). get_metadata and get_quantization ask the engine through one new glue entry point (arcadedb_n_vindex, op metadata), so an index this session did not create reports the same keys as on the JVM build, defaults included. find_nearest_approximate runs the engine's PQ search through the same entry point (op approximate): no statement runs it. warm_up runs the first search that loads a persisted graph. AsyncExecutor.transaction is a new op of arcadedb_n_async: the engine begins the transaction on a writer thread, Python runs the block in the upcall, and a Python exception is thrown on the Java side so the writer reports it. The block runs in the upcall of #67, and a Python exception comes back through its out slot as {"error": ...}, which the Java side throws so the writer rolls back and reports it. Tests first, shared by both builds: tests/test_vector_index_introspection.py and tests/test_async_transaction.py (18 tests on the JVM build, 17 of 18 on native, see the PR). The two get_java_* calls stay JVM-only and are documented as such. The module docstring of _native/__init__.py listed far more as missing than is true. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cutor has finished its work AsyncExecutor.transaction registers up to three callbacks per call (block, ok, error). They are kept in the executor's _cb_ids (declared, never used so far) and forgotten when wait_completion has returned or the executor is closed, because a retry calls the block again. The other async operations still register callbacks that are never released; that is unchanged. The reply of a failing block needs nothing new here: with #67 it is written into the out slot as library memory, which outlives the Python thread state of an async writer, so the Java side reads {"error": ...} and the writer rolls back and reaches the error callback. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, as on the JVM build The base added these two keys to the JVM get_metadata when the engine has them (hasattr). The image links engine 26.10.2, which has not, so the native dict carries them only when the glue reports them; the shared test accepts them as optional. The glue reports them with the move to 26.11.1. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
488d003 to
ae9cf58
Compare
…name, parity setuptools) into native-not-yet-gaps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…) into native-ci-python-versions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…umnar-port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes four
NotOnNativeYetgaps of the native build and rewrites the module docstring ofsrc/arcadedb/_native/__init__.pyto what is true today. Lands after #67: this branch is restacked onnative-callbacksand the base of this PR isnative-callbacks, so the diff shows only the changes of this PR. Retarget tomainonce #67 is merged.What works now on native
VectorIndex.get_stats: the counters of the primary bucket index, from itsschema:index:<name>row (a statement, no glue). The row's own columns (name,fileId, ...) are left out.VectorIndex.get_metadataandget_quantizationof an index this session did not create (a reopened database). The engine's configuration comes from one new glue entry point,arcadedb_n_vindex(opmetadata), because no statement reports it. The dict has the same 20 keys as on the JVM build, engine defaults included; before, native returned only what the session had passed.VectorIndex.find_nearest_approximateover PRODUCT quantization:findNeighborsFromVectorApproximatethrough the same entry point (opapproximate). No SQL option runs the PQ search (vectorNeighborstakesefSearch,filter,where,groupBy,groupSize,maxDistanceonly).warm_upon native was a no-op; it now runs the first search that loads a persisted graph (this test had been hidden behind theget_statsskip).AsyncExecutor.transaction: a new op ofarcadedb_n_async. The engine begins the transaction on a writer thread and Python runs the block in the upcall. A block that raises is rolled back and reaches the per-call and the global error callback, as on the JVM build.get_java_databaseandget_java_documentstay JVM-only by design.which-package.adocandinstallation.adocnow say so, and no longer list the four calls above as missing.Callback registration:
AsyncExecutor.transactionregisters up to three callbacks per call. They are kept in the executor's_cb_ids(declared in__init__, unused until now) and forgotten (_forget, from #67) whenwait_completionhas returned or the executor is closed. A retry calls the block again, so they cannot be released earlier. The other async operations still register callbacks that are never released; that leak is known and unchanged here.Dropped when restacking on #67, and why: this PR used to carry two fixes that #67 solves in its own way, and #67 wins in both places.
extras._dispatch, added so that an{"error": ...}reply from an async writer thread outlives that thread's Python state). Native: Python functions in SQL and openCypher, record events, Ctrl-C, and the async writer crash #67 removes every Python-owned reply buffer: the callback is void and writes library memory (arcadedb_n_alloc) into an out slot that Java zeroes before the call, and Java copies and frees it. That memory does not depend on the Python thread state, so the failure path oftransactionneeds no code of its own: the block's exception comes back as{"error": ...}through the out slot, the Javatransactioncase throws it, and the writer rolls back and reaches the error callback (test_async_transaction).lib._thread_handle()taking its isolate thread fromgraal_get_current_thread. Native: Python functions in SQL and openCypher, record events, Ctrl-C, and the async writer crash #67 has the same change, so nothing is left here.Tests (written first, shared by both builds):
tests/test_vector_index_introspection.py(14) andtests/test_async_transaction.py(4). They use no Java object, so they run on both backends unchanged, and neither file is collect-ignored.XMX=12g, glue includes Native: Python functions in SQL and openCypher, record events, Ctrl-C, and the async writer crash #67 and this PR) and the tests run in the foreground:test_async_transaction4,test_vector_index_introspection14,test_async_executor21,test_python_callbacks40,test_python_function_interrupt3,test_native_udf8,test_percall_calls30,test_concurrency5,test_free_threading4,test_cypher30,test_core39 and the other sixtest_vector*.pyfiles (test_vector41,test_vector_delta_visibility2,test_vector_params_verification2,test_vector_sql39,test_vector_second_pass1,test_vector_product_quantization12): 295 passed, 0 failed, 40 skipped (the existing NOTYET and NA-JVM skips only).test_sigint,test_interrupt_hygiene,test_transaction_context_interrupt,test_results_after_close,test_resultset,test_native_startup_runtime: 51 passed, 0 failed, 11 skipped.scripts/check_no_embedded_name.pyand Bandit (--severity-level low --confidence-level low, 0 issues) pass.Glue change:
native/src/main/java/org/humemai/nativelib/NativeExtras.javaonly (thetransactioncase inasync()and the newarcadedb_n_vindexentry point), so it should merge with the other glue PRs except for the import lines. Not run in CI yet: the native build and parity jobs ofnative.yml.test_free_threadingran on Python 3.12 (GIL enabled), not on a free-threaded build.🤖 Generated with Claude Code