[tae898]

Stacked on #66 (native-two-dist, rebased on 56aab39). Closes the hardest native parity gap: Python functions called from SQL and openCypher, record event callbacks, the Ctrl-C behaviour of a Python function, and the crash in tests/test_async_executor.py. It covers 31 failures of test_python_callbacks.py, 1 of test_python_function_interrupt.py and the crash, from the 68 failures of the native parity run (parity-native-two-dist). Nothing here raises NotOnNativeYet: the features work on the native arcadedb, with the behaviour of arcadedb-jvm.

Test results

Source tree with ARCADEDB_BACKEND=native, on the image built from this branch (~/build-scratch/native-lib/callbacks2, GraalVM CE 25.4.4.1.1, -march=compatibility).

• test_python_callbacks.py: 40 passed (9 passed, 31 failed before).
• test_python_function_interrupt.py: 3 passed, 2 skipped (the two JPype-only cases). The in-process Ctrl-C test passes, and so does a new one that runs the same loop in a fresh process and fails on a nonzero exit.
• test_async_executor.py: 21 passed (the file crashed before).
• Together with test_native_udf, test_sigint, test_interrupt_hygiene, test_transaction_context_interrupt, test_results_after_close, test_core, test_cypher, test_resultset, test_concurrency, test_percall_calls, test_native_startup_runtime, test_schema and test_select_matrix: 303 passed, 26 skipped, 0 failed.
• A Ctrl-C test run from a background job (nohup ... &) sees no interrupts at all: a non-interactive shell starts background jobs with SIGINT ignored, and Python then installs no handler. Run the signal tests in the foreground. The fresh-process test sets the default handler itself.

Design: how the native image calls back into Python

• One function pointer. Python registers a single ctypes CFUNCTYPE pointer with arcadedb_n_set_upcall (this existed for async callbacks). The image keeps it as a CFunctionPointer and calls it as fn(callbackId, json, out). A Python-side table maps the id to the callable; the entry is removed when the function is unregistered or replaced, when the listener is removed, and when the database closes, so the table does not grow.
• Registry entry point. A new arcadedb_n_cb(op, json) registers and removes functions and event listeners. Its checks, error wording, veto and failure rules are those of the JPype bridge (PythonFunctions, PythonRecordEvents): an engine function name is refused, replace=True is needed to replace, a revoked function says it was unregistered, a failing callback is Python function 'x' failed: ValueError: ... or Python 'before_create' callback failed: ..., a before_* callback returning False vetoes (and save() of the vetoed new record returns it unsaved, as on JPype). The Python-side checks (name, callable, keyword-only, argument counts, event names) are one shared helper in arcadedb.callbacks, used by both backends.
• GIL. ctypes takes the GIL for a CFUNCTYPE call (PyGILState_Ensure), on the calling thread of the query or on a foreign Java thread (an async writer). The foreign-library call that is running the statement released the GIL, so there is no deadlock; a function may call back into the database.
• Marshalling. Arguments are the tagged JSON of query parameters (NativeCodec), results go back through the same encoder as parameters. A record argument or event record arrives as a dict with @rid, @type and @cat and is wrapped as Document, Vertex or Edge; the after_delete event of a record that is not loaded carries its identity only. A result past a Java long is an OverflowError; numpy arrays, Decimal, datetime, list and dict work as on JPype.
• Exceptions. A Python exception never crosses the C frame. _dispatch catches BaseException and replies {"error": "Type: text"}; the engine turns it into a CommandExecutionException (functions) or DatabaseOperationException (events), so the statement fails with the Python message and the surrounding transaction does not commit.
• Reply memory, one owner. The reply does not travel as a return value but through an out slot that Java zeroes before the call. The text is in memory from arcadedb_n_alloc (the library's allocator); Java copies it and frees it, and Python writes nothing else into the slot. Two failures forced this. A thread-local Python buffer dies with the Python thread state of a callback on a Java thread (an async writer) before Java has read it. And a C callback that returns without setting its return value hands the engine whatever the return register held, here the pointer of an earlier reply that Java had freed: the second free aborted the process (free(): invalid pointer). That happens when a Ctrl-C is raised as the callback is entered, before its try. With the slot, an early exit leaves NULL.
• Ctrl-C. A KeyboardInterrupt in a function or callback is remembered per thread; the place that turns the statement's failure into an ArcadeDBError raises the KeyboardInterrupt instead, chained from that error, for query, command, iteration and to_list alike. A callback that ended before it could reply (the Ctrl-C raised on entry) makes the engine fail the statement with a fixed text, and the same place turns that into a KeyboardInterrupt too. The interrupt is cleared when a statement starts, so a stale one never turns a later error into a KeyboardInterrupt. This is the rule of the JPype backend (arcadedb.callbacks); sys.unraisablehook is not touched (CPython prints the usual "Exception ignored on calling ctypes callback function" line for the entry case).

The async-executor crash

Not related to the callbacks themselves. test_per_record_callback_runs_before_the_batch_commits_and_on_ok_after calls db.count_type from an on_ok callback, which runs on an async writer thread. _thread_handle() saw a thread that was not a Python-created one and attached it to the isolate again as an owned handle. When the callback returned, CPython deleted the callback's thread state, _Thread.__del__ called graal_detach_thread on a thread that was still running Java code, and the image aborted (Fatal Python error: Aborted). The thread handle now asks graal_get_current_thread first: a thread that is attached already gets a handle that is never detached.

Behaviour change

NativeDatabase.register_function had positional min_args and max_args; it has the signature of the JPype one now (keyword-only min_args, max_args, deterministic, replace), and unregister_function and on exist. tests/test_native_udf.py uses the keywords. test_registering_again_and_again_does_not_grow_the_python_heap only forces a JVM gc on the JVM backend.

Notes for the other native changes

• Native: get_stats, get_metadata, PQ search and AsyncExecutor.transaction work; the NotOnNativeYet docs say what is true #69 (native-not-yet-gaps) also changes the reply buffer in extras._dispatch (a dict keyed by thread id instead of a thread-local) and touches NativeExtras.java and lib.py. This PR removes the Python-owned reply buffer altogether (library memory, freed by Java), so the dict is not needed here and the two _dispatch hunks conflict: when both land, keep this PR's _dispatch. This PR does not depend on Native: get_stats, get_metadata, PQ search and AsyncExecutor.transaction work; the NotOnNativeYet docs say what is true #69.
• The earlier parity failure test_cypher.py::test_opencypher_ddl_auto_creates_vertex_and_edge_types ('function' object has no attribute 'exists_type') was mine: replacing register_function in database.py dropped the @property line of schema. It is restored (test_cypher.py: 30 passed, as on the base).

What is not here

The other parity failures (positional counts, record id values, bulk insert, log directory, PQ settings, docs examples, XML limits) are other changes.

🤖 Generated with Claude Code