Skip to content

Native: release the callback ids of every async executor call - #80

Merged
tae898 merged 1 commit into
mainfrom
native-async-callback-release
Oct 25, 2026
Merged

tae898 merged 1 commit into
mainfrom
native-async-callback-release

Conversation

@tae898

@tae898 tae898 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Every native async call that takes a callback registers a Python function in extras._callbacks and hands the library its id. Only AsyncExecutor.transaction ever gave the ids back (#67, #69). create_record, query, command, new_edge, new_edge_by_keys and scan_type leaked one to three ids per call, and on_ok / on_error leaked two more each time they were set, so the table grew without bound over a long-lived executor.

What changes

All async calls now go through one _queue(op, **args) helper in NativeAsyncExecutor, which is the rule transaction already used, applied to every op:

  • A queued call puts its ids (cb, ok, err) on the executor's list after the library accepted it. wait_completion releases the ids that were queued before it started, so a call that another thread queues during the wait keeps its callbacks. A wait that times out or is interrupted puts the ids back.
  • close releases everything the executor holds, including calls that were never waited for. Closing the database closes its executors, so it releases them too.
  • A call that the library refuses (a bad record id, for example) gives its ids back at once, as no callback of it will ever run.
  • on_ok and on_error are executor-wide, so a wait does not drop them. They keep one id each until close, and a new handler replaces the old id (the engine replaces the handler, and a test pins that).
  • new_edge and _statement now build their arguments before they register callbacks, so a bad argument cannot leave an id behind.

Python only. The library is unchanged.

Verification

New tests/test_native_async_callbacks.py (native only, listed in collect_ignore for the JVM build). Against the unfixed code, six of the first seven tests failed with growth (for example 71 ids against a baseline of 21 after one close).

  • Every kind of call, 25 rounds with wait_completion: the table size returns to the baseline after each round, and every kind ran its callback.
  • A poll with wait_completion(0) that finds the writers busy keeps the ids.
  • A refused call gives its ids back.
  • close without a wait, and closing the database, release the ids.
  • on_ok / on_error: two slots no matter how often they are set, kept across a wait, released on close, and the replaced on_ok does not fire.
  • Four producer threads queue 200 commands while the main thread loops wait_completion: all 200 callbacks run and the table returns to the baseline.

Run on the native build (ARCADEDB_BACKEND=native, library image: the newest local build, native-lib/winlog of 2026-10-26; no Java change is needed): test_async_executor, test_async_transaction, test_native_async_callbacks, test_python_callbacks, test_concurrency and test_core give 118 passed. Bandit (-c pyproject.toml -r src tests --severity-level low --confidence-level low), isort --profile black, black and scripts/check_no_embedded_name.py are clean.

🤖 Generated with Claude Code

Only AsyncExecutor.transaction gave its callback ids back. create_record, query, command, new_edge, new_edge_by_keys and scan_type registered an id per callback in extras._callbacks and never released it, so the table grew without bound over a long-lived executor, and on_ok / on_error added two more each time they were set.

All calls now go through one _queue helper. A queued call puts its ids on the executor's list after the library accepted it; wait_completion releases the ids that were queued before it started (a call queued by another thread during the wait keeps its callbacks, and a timed-out or interrupted wait puts the ids back); close releases everything. A call the library refuses gives its ids back at once. on_ok and on_error keep one id each until close, and a new handler replaces the old id.

Python only: no change to the library.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit e60a748 into main Oct 25, 2026
27 checks passed
@tae898
tae898 deleted the native-async-callback-release branch October 25, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant