Repository navigation
feat(cuda.core): add host_launch for stream-ordered host functions - #3258
Merged
Merged
Conversation
Add cuda.core.host_launch(stream, fn, *, user_data=None), the stream-side counterpart of GraphBuilder.callback. It enqueues a Python callable or a ctypes function pointer with the CUhostFn prototype on a stream through cuLaunchHostFunc; a GraphBuilder argument forwards to its callback. The callable and a copied user_data stay alive until the function has run and are released afterwards on the main thread through the deferred-cleanup queue, never on the driver thread. A plain stream under active capture is rejected, because the owners could not be attached to the captured node. The shared callback helper moves from cuda/core/graph/_host_callback to cuda/core/_host_callback, since a core module must not import the graph package at module level. The launch record and trampoline live in _cpp/rt/host_launch.cpp, and cuLaunchHostFunc joins the driver table. Written by Codex (gpt-6.1-sol); reviewed by Claude. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Andy-Jost
added
P1
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Runtime host-function launch cuda_core/cuda/core/_cpp/rt/api.hpp, cuda_core/cuda/core/_cpp/rt/driver_api.hpp, cuda_core/cuda/core/_cpp/rt/host_launch.cpp, cuda_core/cuda/core/_rt.pxd, cuda_core/cuda/core/_rt.pyx, cuda_core/cuda/core/_cpp/rt/DESIGN.md, cuda_core/cuda/core/_cpp/rt/GRAPH_ATTACHMENTS.md |
The runtime adds cuLaunchHostFunc support and retains callback and user-data owners until invocation. The trampoline queues the record for cleanup after the callback returns. |
Python API and graph integration cuda_core/cuda/core/_launcher.pyx, cuda_core/cuda/core/_launcher.pyi, cuda_core/cuda/core/_host_callback.pxd, cuda_core/cuda/core/_host_callback.pyx, cuda_core/cuda/core/_host_callback.pyi, cuda_core/cuda/core/graph/_graph_builder.pyx, cuda_core/cuda/core/graph/_graph_node.pyx, cuda_core/cuda/core/graph/_subclasses.pyx |
host_launch accepts streams and forwards GraphBuilder inputs to GraphBuilder.callback. It resolves callback inputs and rejects active stream capture. Graph modules now import callback helpers from cuda.core._host_callback. |
Validation, example, and API documentation cuda_core/tests/test_host_launch.py, cuda_core/examples/host_launch.py, cuda_core/docs/source/api.rst, cuda_core/docs/source/examples.rst, cuda_core/docs/source/release/1.3.0-notes.rst |
Tests cover ordering, callback lifetime, input validation, graph behavior, exceptions, and streams. The example demonstrates a callback between two kernels. The documentation lists and describes host_launch. |
Assessment against linked issues
| Objective | Addressed | Explanation |
|---|---|---|
Add a top-level API for scheduling Python callables or C function pointers in stream order [#2058] |
✅ | |
Add a corresponding HostNode type in cuda.core.graph._subclasses [#2058] |
❌ | The change forwards GraphBuilder inputs to GraphBuilder.callback, but does not add a HostNode type. |
Document callback threading restrictions, add an example, and add API reference entries [#2058] |
✅ |
Possibly related issues
- cuda.core: a host copy function with the CUhostFn signature for host_launch #3256: The change provides the
host_launchanduser_datamechanism, but does not add the compiled host-copy function. - cuda.core: release synchronous allocations in stream order through a host callback instead of blocking #2989: The change provides the host-callback primitive, but does not change resource deallocation behavior.
Priority:
Change: Feature
Merge Risk: 
66c55
-
The previously missing deadlock warning is present, and no actionable merge-blocking risk remains after normal checks.
Comment @coderabbitai help to get the list of available commands.
Andy-Jost
changed the title
State that a ctypes callback that wraps a Python function acquires the GIL through ctypes, so the deadlock rule applies to it; complete the Raises section; say the driver may reject CUDA calls from the callback rather than that it does. Upcast the launch record to the cleanup base before the void* parameter, as the graph producers do. Prove the user_data copy by changing the caller's buffer while the stream is held, pin the unraisable type and message, and link the example from the examples page. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
-
-
- Source mode: Auto
- Repository: NVIDIA/cuda-python
- Branch: main
- File: .coderabbit.yaml
- Commit: 94625c1
- Layers: Repository YAML
- Review profile: CHILL
- Plan: Enterprise
- Run ID:
7a78139b-bfca-4094-acd9-991e49bd39f7
-
cuda_core/cuda/core/_cpp/rt/DESIGN.mdcuda_core/cuda/core/_cpp/rt/GRAPH_ATTACHMENTS.mdcuda_core/cuda/core/_cpp/rt/host_launch.cppcuda_core/cuda/core/_launcher.pyicuda_core/cuda/core/_launcher.pyxcuda_core/docs/source/examples.rstcuda_core/docs/source/release/1.3.0-notes.rstcuda_core/tests/test_host_launch.py
-
- cuda_core/cuda/core/_cpp/rt/DESIGN.md
- cuda_core/cuda/core/_cpp/rt/GRAPH_ATTACHMENTS.md
- cuda_core/docs/source/release/1.3.0-notes.rst
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
-
| .. warning:: | ||
|
|
||
| The function must not call CUDA, directly or through a library. The | ||
| driver does not permit such calls and may reject them with an error. |
There was a problem hiding this comment.
suggestion: State that CUDA calls from the callback can deadlock.
The driver does not guarantee an error for a CUDA call from a host callback. Such a call can also deadlock. State that consequence here and regenerate the matching stub documentation. (docs.nvidia.com)
This was referenced Oct 28, 2026
CodeRabbit's review of host_launch: the driver does not guarantee an error for a CUDA call made from a host function; such a call may also deadlock. State both outcomes in the warning. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mdboom
approved these changes
Oct 30, 2026
| HANDLE_RETURN(cydriver.cuLaunchKernelEx(&drv_cfg, func_handle, args_ptr, NULL)) | ||
|
|
||
|
|
||
| def host_launch(stream: Stream | GraphBuilder | IsStreamType, fn, *, user_data=None) -> None: |
Contributor
There was a problem hiding this comment.
nit: Maybe a type annotation on fn? Definitely add one to user_data which is a lot more restricted than one might expect.
| # ################################################################################ | ||
|
|
||
| # /// script | ||
| # dependencies = ["cuda_bindings", "cuda_core"] |
Contributor
There was a problem hiding this comment.
Nit: Since this example doesn't import cuda_bindings directly, I would remove it as a direct dependency.
Contributor
Author
|
Thanks @mdboom. I'll take both nits in a follow-up PR:
Merging this one as is. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cuda.corehad host callbacks for graphs but no way to enqueue one on a plain stream, so mixed host and device ordering neededcuda.bindings. This PR addscuda.core.host_launch(stream, fn, *, user_data=None), the stream-side counterpart ofGraphBuilder.callback. It enqueues a host function on a stream in stream order throughcuLaunchHostFunc: a Python callable, which runs with the GIL, or a ctypes function pointer with theCUhostFnprototype, whichcuda.corepasses to the driver as is, so a pointer to compiled code runs without the GIL (a ctypes callback that wraps a Python function still acquires it through ctypes). Thefnanduser_dataforms and their validation are the ones the graph callback methods already use, and aGraphBuilderargument forwards to itscallback, so the same call works under stream capture.cuda.coreowns the callable and a copieduser_datauntil the function has run. The trampoline hands the launch record to the existing deferred-cleanup queue after the user function returns, so the owners are released on the main thread, never on the driver's callback thread, where releasing a Python object would take the GIL and a finalizer could call CUDA. A plainStreamunder active capture raisesRuntimeErrorand points toGraphBuilder.callback, because the record's owners could not be attached to the captured node and the record must not be reused by graph replays. A Python exception inside the callback is reported throughsys.unraisablehookand the stream continues, as on the graph path. The docstring states the driver's rules: the function must not call CUDA, and a thread that holds the GIL while blocked in a CUDA wait on the stream deadlocks with a Python callback.Changes
cuda_core/cuda/core/_launcher.pyx:host_launch, exported fromcuda.core.cuda_core/cuda/core/_host_callback.pyx,.pxd: the shared callback helper, moved fromcuda/core/graph/unchanged; a core module must not import the graph package at module level. The three graph cimports follow.cuda_core/cuda/core/_cpp/rt/host_launch.cpp: the launch record and trampoline behind one entry point,launch_host_function, declared inapi.hpp,_rt.pxd, and_rt.pyx;cuLaunchHostFuncjoins the driver table indriver_api.hpp.cuda_core/docs/source/api.rst:host_launchnext tolaunch;_cpp/rt/DESIGN.mdandGRAPH_ATTACHMENTS.mdname the new file and the queue's second producer.cuda_core/examples/host_launch.py: a host function ordered between two kernel launches, linked fromdocs/source/examples.rst.cuda_core/tests/test_host_launch.py: ordering between prior and later stream work, release of the callable after it ran, copied bytesuser_data, pass-through of an intuser_data, theTypeErrorandValueErrorcases, forwarding of aGraphBuilderwith two graph launches, rejection of a capturing stream without invalidating the capture, unraisable reporting with the stream still usable, two streams keeping their own data, rejection ofNone, and the deprecated stream-protocol path.Related Work
Closes #2058. The host copy function requested in #3256 builds on this.
🤖 Generated with Claude Code