You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I started from turning on all of the checks, and then walked back the ones that felt like were a strong diversion from the existing style. I commented the reason for turning each one off in the config file.
One thing to note -- running clang-tidy requires using the same compiler flags and access to headers as a full build of cuda-core. This is a bit at odds with running in a pre-commit hook, where it's generally important that it's totally hermetic (unaffected by the files installed on your system) for reproducibility and portability. For that reason, this installs the nvidia-cuda-runtime header package from PyPI, pinned to a version that is the same as the current CTK version floor. I think that's a reasonable choice, but it's possible it will give different results if building against a different version of the CTK.
Many C++ projects don't run it in a pre-commit hook but instead run it as part of the build. setuptools (our current build system) does not currently have clang-tidy integration, so we would need to build it as an extension or move to a more modern build tool.
EDIT: Another interesting finding is that because the C++ stdlib is significantly different on Linux and Windows (especially around which methods throw exceptions), the set of findings from clang-tidy can be quite different. This is mostly a good thing in terms of finding issues, but may be an additional barrier/hassle given that most of us are on Linux most of the time. The "Windows fixes" commit shows the additional things found there.
Added clang-tidy checks for CUDA C++ source files, using CUDA headers from an installed toolkit or NVIDIA CUDA wheels.
Enabled a broad set of static-analysis checks and configured warnings to fail the check.
Added the check to pre-commit so it can run automatically before commits.
Documentation
Updated contributor guidance with instructions for running the C++ checks and applying suggested fixes.
Bug Fixes
Improved resource cleanup when graph creation fails.
The pull request adds clang-tidy configuration, a runner, and a pre-commit hook for cuda_core C++ sources. It documents the workflow and updates runtime code with named buffer sizes and other lint-related changes.
Changes: clang-tidy setup and runtime updates
Layer / File(s)
Summary
Configure and run clang-tidy .clang-tidy, toolshed/run_clang_tidy.py, .pre-commit-config.yaml, CONTRIBUTING.md
Adds clang-tidy checks and a runner that uses CUDA headers from environment paths or CUDA wheels. Adds a pre-commit hook and contributor instructions.
Updates linkage, handle-tag names, declarations, lock guards, and formatting across runtime code. Adds a scoped analyzer suppression around cleanup-queue allocation and publication.
Possibly related PRs
NVIDIA/cuda-python#3133: Adds a clang-format pre-commit hook for the same C++ tree; this pull request adds a companion clang-tidy hook and related source updates.
Suggested labels:enhancement
Suggested reviewers:andy-jost, rparolin
Priority: Low
Change: Feature
Merge Risk: Low · up to b0a48
Capture cleanup can leak a graph after an allocation failure. Some lint workflows also require manual setup or workarounds; these limitations should be tracked.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Reviewing files that changed from the base of the PR and between f9212dd and 4e4c0e4.
Files selected for processing: 22
.clang-tidy
.pre-commit-config.yaml
CONTRIBUTING.md
cuda_core/cuda/core/_cpp/rt/context.cpp
cuda_core/cuda/core/_cpp/rt/context_scope.hpp
cuda_core/cuda/core/_cpp/rt/driver_api.cpp
cuda_core/cuda/core/_cpp/rt/driver_api.hpp
cuda_core/cuda/core/_cpp/rt/error.cpp
cuda_core/cuda/core/_cpp/rt/error.hpp
cuda_core/cuda/core/_cpp/rt/graph.cpp
cuda_core/cuda/core/_cpp/rt/graph_exec.cpp
cuda_core/cuda/core/_cpp/rt/internal.hpp
cuda_core/cuda/core/_cpp/rt/memory.cpp
cuda_core/cuda/core/_cpp/rt/program.cpp
cuda_core/cuda/core/_cpp/rt/py.hpp
cuda_core/cuda/core/_cpp/rt/py_driver_fns.cpp
cuda_core/cuda/core/_cpp/rt/py_report.cpp
cuda_core/cuda/core/_cpp/rt/stream.cpp
cuda_core/cuda/core/_cpp/rt/texture.cpp
cuda_core/cuda/core/_cpp/rt/types.hpp
cuda_core/cuda/core/_cpp/rt/virtual_memory.cpp
toolshed/run_clang_tidy.py
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Functional Correctness · Quick win
suggestion: Document the tool needed for direct runs. The pre-commit hook installs clang-tidy in its own environment, but the documented direct command requires clang-tidy on PATH. A contributor who follows only the pre-commit setup gets error: clang-tidy not found on PATH. Add an installation command with the hook’s pinned version before the direct-run command. (pre-commit.com)
The reason will be displayed to describe this comment to others. Learn more.
+1. The command also omits --cuda-wheel, so it reads cuda.h from the local toolkit instead of the pinned wheel that the hook uses, and the results can differ.
The reason will be displayed to describe this comment to others. Learn more.
@Andy-Jost suggestion: Document installation of clang-tidy and the CUDA wheel dependencies at the versions pinned in .pre-commit-config.yaml. Then use this command to select wheel headers, as the hook does:
The reason will be displayed to describe this comment to others. Learn more.
Minor · Functional Correctness · Quick win
suggestion: Select the requested CUDA wheel major. If an environment has both nvidia/cu12 and nvidia/cu13, the sorted search returns cu12 first. The runner then analyzes against CUDA 12 headers even when invoked by the hook that pins CUDA 13. Select the major of the installed runtime wheel instead of the first directory containing cuda.h. (raw.githubusercontent.com)
The reason will be displayed to describe this comment to others. Learn more.
Minor · Functional Correctness · Quick win
suggestion: Preserve values of forwarded clang-tidy options. For --checks bugprone-* FILE.cpp, parse_known_args treats bugprone-* as a positional file. The suffix filter removes that value and forwards --checks without its argument. Parse source filenames without consuming unknown option values, or require and document the --checks=bugprone-* form. (docs.python.org)
The reason will be displayed to describe this comment to others. Learn more.
@Andy-Jost suggestion: Yes. Documenting the --option=value requirement in the module usage text and _parser() help is enough. No parser change is needed for that documented interface.
Suggested wording:
For clang-tidy options that take a value, use --option=value
(for example, --checks=bugprone-*), not --option value.
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🔇 Additional comments: 1
.clang-tidy-44-44: 1
44-44: Major · Stability & Availability · Heavy lift
⚠️Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.
important: Keep bugprone-throwing-static-initialization enabled or make registry initialization non-throwing.
This exclusion suppresses the diagnostic for the process-lifetime registries described in the file. The runtime defines non-local context_registry and graph_registry objects backed by HandleRegistry; if initialization throws, module loading can fail before Python can report the error. Move these registries to explicit or lazy initialization, or use a targeted suppression only after proving the constructor is safe. (raw.githubusercontent.com)
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
Reviewing files that changed from the base of the PR and between 1927fa5 and b0a4848.
Files selected for processing: 1
cuda_core/cuda/core/_cpp/rt/graph.cpp
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Stability & Availability · Quick win
Supported by static analysis
Script executed:
sed -n '460,525p;1050,1130p' cuda_core/cuda/core/graph/_graph_builder.pyx
sed -n '285,340p' cuda_core/cuda/core/_cpp/rt/graph.cpp
git diff 20a4aeeebd2d3226955d3caa142bd19b4d828ced b0a484869e4da84fb317cd2c3e25175144bd1592 -- cuda_core/cuda/core/_cpp/rt/graph.cpp
Repository: NVIDIA/cuda-python
Length of output: 12295
Web query:
site:docs.nvidia.com/cuda/cuda-driver-api cuGraphDestroy active stream capture cuStreamEndCapture graph ownership contract
Result:
<source_evidence>
<source kind="find_in_page">
<title>6.20. Graph Management — CUDA Driver API Reference Manual 13.4 documentation</title>
<location>https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html</location>
<excerpt>citeturn2view1 [wordlim: 200] Crawled: today; Content type: text/html; Source: find({"ref_id":"turn1view0","pattern":"Destroys a graph"}); Total lines: 3422
L18: CUresult 【4†cuDeviceSetGraphMemAttribute】(CUdevice device, CUgraphMem_attribute attr, void *value)
L19:
L20:
L21: Set asynchronous allocation attributes related to graphs.
L22:
L23: CUresult 【5†cuGraphAddBatchMemOpNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, const CUDA_BATCH_MEM_OP_NODE_PARAMS *nodeParams)
L24:
L25:
L26: Creates a batch memory operation node and adds it to a graph.
L27: CUresult 【6†cuGraphAddChildGraphNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, CUgraph childGraph)
L28:
L29:
L30: Creates a child graph node and adds it to a graph.
L31:
L32: CUresult 【7†cuGraphAddDependencies】(CUgraph hGraph, const CUgraphNode *from, const CUgraphNode *to, const CUgraphEdgeData *edgeData, size_t numDependencies)
L33:
L34:
L35: Adds dependency edges to a graph.
L36: CUresult 【8†cuGraphAddEmptyNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies)
L37:
L38:
L39: Creates an empty node and adds it to a graph.
L40:
L41: CUresult 【9†cuGraphAddEventRecordNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, CUevent event)
L42:
L43:
L44: Creates an event record node and adds it to a graph.
L45: CUresult 【10†cuGraphAddEventWaitNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, CUevent event)
L46:
L47:
L48: Creates an event wait node and adds it to a graph.
L49:
L50: CUresult 【11†cuGraphAddExternalSemaphoresSignalNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, const CUDA_EXT_SEM_SIGNAL_NODE_PARAMS *nodeParams)
L51:
L52:
L53: Creates an external semaphore signal node and adds it to a graph.
L54: CUresult 【12†cuGraphAddExternalSemaphoresWaitNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, const CUDA_EXT_SEM_WAIT_NODE_PARAMS *nodeParams)
L55:
L56:
L57: Creates an external semaphore wait node and adds it to a graph.
L58:
L59: CUresult 【13†cuGraphAddHostNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, const CUDA_HOST_NODE_PARAMS *nodeParams)
L60:
L61:
L62: Creates a host execution node and adds it to a graph.
L63: CUresult 【14†cuGraphAddKernelNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, const CUDA_KERNEL_NODE_PARAMS *nodeParams)
L64:
L65:
L66: Creates a kernel execution node and adds it to a graph.
L67:
L68: CUresult 【15†cuGraphAddMemAllocNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, CUDA_MEM_ALLOC_NODE_PARAMS *nodeParams)
L69:
L70:
L71: Creates an allocation node and adds it to a graph.
L72: CUresult 【16†cuGraphAddMemFreeNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, CUdeviceptr dptr)
L73:
L74:
L75: Creates a memory free node and adds it to a graph.
L76:
L77: CUresult 【17†cuGraphAddMemcpyNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, const CUDA_MEMCPY3D *copyParams, CUcontext ctx)
L78:
L79:
L80: Creates a memcpy node and adds it to a graph.
L81: CUresult 【18†cuGraphAddMemsetNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, size_t numDependencies, const CUDA_MEMSET_NODE_PARAMS *memsetParams, CUcontext ctx)
L82:
L83:
L84: Creates a memset node and adds it to a graph.
L85:
L86: CUresult 【19†cuGraphAddNode】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, const CUgraphEdgeData *dependencyData, size_t numDependencies, CUgraphNodeParams *nodeParams)
L87:
L88:
L89: Adds a node of arbitrary type to a graph.
L90: CUresult 【20†cuGraphAddNode_v3】(CUgraphNode *phGraphNode, CUgraph hGraph, const CUgraphNode *dependencies, const CUgraphEdgeData *dependencyData, size_t numDependencies, CUgraphNodeParams *nodeParams)
L91:
L92:
L93: Adds a node of arbitrary type to a graph.
L94:
L95: CUresult 【21†cuGraphBatchMemOpNodeGetParams】(CUgraphNode hNode, CUDA_BATCH_MEM_OP_NODE_PARAMS *nodeParams_out)
L96:
L97:
L98: Returns a batch mem op node's parameters.
L99: CUresult 【22†cuGraphBatchMemOpNodeSetParams】(CUgraphNode hNode, const CUDA_BATCH_MEM_OP_NODE_PARAMS *nodeParams)
L100:
L101:
L102: Sets a batch mem op node's parameters.
L103:
L104: CUresult 【23†cuGraphChildGraphNodeGetGraph】(CUgraphNode hNode, CUgraph *phGraph)
L105:
L106:
L107: Gets a handle to the embedded graph of a child graph node.
L108:
L109: CUresult 【24†cuGraphClone】(CUgraph *phGraphClone, CUgraph originalGraph)
L110:
L111:
L112: Clones a graph.
L113: CUresult 【25†cuGraphConditionalHandleCreate】(CUgraphConditionalHandle *pHandle_out, CUgraph hGraph, CUcontext ctx, unsigned int defaultLaunchValue, unsigned int flags)
L114:
L115:
L116: Create a conditional handle.
L117:
L118: CUresult 【26†cuGraphCreate】(CUgraph *phGraph, unsigned int flags)
L119:
L120:
L121: Creates a graph.
L122:
L123: CUresult 【27†cuGraphDebugDotPrint】(CUgraph hGraph, const char *path, unsigned int flags)
L124:
L125:
L126: Write a DOT file describing graph structure.
L127:
L128: CUresult 【28†cuGraphDestroy】(CUgraph hGraph)
L129:
L130:
L131: Destroys a graph.
L132: CUresult 【29†cuGraphDestroyNode】(CUgraphNode hNode)
L133:
L134:
L135: Remove a node from the graph.
L136:
L137: CUresult 【30†cuGraphEventRecordNodeGetEvent】(CUgraphNode hNode, CUevent *event_out)
L138:
L139:
L140: Returns the event associated with an event record node.
L141:
L142: CUresult 【31†cuGraphEventRecordNodeSetEvent】(CUgraphNode hNode, CUevent event)
L143:
L144:
L145: Sets an event record node's event.
L146:
L147: CUresult 【32†cuGraphEventWaitNodeGetEvent】(CUgraphNode hNode, CUevent *event_out)
L148:
L149:
L150: Returns the event associated with an event wait node.
L151: CUresult 【33†cuGraphEventWaitNodeSetEvent】(CUgraphNode hNode, CUevent event)
L152:
L153:
L154: Sets an event wait node's event.
L155:
L156: CUresult 【34†cuGraphExecBatchMemOpNodeSetParams】(CUgraphExec hGraphExec, CUgraphNode hNode, const CUDA_BATCH_MEM_OP_NODE_PARAMS *nodeParams)
L157:
L158:
L159: Sets the parameters for a batch mem op node in the given graphExec.
L160:
L161: CUresult 【35†cuGraphExecChildGraphNodeSetParams】(CUgraphExec hGraphExec, CUgraphNode hNode, CUgraph childGraph)
L207: CUresult 【46†cuGraphExecMemsetNodeSetParams】(CUgraphExec hGraphExec, CUgraphNode hNode, const CUDA_MEMSET_NODE_PARAMS *memsetParams, CUcontext ctx)
L208:
L209:
L210: Sets the parameters for a memset node in the given graphExec.
L211:
L212: CUresult 【47†cuGraphExecNodeSetParams】(CUgraphExec hGraphExec, CUgraphNode hNode, CUgraphNodeParams *nodeParams)
L213:
L214:
L215: Update a graph node's parameters in an instantiated graph.
L216: CUresult 【48†cuGraphExecUpdate】(CUgraphExec hGraphExec, CUgraph hGraph, CUgraphExecUpdateResultInfo *resultInfo)
L217:
L218:
L219: Check whether an executable graph can be updated with a graph and perform the update if possible.
L220:
L221: CUresult 【49†cuGraphExternalSemaphoresSignalNodeGetParams】(CUgraphNode hNode, CUDA_EXT_SEM_SIGNAL_NODE_PARAMS *params_out)
L222:
L223:
L224: Returns an external semaphore signal node's parameters.
L236: Sets an external semaphore wait node's parameters.
L237:
L238: CUresult 【53†cuGraphGetEdges】(CUgraph hGraph, CUgraphNode *from, CUgraphNode *to, CUgraphEdgeData *edgeData, size_t *numEdges)
L239:
L240:
L241: Returns a graph's dependency edges.
L242:
L243: CUresult 【54†cuGraphGetId】(CUgraph hGraph, unsigned int *graphId)
L244:
L245:
L246: Returns the id of a given graph.
L247:
L248: CUresult 【55†cuGraphGetNodes】(CUgraph hGraph, CUgraphNode *nodes, size_t *numNodes)
L249:
L250:
L251: Returns a graph's nodes.
L252: CUresult 【56†cuGraphGetRootNodes】(CUgraph hGraph, CUgraphNode *rootNodes, size_t *numRootNodes)
L253:
L254:
L255: Returns a graph's root nodes.
L256:
L257: CUresult 【57†cuGraphHostNodeGetParams】(CUgraphNode hNode, CUDA_HOST_NODE_PARAMS *nodeParams)
L258:
L259:
L260: Returns a host node's parameters.
L261:
L262: CUresult 【58†cuGraphHostNodeSetParams】(CUgraphNode hNode, const CUDA_HOST_NODE_PARAMS *nodeParams)
L263:
L264:
L265: Sets a host node's parameters.
L266:
L267: CUresult 【59†cuGraphInstantiate】(CUgraphExec *phGraphExec, CUgraph hGraph, unsigned long long flags)
L268: Creates an executable graph from a graph.
L269:
L270: CUresult 【60†cuGraphInstantiateWithParams】(CUgraphExec *phGraphExec, CUgraph hGraph, CUDA_GRAPH_INSTANTIATE_PARAMS *instantiateParams)
L271:
L272:
L273: Creates an executable graph from a graph.
L274:
L275: CUresult 【61†cuGraphKernelNodeCopyAttributes】(CUgraphNode dst, CUgraphNode src)
L276:
L277:
L278: Copies attributes from source node to destination node.
L279:
L280: CUresult 【62†cuGraphKernelNodeGetAttribute】(CUgraphNode hNode, CUkernelNodeAttrID attr, CUkernelNodeAttrValue *value_out)
L357: CUresult 【79†cuGraphNodeGetParams】(CUgraphNode hNode, CUgraphNodeParams *nodeParams)
L358:
L359:
L360: Return a graph node's parameters.
L361:
L362: CUresult 【80†cuGraphNodeGetToolsId】(CUgraphNode hNode, unsigned long long *toolsNodeId)
L363:
L364:
L365: Returns an id used by tools to identify a given node.
L366:
L367: CUresult 【81†cuGraphNodeGetType】(CUgraphNode hNode, CUgraphNodeType *type)
L368:
L369:
L370: Returns a node's type.
L371:
L372: CUresult 【82†cuGraphNodeSetEnabled】(CUgraphExec hGraphExec, CUgraphNode hNode, unsigned int isEnabled)
L373: Enables or disables the specified node in the given graphExec.
L374:
L375: CUresult 【83†cuGraphNodeSetParams】(CUgraphNode hNode, CUgraphNodeParams *nodeParams)
L376:
L377:
L378: Update a graph node's parameters.
L379:
L380: CUresult 【84†cuGraphNodeSetParams_v2】(CUgraphNode hNode, CUgraphNodeParams *nodeParams)
L381:
L382:
L383: Update a graph node's parameters.
L384:
L385: CUresult 【85†cuGraphReleaseUserObject】(CUgraph graph, CUuserObject object, unsigned int count)
L386:
L387:
L388: Release a user object reference from a graph.
L389: CUresult 【86†cuGraphRemoveDependencies】(CUgraph hGraph, const CUgraphNode *from, const CUgraphNode *to, const CUgraphEdgeData *edgeData, size_t numDependencies)
L390:
L391:
L392: Removes dependency edges from a graph.
L393:
L394: CUresult 【87†cuGraphRetainUserObject】(CUgraph graph, CUuserObject object, unsigned int count, unsigned int flags)
L395:
L396:
L397: Retain a reference to a user object from a graph.
L398:
L399: CUresult 【88†cuGraphUpload】(CUgraphExec hGraphExec, CUstream hStream)
L400:
L401:
L402: Uploads an executable graph in a stream.
L1341: * flags – - Flags from CUgraphDebugDot_flags for specifying which additional node information to write
L1342:
L1343: Returns
L1344:
L1345:
L1346: 【101†CUDA_SUCCESS】, 【112†CUDA_ERROR_INVALID_VALUE】, 【169†CUDA_ERROR_OPERATING_SYSTEM】
L1347:
L1348: 【93†CUresult】 cuGraphDestroy(【106†CUgraph】 hGraph)【170†】
L1349:
L1350:
L1351: Destroys a graph.
L1352:
L1353: Destroys the graph specified by `hGraph`, as well as all of its nodes.
L1354:
L1355: See also
L1356:
L1357: 【26†cuGraphCreate】
L1358:
L1359: Note
L1360:
L1361: Graph objects are not threadsafe. More here.
L1362:
L1363: Note
</excerpt>
</source>
<source kind="find_in_page">
<title>6.36. Stream Management — CUDA Driver API Reference Manual 13.4 documentation</title>
<location>https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__STREAM.html</location>
<excerpt>citeturn2view2 [wordlim: 200] Crawled: today; Content type: text/html; Source: find({"ref_id":"turn1view1","pattern":"cuStreamEndCapture("}); Total lines: 1059
L59: CUresult 【13†cuStreamEndCapture】(CUstream hStream, CUgraph *phGraph)
L60:
L61:
L62: Ends capture on a stream, returning the captured graph.
L63:
L64: CUresult 【14†cuStreamEndCaptureToCig】(CUstream hStream)
L65:
L66:
L67: Ends CIG capture on a stream.
L68:
L69: CUresult 【15†cuStreamGetAttribute】(CUstream hStream, CUstreamAttrID attr, CUstreamAttrValue *value_out)
L70:
L71:
L72: Queries stream attribute.
L263: If `mode` is not CU_STREAM_CAPTURE_MODE_RELAXED, 【13†cuStreamEndCapture】 must be called on this stream from the same thread.
L264:
L265: See also
L266:
L267: 【10†cuStreamCreate】, 【23†cuStreamIsCapturing】, 【13†cuStreamEndCapture】, 【29†cuThreadExchangeStreamCaptureMode】
L268:
L269: Note
L270:
L271: Kernels captured using this API must not use texture and surface references. Reading or writing through any texture or surface reference is undefined behavior. This restriction does not apply to texture and surface objects.
L272:
L273: Note
L328: Begin graph capture on `hStream`, placing new nodes into an existing graph. When a stream is in capture mode, all operations pushed into the stream will not be executed, but will instead be captured into `hGraph`. The graph will not be instantiable until the user calls 【13†cuStreamEndCapture】.
L329: Capture may not be initiated if `stream` is CU_STREAM_LEGACY. Capture must be ended on the same stream in which it was initiated, and it may only be initiated if the stream is not already in capture mode. The capture mode may be queried via 【23†cuStreamIsCapturing】. A unique id representing the capture sequence may be queried via cuStreamGetCaptureInfo.
L330:
L331: If `mode` is not CU_STREAM_CAPTURE_MODE_RELAXED, 【13†cuStreamEndCapture】 must be called on this stream from the same thread.
L332:
L333: See also
L334: cuStreamBeginCapture, 【10†cuStreamCreate】, 【23†cuStreamIsCapturing】, 【13†cuStreamEndCapture】, 【29†cuThreadExchangeStreamCaptureMode】, cuGraphAddNode
L335:
L336: Note
L337:
L338: Kernels captured using this API must not use texture and surface references. Reading or writing through any texture or surface reference is undefined behavior. This restriction does not apply to texture and surface objects.
L339:
L340: Note
L341:
L342: Note that this function may also return error codes from previous, asynchronous launches.
L343:
L344: Parameters
L377: 【10†cuStreamCreate】, cuStreamBeginCapture, 【23†cuStreamIsCapturing】, 【13†cuStreamEndCapture】, 【29†cuThreadExchangeStreamCaptureMode】
L378:
L379: Note
L380:
L381: Any user objects associated with `graph` will be released prior to the recapture.
L382:
L383: Note
L384:
L385: Note that this function may also return error codes from previous, asynchronous launches.
L386:
L387: Parameters
L388:
L389:
L390: * hStream – - Stream in which to initiate capture
L627: * graph_out – - Optional location to return the graph being captured into. All operations other than destroy and node removal are permitted on the graph while the capture sequence is in progress. This API does not transfer ownership of the graph, which is transferred or destroyed at 【13†cuStreamEndCapture】. Note that the graph handle may be invalidated before end of capture for certain errors.
L628: Nodes that are or become unreachable from the original stream at 【13†cuStreamEndCapture】 due to direct actions on the graph do not trigger 【99†CUDA_ERROR_STREAM_CAPTURE_UNJOINED】.
L629: * dependencies_out – - Optional location to store a pointer to an array of nodes. The next node to be captured in the stream will depend on this set of nodes, absent operations such as event wait which modify this set. The array pointer is valid until the next API call which operates on the stream or until the capture is terminated. The node handles may be copied out and are valid until they or the graph is destroyed.
L810: * 【112†CU_STREAM_CAPTURE_STATUS_NONE】: The stream is not capturing.
L811:
L812: * 【97†CU_STREAM_CAPTURE_STATUS_ACTIVE】: The stream is capturing.
L813:
L814: * 【113†CU_STREAM_CAPTURE_STATUS_INVALIDATED】: The stream was capturing but an error has invalidated the capture sequence. The capture sequence must be terminated with 【13†cuStreamEndCapture】 on the stream where it was initiated in order to continue using `hStream`.
L997: Therefore, stream capture places restrictions on API calls that can be made within or concurrently to a cuStreamBeginCapture-【13†cuStreamEndCapture】 sequence. This behavior can be controlled via this API and flags to cuStreamBeginCapture.
L998:
L999: A thread’s mode is one of the following:
</excerpt>
</source>
<source kind="open_page">
<title>6.20. Graph Management — CUDA Driver API Reference Manual 13.4 documentation</title>
<location>https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html</location>
<excerpt>citeturn1view0 [wordlim: 200] Crawled: today; Content type: text/html; Source: open({"ref_id":"turn0search1","lineno":null}); Total lines: 3422
L2385: Instantiates `hGraph` as an executable graph. The graph is validated for any structural constraints or intra-node constraints which were not previously validated. If instantiation is successful, a handle to the instantiated graph is returned in `phGraphExec`.
L2386:
L2387: The `flags` parameter controls the behavior of instantiation and subsequent graph launches. Valid flags are:
L2388: * 【143†CUDA_GRAPH_INSTANTIATE_FLAG_AUTO_FREE_ON_LAUNCH】, which configures a graph containing memory allocation nodes to automatically free any unfreed memory allocations before the graph is relaunched.
L2389: * 【208†CUDA_GRAPH_INSTANTIATE_FLAG_DEVICE_LAUNCH】, which configures the graph for launch from the device. If this flag is passed, the executable graph handle returned can be used to launch the graph from both the host and device. This flag can only be used on platforms which support unified addressing. This flag cannot be used in conjunction with 【143†CUDA_GRAPH_INSTANTIATE_FLAG_AUTO_FREE_ON_LAUNCH】.
L2390: * 【209†CUDA_GRAPH_INSTANTIATE_FLAG_USE_NODE_PRIORITY】, which causes the graph to use the priorities from the per-node attributes rather than the priority of the launch stream during execution. Note that priorities are only available on kernel nodes, and are copied from stream priority during stream capture.
L2391: If `hGraph` contains any allocation or free nodes, there can be at most one executable graph in existence for that graph at a time. An attempt to instantiate a second executable graph before destroying the first with 【36†cuGraphExecDestroy】 will result in an error. The same also applies if `hGraph` contains any device-updatable kernel nodes.
L2392:
L2393: If `hGraph` contains kernels which call device-side cudaGraphLaunch() from multiple contexts, this will result in an error.
L2394: Graphs instantiated for launch on the device have additional restrictions which do not apply to host graphs:
L2395:
L2396: * The graph’s nodes must reside on a single context.
L2397:
L2398: * The graph can only contain kernel nodes, memcpy nodes, memset nodes, and child graph nodes.
L2399:
L2400: * The graph cannot be empty and must contain at least one kernel, memcpy, or memset node. Operation-specific restrictions are outlined below.
L2401:
L2402: * Kernel nodes:
L2403:
L2404: * Use of CUDA Dynamic Parallelism is not permitted.
L2405: * Cooperative launches are permitted as long as MPS is not in use.
L2406:
L2407: * Memcpy nodes:
L2408:
L2409: * Only copies involving device memory and/or pinned device-mapped host memory are permitted.
L2410:
L2411: * Copies involving CUDA arrays are not permitted.
L2412:
L2413: * Both operands must be accessible from the current context, and the current context must match the context of other nodes in the graph.
L2414:
L2415: See also
L2416:
L2417: cuGraphInstantiate, 【26†cuGraphCreate】, 【88†cuGraphUpload】, 【66†cuGraphLaunch】, 【36†cuGraphExecDestroy】
L2418:
L2419: Note
L2420: Graph objects are not threadsafe. More here.
L2421:
L2422: Note
L2423:
L2424: Note that this function may also return error codes from previous, asynchronous launches.
L2425:
L2426: Parameters
L2427:
L2428:
L2429: * phGraphExec – - Returns instantiated graph
L2430:
L2431: * hGraph – - Graph to instantiate
L2432:
L2433: * flags – - Flags to control instantiation. See 【210†CUgraphInstantiate_flags】.
L2434:
L2435: Returns
L2436:
L2437:
L2438: 【101†CUDA_SUCCESS】, 【109†CUDA_ERROR_DEINITIALIZED】, 【110†CUDA_ERROR_NOT_INITIALIZED】, 【112†CUDA_ERROR_INVALID_VALUE】
L2439: 【93†CUresult】 cuGraphInstantiateWithParams(【176†CUgraphExec】 *phGraphExec, 【106†CUgraph】 hGraph, 【211†CUDA_GRAPH_INSTANTIATE_PARAMS】 *instantiateParams)【212†】
L2440:
L2441:
L2442: Creates an executable graph from a graph.
L2443: Instantiates `hGraph` as an executable graph according to the `instantiateParams` structure. The graph is validated for any structu…[truncated]</excerpt>
</source>
<source kind="open_page">
<title>6.36. Stream Management — CUDA Driver API Reference Manual 13.4 documentation</title>
<location>https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__STREAM.html</location>
<excerpt>citeturn1view1 [wordlim: 200] Crawled: today; Content type: text/html; Source: open({"ref_id":"turn0search0","lineno":null}); Total lines: 1059
L619: Parameters
L620:
L621:
L622: * hStream – - The stream to query
L623:
L624: * captureStatus_out – - Location to return the capture status of the stream; required
L625:
L626: * id_out – - Optional location to return an id for the capture sequence, which is unique over the lifetime of the process
L627: * graph_out – - Optional location to return the graph being captured into. All operations other than destroy and node removal are permitted on the graph while the capture sequence is in progress. This API does not transfer ownership of the graph, which is transferred or destroyed at 【13†cuStreamEndCapture】. Note that the graph handle may be invalidated before end of capture for certain errors.
L628: Nodes that are or become unreachable from the original stream at 【13†cuStreamEndCapture】 due to direct actions on the graph do not trigger 【99†CUDA_ERROR_STREAM_CAPTURE_UNJOINED】.
L629: * dependencies_out – - Optional location to store a pointer to an array of nodes. The next node to be captured in the stream will depend on this set of nodes, absent operations such as event wait which modify this set. The array pointer is valid until the next API call which operates on the stream or until the capture is terminated. The node handles may be copied out and are valid until they or the graph is destroyed.
L630: The driver-owned array may also be passed directly to APIs that operate on the graph (not the stream) without copying.
L631: * edgeData_out – - Optional location to store a pointer to an array of graph edge data. This array parallels `dependencies_out`; the next node to be added has an edge to `dependencies_out`[i] with annotation `edgeData_out`[i] for each `i`. The array pointer is valid until the next API call which operates on the stream or until the capture is terminated.
L632:
L633: * numDependencies_out – - Optional location to store the size of the array returned in dependencies_out.
L634:
L635: Returns
L636: 【41†CUDA_SUCCESS】, 【61†CUDA_ERROR_INVALID_VALUE】, 【96†CUDA_ERROR_STREAM_CAPTURE_IMPLICIT】, 【98†CUDA_ERROR_LOSSY_QUERY】
L637:
L638: 【37†CUresult】 cuStreamGetCtx(【38†CUstream】 hStream, 【100†CUcontext】 *pctx)【101†】
L639:
L640:
L641: Query the context associated with a stream.
L642:
L643: Returns the CUDA context that the stream is associated with.
L644: If the stream was created via the API 【79†cuGreenCtxStreamCreate】, the returned context is equivalent to the one returned by 【102†cuCtxFromGreenCtx()】 on the green context associated with the stream at creation time.
L645:
L646: The stream handle `hStream` can refer to any of the following:
L647: * a stream created via any of the CUDA driver APIs such as 【10†cuStreamCreate】 and 【11†cuStreamCreateWithPriority】, or their runtime API equivalents such as ::cudaStreamCreate, ::cudaStreamCreateWithFlags and ::cudaStreamCreateWithPriority. The returned context is the context that was active in the calling thread when the stream was created. Passing an invalid handle will result in undefined behavior.
L648: * any of the special streams such as the NULL stream, 【95†CU_STREAM_LEGACY】 and 【103†CU_STREAM_PER_THREAD】. The runtime API equivalents of these are also accepted, which are NULL, ::cudaStreamLegacy and ::cudaStreamPerThread respectively. Specifying any of the special handles will return the context current to the calling thread. If no context is current to the calling thread, 【48†CUDA_ERROR_INVALID_CONTEXT】 is returned.
L649:
L650: See also
L651: cuStreamDestroy, 【11†cuStreamCreateWithPriority】, 【22†cuStreamGetPriority】, 【20†cuStreamGetFlags】, 【19†cuStreamGetDevice】 【28†cuStreamWaitEvent】, 【24†cuStreamQuery】, 【26†cuStreamSynchronize】, 【3†cuStreamAddCallback】, ::cudaStreamCreate, ::cudaStreamCreateWithFlags
L652:
L653: Note
L654:
L655: Note that this functio…[truncated]</excerpt>
</source>
<source kind="search">
<title>6.36. Stream Management — CUDA Driver API Reference Manual 13.4 documentation</title>
<location>https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__STREAM.html</location>
<excerpt>citeturn0search0 [wordlim: 200] Crawled: 2 weeks ago; cuStreamCreate, cuStreamBeginCapture, cuStreamIsCapturing, cuGraphDestroy ... * a stream created via any of the CUDA driver APIs such as cuStreamCreate, cuStreamCreateWithPriority and cuGreenCtxStreamCreate, or their runtime API equivalents such as ::cudaStreamCreate, ::cudaStreamCreateWithFlags and ::cudaStreamCreateWithPriority. ... Nodes that are removed from the dependency set via this API do not result in CUDA_ERROR_STREAM_CAPTURE_UNJOINED if they are unreachable from the stream at cuStreamEndCapture.
Begins graph capture on a stream to an existing graph.
CUresult cuStreamBeginRecaptureToGraph(CUstream hStream, CUstreamCaptureMode mode, CUgraph hGraph, CUgraphRecaptureCallback callbackFunc, void *userData)
Begin graph capture on a stream to an existing graph.
CUresult cuStreamCopyAttributes(CUstream dst, CUstream src)
Copies attributes from source stream to destination stream.
CUresult cuStreamCreate(CUstream *phStream, unsigned int Flags)
Create a stream.
CUresult cuStreamCreateWithPriority(CUstream *phStream, unsigned int flags, int priority)
Create a stream with the given priority.
CUresult cuStreamDestroy(CUstream hStream)
Destroys a stream.
CUresult cuStreamEndCapture(CUstream hStream, CUgraph *phGraph)
Ends capture on a stream, returning the captured graph.
CUresult cuStreamEndCaptureToCig(CUstream hStream)
Ends CIG capture on a stream.
CUresult cuStreamGetAttribute(CUstream hStream, CUstreamAttrID attr, CUstreamAttrValue *value_out)
CUresult cuStreamBeginCapture(CUstream hStream, CUstreamCaptureMode mode)
Begins graph capture on a stream.
Begin graph capture on `hStream`. When a stream is in capture mode, all operations pushed into the stream will not be executed, but will instead be captured into a graph, which will be returned via cuStreamEndCapture. Capture may not be initiated if `stream` is CU_STREAM_LEGACY. Capture must be ended on the same stream in which it was initiated, and it may only be initiated if the stream is not already in capture mode. The capture mode may be queried via cuStreamIsCapturing. A unique id representing the capture sequence may be queried via cuStreamGetCaptureInfo.
If `mode` is not CU_STREAM_CAPTURE_MODE_RELAXED, cuStreamEndCapture must be called on this stream from the same thread.
See also
cuStreamCreate, cuStreamIsCapturing, cuStreamEndCapture, cuThreadExchangeStreamCaptureMode
Parameters
* hStream – - Stream in which to initiate capture to CIG
* streamCigCaptureParams – - CIG capture parameters
Returns
CUDA_SUCCESS, CUDA_ERROR_DEINITIALIZED, CUDA_ERROR_NOT_INITIALIZED, CUDA_ERROR_NOT_SUPPORTED, CUDA_ERROR_INVALID_CONTEXT, CUDA_ERROR_INVALID_VALUE,
CUresult cuStreamBeginCaptureToGraph(CUstream hStream, CUgraph hGraph, const CUgraphNode *dependencies, const CUgraphEdgeData *dependencyData, size_t numDependencies, CUstreamCaptureMode mode)
Begins graph capture on a stream to an existing graph.
Begin graph capture on `hStream`, placing new nodes into an existing graph. When a stream is in capture mode, all operations pushed into the stream will not be executed, but will instead be captured into `hGraph`. The graph will not be instantiable until the user calls cuStreamEndCapture.
Capture may not be initiated if `stream` is CU_STREAM_LEGACY.
CUresult cuStreamDestroy(CUstream hStream)
Destroys a stream.
Destroys the stream specified by `hStream`.
In case the device is still doing work in the stream `hStream` when cuStreamDestroy() is called, the function will return immediately and the resources associated with `hStream` will be released automatically once the device has completed all work in `hStream`.
See also
cuStreamCreate, cuStreamWaitEvent, cuStreamQuery, cuStreamSynchronize, cuStreamAddCallback, ::cudaStreamDestroy
Note
Note that this function may also return error codes from previous, asynchronous lau…[truncated]</excerpt>
</source>
<omitted_sources count="15">Additional source records were omitted from this bounded result.</omitted_sources>
</source_evidence>
Citations:
- 1: https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__STREAM.html
- 2: https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__STREAM.html
Script executed:
set -o pipefail
printf'%s\n''--- repository references to capture graph ownership and end capture ---'
rg -n -F --glob '*.{pyx,pxd,cpp,hpp,rst,md,py}' -e 'cuStreamEndCapture' -e 'cuGraphDestroy' -e 'invalidate_root_graph_state' -e 'end_building' -e 'capture graph' cuda_core | head -240
printf'%s\n''--- relevant base/head source blocks ---'forrevin 20a4aeeebd2d3226955d3caa142bd19b4d828ced b0a484869e4da84fb317cd2c3e25175144bd1592;doprintf'\\nREV %s\\n'"$rev"
git show "$rev:cuda_core/cuda/core/graph/_graph_builder.pyx"| nl -ba | sed -n '1060,1125p'
git show "$rev:cuda_core/cuda/core/_cpp/rt/graph.cpp"| nl -ba | sed -n '285,345p'done
important: End the active capture before releasing its graph.
create_graph_handle can call cuGraphDestroy while the graph still belongs to an active primary capture. CUDA transfers or destroys that graph at cuStreamEndCapture and does not permit graph destruction during capture.
If the driver rejects the destroy call, begin_building() leaves _h_graph unset. The later GB_end_capture() can return a non-null graph, but its non-null branch only marks the capture ended and does not destroy the returned graph. The graph can leak.
Handle capture cleanup in the primary-capture caller. End the capture before destroying the returned graph. Keep the direct destruction path for GraphDefinition, which passes a non-capture graph.
The reason will be displayed to describe this comment to others. Learn more.
Thanks, Mike. This mostly looks good. There is one breaking change for developers on macOS, a few functional issues, and some nits; details are inline.
For the record, the C++ edits change runtime behavior in a few places:
GraphBox is no longer noexcept. On MSVC, an allocation failure in its constructor now propagates instead of terminating. The libstdc++ members are noexcept, so Linux is unchanged.
create_graph_handle now destroys the graph when an allocation fails (see the inline comment on graph.cpp).
Four error-message buffers grow: report_status_code from 256 to 512 bytes, the note_context_not_restored cause from 128 to 512, and the two in ensure_fn_table from 640 to 768 and from 384 to 512.
The reason will be displayed to describe this comment to others. Learn more.
This blocks every Mac developer from committing to the repo, including docs-only and Python-only changes. nvidia-cuda-runtime and nvidia-cuda-nvrtc 13.4.92 publish only manylinux and Windows wheels, with no sdist. pre-commit installs every hook's environment before it filters by file, so on macOS pre-commit run and the commit hook fail for all hooks. CONTRIBUTING says the hooks run on macOS. clang-tidy==22.1.8 has the same problem on Windows-on-ARM: there is no wheel, and the sdist builds LLVM.
Could we add environment markers to these three dependencies, as the config already does for tomli, and have run_clang_tidy.py exit 0 with a notice when the tool, the CUDA headers, or Python.h is missing? CI on Linux and Windows still enforces the check.
The reason will be displayed to describe this comment to others. Learn more.
This point makes me think (as I contemplated in my OP) that doing this in pre-commit is not the right way to go, since setting up a working development environment across all the configs is so fiddly. I think I'd rather have only one set of build environment setup (the one used for the actual build) and piggy-back on that. Plus, then you are sure you are linting with all of the exact same settings as you are building in. I think it probably makes sense to rework this as a setuptools extension. WDYT, @Andy-Jost?
The reason will be displayed to describe this comment to others. Learn more.
On the stream-capture path this graph is not ours to destroy. GraphBuilder.begin_building passes the graph from cuStreamGetCaptureInfo, and the Driver API says of graph_out: "All operations other than destroy and node removal are permitted on the graph while the capture sequence is in progress. This API does not transfer ownership of the graph, which is transferred or destroyed at cuStreamEndCapture."
Before this PR, an allocation failure here leaked the graph, which is what cuda_core/AGENTS.md asks for when ownership cannot be established ("a leak is always preferred to a use-after-free"). The other caller, GraphDefinition, owns its graph. The simplest fix is to drop the destroy and keep the leak; the alternative is to tell create_graph_handle whether the caller owns graph. This is the same point as CodeRabbit's thread on line 332.
The reason will be displayed to describe this comment to others. Learn more.
I think in both cases my agent and CodeRabbit suggested this change because of the noexcept being dropped from GraphBox (as flagged on Windows). It's probably best to revert all of these changes and just suppress the lint and address that separately (if we even need to, which I think you are saying we don't).
The reason will be displayed to describe this comment to others. Learn more.
toolshed/check_clang_tidy_flags.py does not exist in this PR or on main, so nothing checks that INCLUDE_DIRS, the C++ standard, and the header-wheel pin stay in step with build_hooks.py and the CTK floor. Could you add the check or drop the sentence?
Two related points: INCLUDE_DIRS has cuda/core, which the build does not use. Passing -DCUDA_CORE_BUILD_MAJOR=13 and -DCUDA_CORE_MIN_CUDA_VERSION=<floor> would let the #error checks in versions.hpp catch a stale pin.
The reason will be displayed to describe this comment to others. Learn more.
With files limited to .cpp files under _cpp/, a change to a header (_cpp/rt/*.hpp, _include/), to .clang-tidy, or to toolshed/run_clang_tidy.py skips the hook locally and then fails in CI's --all-files run. Could we widen the pattern to those paths and have the runner analyze every translation unit when it gets a non-.cpp path? A full run is cheap: the pre-commit jobs on this PR, which analyze every translation unit, finish within about a minute of the same jobs on main.
The reason will be displayed to describe this comment to others. Learn more.
+1. The command also omits --cuda-wheel, so it reads cuda.h from the local toolkit instead of the pinned wheel that the hook uses, and the results can differ.
Closing this for now. I plan to rework this to run as part of the existing build rather than outside of it in precommit. I think most of the issues relate to that, suggesting it's the wrong approach.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CI/CDCI/CD infrastructurecuda.coreEverything related to the cuda.core module
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the second half of https://github.com/NVIDIA-dev/cuda-python-private/issues/704 (the first half adding
clang-format).I started from turning on all of the checks, and then walked back the ones that felt like were a strong diversion from the existing style. I commented the reason for turning each one off in the config file.
One thing to note -- running
clang-tidyrequires using the same compiler flags and access to headers as a full build ofcuda-core. This is a bit at odds with running in apre-commithook, where it's generally important that it's totally hermetic (unaffected by the files installed on your system) for reproducibility and portability. For that reason, this installs the nvidia-cuda-runtime header package from PyPI, pinned to a version that is the same as the current CTK version floor. I think that's a reasonable choice, but it's possible it will give different results if building against a different version of the CTK.Many C++ projects don't run it in a pre-commit hook but instead run it as part of the build. setuptools (our current build system) does not currently have
clang-tidyintegration, so we would need to build it as an extension or move to a more modern build tool.EDIT: Another interesting finding is that because the C++ stdlib is significantly different on Linux and Windows (especially around which methods throw exceptions), the set of findings from
clang-tidycan be quite different. This is mostly a good thing in terms of finding issues, but may be an additional barrier/hassle given that most of us are on Linux most of the time. The "Windows fixes" commit shows the additional things found there.