Repository navigation
[BUG]: Wrong results when two ct.mma calls share one accumulator operand in a loop iteration (tile 32x256, tn=16, sm_121) #113
Description
Version
cuda-tile 1.7.0
Version (CUDA Toolkit)
13.4 (nvcc 13.4.59; tileiras built 2026-08-03; driver 580.178.04 with the container's CUDA forward-compatibility libcuda 615.71.09)
Which installation method(s) does this occur on?
Pip (cuda-tile 1.7.0 installed with pip over the NGC PyTorch 26.09 image's cuda-tile 1.5.0; not tried from source)
Describe the bug.
Two ct.mma calls whose accumulator argument is the same value (here a zero tile) and whose results are added in the same loop iteration return wrong values for one tile shape, tm=32, tk>=256, tn=16, on a GB10 (sm_121); we would like the cuTile team to triage and fix it, and to tell us whether other shapes or architectures are affected. The same kernel is correct at neighboring tile shapes, and giving the two mma calls two different zero tiles loaded from memory gives correct results.
Expected: relative error against the exact result of about 1e-7 (the operands are rounded to TF32-representable values, so all products are exact). Observed: about 0.7 at (tm=32, tk=256) and 0.87 at (tm=32, tk=512), with finite, deterministic values of the right magnitude (no NaN or zeros), identical across repeats, input scales and matrix sizes from 64x256 to 5000x10000.
This is a silent wrong result, not a compiler crash, so it is different from the open tileiras SIGSEGV reports #96 and #97 (which also involve mma loops); we mention them only to help triage.
What we established (all on GB10, cuda-tile 1.7.0, one run each; we have not tried other GPUs or main):
- Fails for TF32, fp16 and bf16 operands alike.
- Fails only for
tn=16;tn=32andtn=64at the sametm, tkare correct. Fails attm=32, tk=256andtk=512;tm=16,tm=64,tm=128andtm=32, tk=128are correct. - One
ct.mmaper iteration is correct at the same tile. - Two
ct.fullzero tiles created separately are merged by the compiler into one value (visible in the TileIR below), so they behave like the shared case. Two zero tiles that cannot be merged (loaded from two memory arrays) give correct results. The same loaded zero tile used for bothmmacalls fails again. So the trigger is the same accumulator operand feeding twommacalls, not the zero constant itself. - A chained form,
p = mma(a1, x, zero); p = mma(a2, x, p); acc = acc + p, is correct at every tile we tried. num_ctas=2also makes the failing kernel correct;occupancy=1does not.- Consistent with, but not proof of, the second
mmastarting from the first one's result: that would give2*A1X + A2X, an error of||A1X|| / ||C|| = 1/sqrt(2) = 0.707for independent equal-scaleA1, A2, and we observe 0.70 to 0.72 attk=256. It does not explain 0.87 attk=512. We have not looked at the generated machine code.
Minimum reproducible example
"""cuTile: two mma calls sharing one accumulator operand in a loop iteration give wrong results at tile (tm=32, tk=256, tn=16).
Computes C = A1 @ X + A2 @ X. Operands are rounded to TF32-representable values, so every product is exact and the reference is exact.
Needs only cupy, numpy, cuda.tile."""
import math
import cupy as cp
import numpy as np
import cuda.tile as ct
CI = ct.Constant[int]
@ct.kernel
def shared_acc(A1, A2, X, Z1, Z2, O, tm: CI, tk: CI, tn: CI):
bid = ct.bid(0); z = ct.PaddingMode.ZERO
acc = ct.full((tm, tn), 0, dtype=ct.float32)
zero = ct.full((tm, tn), 0, dtype=ct.float32) # ONE zero tile feeds both mma
for k in range(ct.num_tiles(A1, axis=1, shape=(tm, tk))):
a1 = ct.load(A1, index=(bid, k), shape=(tm, tk), padding_mode=z).astype(ct.tfloat32)
a2 = ct.load(A2, index=(bid, k), shape=(tm, tk), padding_mode=z).astype(ct.tfloat32)
x = ct.load(X, index=(k, 0), shape=(tk, tn), padding_mode=z).astype(ct.tfloat32)
acc = acc + ct.mma(a1, x, zero)
acc = acc + ct.mma(a2, x, zero)
ct.store(O, index=(bid, 0), tile=acc)
@ct.kernel
def distinct_acc(A1, A2, X, Z1, Z2, O, tm: CI, tk: CI, tn: CI):
bid = ct.bid(0); z = ct.PaddingMode.ZERO
acc = ct.full((tm, tn), 0, dtype=ct.float32)
z1 = ct.load(Z1, index=(bid, 0), shape=(tm, tn), padding_mode=z) # two different zero tiles, loaded from memory
z2 = ct.load(Z2, index=(bid, 0), shape=(tm, tn), padding_mode=z)
for k in range(ct.num_tiles(A1, axis=1, shape=(tm, tk))):
a1 = ct.load(A1, index=(bid, k), shape=(tm, tk), padding_mode=z).astype(ct.tfloat32)
a2 = ct.load(A2, index=(bid, k), shape=(tm, tk), padding_mode=z).astype(ct.tfloat32)
x = ct.load(X, index=(k, 0), shape=(tk, tn), padding_mode=z).astype(ct.tfloat32)
acc = acc + ct.mma(a1, x, z1)
acc = acc + ct.mma(a2, x, z2)
ct.store(O, index=(bid, 0), tile=acc)
def tf32(a): # round float32 to 10 mantissa bits (nearest-even)
u = a.view(np.uint32).astype(np.uint64)
return ((u + ((u >> 13) & 1) + 0xFFF) & np.uint64(0xFFFFE000)).astype(np.uint32).view(np.float32)
rng = np.random.default_rng(0)
n, N, tn = 5000, 10000, 16
A1 = tf32((rng.standard_normal((n, N)) / np.sqrt(N)).astype(np.float32))
A2 = tf32((rng.standard_normal((n, N)) / np.sqrt(N)).astype(np.float32))
X = tf32(rng.standard_normal((N, tn)).astype(np.float32))
ref = A1.astype(np.float64) @ X.astype(np.float64) + A2.astype(np.float64) @ X.astype(np.float64)
Z1, Z2 = cp.zeros((n, tn), dtype=cp.float32), cp.zeros((n, tn), dtype=cp.float32)
print('cuda-tile', __import__('importlib.metadata', fromlist=['x']).version('cuda-tile'), '| GPU', cp.cuda.runtime.getDeviceProperties(0)['name'].decode())
for kernel in (shared_acc, distinct_acc):
for tm, tk in ((32, 128), (32, 256), (64, 256)):
O = cp.full((n, tn), cp.nan, dtype=cp.float32)
ct.launch(cp.cuda.get_current_stream(), (math.ceil(n / tm), 1, 1), kernel,
(cp.asarray(A1), cp.asarray(A2), cp.asarray(X), Z1, Z2, O, tm, tk, tn))
cp.cuda.Device().synchronize()
o = cp.asnumpy(O).astype(np.float64)
print(f'{kernel._pyfunc.__name__:12s} tile (tm={tm:3d}, tk={tk:3d}, tn={tn}): relative error vs float64 = {np.linalg.norm(o - ref) / np.linalg.norm(ref):.1e}')Relevant log output
cuda-tile 1.7.0 | GPU NVIDIA GB10
shared_acc tile (tm= 32, tk=128, tn=16): relative error vs float64 = 4.0e-07
shared_acc tile (tm= 32, tk=256, tn=16): relative error vs float64 = 7.1e-01
shared_acc tile (tm= 64, tk=256, tn=16): relative error vs float64 = 6.5e-07
distinct_acc tile (tm= 32, tk=128, tn=16): relative error vs float64 = 4.0e-07
distinct_acc tile (tm= 32, tk=256, tn=16): relative error vs float64 = 6.5e-07
distinct_acc tile (tm= 64, tk=256, tn=16): relative error vs float64 = 6.5e-07
TileIR of shared_acc at (tm=32, tk=256) (CUDA_TILE_DUMP_TILEIR); both mmaf take %cst_0_f32_37:
%cst_0_f32 = constant <f32: 0.000000e+00> : tile<32x16xf32>
%cst_0_f32_37 = constant <f32: 0.000000e+00> : tile<32x16xf32>
%6 = mmaf %3, %5, %cst_0_f32_37 : tile<32x256xtf32>, tile<256x16xtf32>, tile<32x16xf32>
%7 = addf %iterArg0, %6 : tile<32x16xf32>
%8 = mmaf %4, %5, %cst_0_f32_37 : tile<32x256xtf32>, tile<256x16xtf32>, tile<32x16xf32>
%9 = addf %7, %8 : tile<32x16xf32>
Full env printout
Output of print_env.sh from the cutile-python repository (run inside the container where the bug reproduces, from a directory outside any git repository):
Click here to see environment details
**git*** Not inside a git repository ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=24.04 DISTRIB_CODENAME=noble DISTRIB_DESCRIPTION="Ubuntu 24.04.4 LTS" PRETTY_NAME="Ubuntu 24.04.4 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.4 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble LOGO=ubuntu-logo Linux 54c5b75ab6ce 7.0.0-1019-nvidia #19~24.04.2-Ubuntu SMP PREEMPT_DYNAMIC Sun Sep 6 18:35:11 UTC 2 aarch64 aarch64 aarch64 GNU/Linux ***GPU Information*** Mon Oct 26 20:06:06 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.178.04 Driver Version: 580.178.04 CUDA Version: 13.4 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GB10 On | 0000000F:01:00.0 Off | N/A | | N/A 49C P0 7W / N/A | Not Supported | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ***CPU*** Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Vendor ID: ARM Model name: Cortex-X925 Model: 1 Thread(s) per core: 1 Core(s) per socket: 10 Socket(s): 1 Stepping: r0p1 Frequency boost: disabled CPU(s) scaling MHz: 100% CPU max MHz: 3900.0000 CPU min MHz: 1378.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt Model name: Cortex-A725 Model: 1 Thread(s) per core: 1 Core(s) per socket: 10 Socket(s): 1 Stepping: r0p1 CPU(s) scaling MHz: 100% CPU max MHz: 2808.0000 CPU min MHz: 338.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt L1d cache: 1.3 MiB (20 instances) L1i cache: 1.3 MiB (20 instances) L2 cache: 25 MiB (20 instances) L3 cache: 24 MiB (2 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-19 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ***CMake*** /usr/local/bin/cmake cmake version 3.31.6 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Copyright (C) 2023 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2026 NVIDIA Corporation Built on Mon_Aug_03_13:24:39_PDT_2026 Cuda compilation tools, release 13.4, V13.4.59 Build cuda_13.4.r13.4/compiler.38657139_0 ***Python*** /usr/bin/python Python 3.12.3 ***Environment Variables*** PATH : /usr/local/lib/python3.12/dist-packages/torch_tensorrt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/amazon/efa/bin:/opt/tensorrt/bin LD_LIBRARY_PATH : /usr/local/cuda/compat/lib.real:/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : PYTHON_PATH : conda not found ***pip packages*** /usr/local/bin/pip Package Version Editable project location ---------------------------------------- ----------------------------------- ------------------------- absl-py 2.5.0 aiohappyeyeballs 2.7.1 aiohttp 3.14.3 aiosignal 1.4.0 annotated-types 0.8.0 antlr4-python3-runtime 4.9.3 anyio 4.15.1 apache-tvm-ffi 0.1.14 apex 0.1 argon2-cffi 25.1.0 argon2-cffi-bindings 26.1.0 array_record 0.8.3 arrow 1.4.0 asttokens 3.0.2 astunparse 1.6.3 async-lru 2.3.0 attrs 26.1.0 babel 2.18.0 beautifulsoup4 4.15.0 black 26.5.1 bleach 6.4.0 certifi 2026.7.22 cffi 2.1.1 charset-normalizer 3.5.1 click 8.5.0 cloudpickle 3.1.2 cmake 3.31.6 comm 0.2.3 contourpy 1.4.0 cuda-bindings 13.4.1 cuda-core 1.2.0 cuda-pathfinder 1.8.1 cuda-python 13.4.1 cuda-tile 1.7.0 cupti-python 13.4.0 cupy-cuda13x 14.2.0 cycler 0.12.1 Cython 3.3.0 datasets 4.7.0 debugpy 1.8.22 defusedxml 0.7.1 dill 0.4.0 dllist 2.0.0 docstring_parser 0.18.0 einops 0.8.2 etils 1.14.0 executing 2.2.1 fastjsonschema 2.22.2 filelock 3.32.7 flash_attn 2.8.3.post1+git91ffc2ac7c.63064690 flash-attn-4 4.0.0b24 fonttools 4.65.0 fqdn 1.5.1 frozenlist 1.8.0 fsspec 2026.2.0 gast 0.7.0 googleapis-common-protos 1.75.3 grain 0.2.18 grpcio 1.84.0 grpcio-tools 1.81.1 h11 0.16.0 hf-xet 1.6.0 httpcore 1.0.9 httpx 0.28.1 huggingface_hub 1.31.0 idna 3.19 importlib_metadata 9.0.1 iniconfig 2.3.1 ipykernel 7.3.0 ipython 9.17.1 ipython_pygments_lexers 1.1.1 isoduration 20.11.0 isort 9.0.1 jedi 0.20.0 Jinja2 3.1.6 json5 0.15.0 jsonpointer 3.1.1 jsonschema 4.26.0 jsonschema-specifications 2025.9.1 jupyter_builder 1.2.2 jupyter_client 8.10.0 jupyter_core 5.9.1 jupyter-events 0.12.1 jupyter-lsp 2.3.1 jupyter_server 2.21.1 jupyter_server_terminals 0.5.4 jupyterlab 4.6.3 jupyterlab_code_formatter 3.1.0 jupyterlab_pygments 0.3.0 jupyterlab_server 2.28.1 jupyterlab_tensorboard_pro 4.0.0 jupytext 1.19.5 kiwisolver 1.5.1 lark 1.3.1 libcst 1.9.0 llvmlite 0.46.0 makefun 1.16.0 Markdown 3.10.3 markdown-it-py 4.2.0 MarkupSafe 3.0.3 matplotlib 3.11.2 matplotlib-inline 0.2.2 mdit-py-plugins 0.6.1 mdurl 0.1.2 mistune 3.3.4 ml_dtypes 0.6.0 mpmath 1.3.0 multidict 6.8.0 multiprocess 0.70.18 mypy_extensions 1.1.0 nbclient 0.11.0 nbconvert 7.17.1 nbformat 5.11.1 nest-asyncio2 1.7.2 networkx 3.6.1 ninja 1.13.2 notebook 7.6.2 notebook_shim 0.2.4 numba 0.64.0 numpy 2.1.0 nvdlfw_inspect 0.2.2 nvidia-cuda-nvdisasm 13.4.49 nvidia-cudnn-frontend 1.28.0 nvidia-cutlass-dsl 4.8.0.dev0 nvidia-cutlass-dsl-libs-base 4.8.0.dev0 nvidia-cutlass-dsl-libs-core 4.8.0.dev0 nvidia-cutlass-dsl-libs-cu12 4.8.0.dev0 nvidia-cutlass-dsl-libs-cu13 4.8.0.dev0 nvidia-cutlass-operators 0.3.0.dev1 nvidia-dali-cuda130 2.3.0 nvidia-libnvcomp-cu13 5.3.0.16 nvidia-ml-py 13.610.43 nvidia-modelopt 0.46.0 nvidia-nvimgcodec-cu13 0.9.0.20 nvidia-nvjpeg 13.2.2.35 nvidia-nvjpeg2k-cu13 0.11.0.51 nvidia-nvtiff-cu13 0.8.0.82 nvidia-resiliency-ext 0.6.0 nvtx 0.2.16 omegaconf 2.3.1 onnx 1.21.0 onnx-ir 1.0.0 onnxscript 0.7.2 opentelemetry-api 1.44.0 opentelemetry-exporter-otlp-proto-common 1.44.0 opentelemetry-exporter-otlp-proto-http 1.44.0 opentelemetry-proto 1.44.0 opentelemetry-sdk 1.44.0 opentelemetry-semantic-conventions 0.65b0 optree 0.20.0 packaging 26.2 pandas 3.0.5 pandocfilters 1.5.1 parso 0.8.7 pathspec 1.1.1 pexpect 4.9.0 pillow 12.3.0 pip 26.2.1 platformdirs 4.11.8 pluggy 1.7.0 polygraphy 0.53.4 portpicker 1.6.0 prometheus_client 0.26.0 prompt_toolkit 3.0.53 propcache 0.5.4 protobuf 6.33.6 psutil 7.2.2 ptyprocess 0.7.0 PuLP 3.3.2 pure_eval 0.2.4 pyarrow 25.0.1 pybind11 3.1.0 pycocotools 2.0+nv0.8.1 pycparser 3.0 pydantic 2.13.5 pydantic_core 2.46.5 pyelftools 0.33 Pygments 2.17.2 pyparsing 3.3.2 pytest 9.1.1 python-dateutil 2.9.0.post0 python_hostlist 2.3.0 python-json-logger 4.2.0 pytokens 0.4.1 PyYAML 6.0.1 pyzmq 27.2.0 quack-kernels 0.6.2 referencing 0.37.0 regex 2026.9.10 requests 2.34.2 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rfc3987-syntax 1.1.0 rich 15.0.0 rpds-py 2026.6.3 safetensors 0.8.0 scipy 1.18.1 Send2Trash 2.1.0 setuptools 81.0.0 six 1.16.0 soupsieve 2.9.2 spmd_types 0.2.3 stack-data 0.6.3 sympy 1.14.0 tabulate 0.10.0 tensorboard 2.20.0 tensorboard-data-server 0.7.2 tensorrt 11.3.0.99 terminado 0.18.1 tinycss2 1.5.1 tokenizers 0.23.2 torch 2.14.0a0+b2c75dd062.nv26.9.68203377 torch_c_dlpack_ext 0.1.5 torch_checkpointing 0.1.0 torch_tensorrt 2.14.0a0 torchao 0.19.0+git4ee58b57 /opt/pytorch/ao torchtitan 0.2.2+git5c0b804c torchvision 0.29.0a0+0fba2e84.nv26.9.68203377 tornado 6.5.10 tqdm 4.70.1 traitlets 5.16.1 transformer_engine 2.19.0+5e52befd triton 3.8.0+gitbd40f349.nv26.9 triton_kernels 1.0.0+gitbd40f349.nv26.9 typeguard 4.6.0 typing_extensions 4.16.0 typing-inspection 0.4.4 tyro 1.0.16 tzdata 2026.4 uri-template 1.3.0 urllib3 2.8.0 wandb 0.30.0 wcwidth 0.8.3 webcolors 25.10.0 webencodings 0.6.1 websocket-client 1.9.2 Werkzeug 3.1.8 wheel 0.48.0 xxhash 4.0.1 yarl 1.25.1 zipp 4.1.0
Activity
haijieg commentedon Oct 27, 2026
Thank you @adonoho for the detailed report. I've triaged the issue to be a tileiras backend bug on GB10 since 13.3. Will update this issue when the fix is landed in a new CTK.