[joelteply]

What this serves

Learning (plan path step 3) and one live engine (line 3b): the in-engine trainer (the dream stage) must never take the serving process down. On 2026-11-07 16:13Z the M5's engine aborted mid-act during a /train on Qwen3.6-35B-A3B and took five serving slots with it (continuum card 6be38d73; excerpt at ~/.continuum/evidence/live-read-20261107/engine-abort-1613Z.txt on the M5).

Cause (receipts)

• Abort: ggml/src/ggml.c:7797 GGML_ASSERT(!src1_needs_grads || kv_sunk || ggml_are_same_shape(src1, cgraph->grads[isrc1])). That is the shape check that runs after every op's backward in ggml_compute_backward. Call path: server_trainer::run -> opt_epoch -> opt_walk::train_chunk -> decode -> opt_ubatch -> ggml_opt_alloc -> ggml_opt_build -> ggml_build_backward_expand.
• build_moe_ffn normalizes the router weights as weights / clamp(sum_rows(weights), eps, inf) (src/llama-graph.cpp:2134-2144). The dividend is [n_expert_used, n_tokens] and the divisor is [1, n_tokens], so the divisor is broadcast. GGML_OP_DIV's backward gave the divisor grad * tensor / src1 in the dividend's shape and never summed it over the broadcast. ADD and MUL do that sum with ggml_repeat_back. Any LoRA upstream of a router (the default attn_q,attn_v targets included) makes that divisor need a gradient.
• How it got past ggml: CLAMP has a backward, and a training graph the gradient cannot pass through refuses by name instead of aborting the server #84: ggml: CLAMP has a backward, and a training graph the gradient cannot pass through refuses by name instead of aborting the server #84 (832d78c) opened the router path by adding CLAMP's backward. It was proved on the toy qwen35moe in tests/test-llama-archs.cpp:291, which sets EXPERT_USED_COUNT 1; with one expert the two shapes agree. The 35B-A3B uses 8.
• Reproduced here: with only the DIV repeat_back removed, the new test is refused at exactly this node: cannot be built through DIV (node "node_3"): its backward gave src[1] no gradient or one of another shape. The deployed build asserts there instead.
• "Only sometimes" (inference): the defect is deterministic for this model whenever a trainable tensor sits upstream of a router. The engine log on the M5 (current file plus the one rotated at 11-06 23:14) holds exactly one training on the served model line, the run that aborted. I found no successful 35B-A3B training run to compare against.
• Same gap in SUB's backward with a broadcast src1: src/models/delta-net-base.cpp:216 (g_cs - g_last) on the chunked delta-net path, used when the fused GDN op is not available. This one is latent.

Fix

• DIV and SUB: the gradient for a broadcast src1 is summed back onto it with ggml_repeat_back.

Isolation (the architectural part)

• ggml_compute_backward now returns its contract break instead of asserting: a source that needs a gradient was given none, or one of a different shape. The contract is checked in one place, so this covers that whole class of defect in every op, not just DIV.
• ggml_build_backward_expand_or_refuse (new, ggml.h) returns the refused node and the reason. That includes the existing mark-walk refusals (no backward, in-place write, non-float source). ggml_build_backward_expand is that call plus the abort, and the abort message now names the node.
• ggml_opt_build builds through ggml_build_backward_expand_or_refuse. On a refusal it sets opt_ctx->refusal, drops the half-built gb_grad/gb_opt, and returns false before anything is allocated. From there the existing opt_ubatch -> opt_fail path carries it to /train's status as state: "error" with the node named, and the process keeps serving. The same holds for /learn on the serving context.
• Not covered: per-op GGML_ASSERTs inside individual backward cases (for example MUL_MAT's ne checks) still abort. The mark walk already refuses the structural ones (ggml_backward_reaches_src). Converting the rest is follow-up work. No asserts are globally disabled.

Tests

• test-opt-dynamic-accum:
  • the router's normalization (2 experts, 1 token, with a clamp) trains one SGD step to the analytic update -lr * (w1, -w0) / (w0 + w1)^2;
  • a backward made mis-shaped by a wrong-shape accumulator is refused by name, not asserted.
• test-backend-ops grad mode: SUB, MUL, DIV and broadcast ADD are checked against numeric gradients with a broadcast src1, which covers the router's nr {k,1,..} shape. A refused build now counts as a FAIL, not "not supported".

Measured

These ran on the M5, CPU only (GGML_METAL=OFF, nice -n 19, ggml libraries only, own build dir, beside live serving):

• test-opt-dynamic-accum: OK. With the mutant (DIV repeat_back removed) it fails with the refusal above.
• test-backend-ops grad -o ADD,SUB,MUL,DIV (CPU): every broadcast ADD/SUB/MUL/DIV grad case passes against finite differences (19839/19840 cases matched by the filter). The one failure is test_flash_attn_ext_grad (hs=16,nh=4,nhkv=2,kv=24,nb=8,native=0, MAA 1.7e-4 > 1e-4). It also fails on the unmodified base 46c6644 (native=0 MAA 3.8e-4, native=1 MAA 1.2e-4), so its tolerance is flaky and the failure predates this change

Not built: Metal and CUDA. The fix adds REPEAT_BACK, which both backends already run (MUL_MAT_ID's backward uses it). Not run: a live /train on the 35B-A3B. That needs a deploy and is continuum's call.

Stack order with #85-#89

No file overlap. #85, #86, #88 and #89 touch include/llama.h, src/llama-context*.{h,cpp} (opt_init/opt_select/opt_release, not train_chunk/opt_ubatch) and tools/server/*. This PR touches ggml/include/ggml.h, ggml/src/ggml.c, ggml/src/ggml-opt.cpp and two tests. None of them changes the crash path (the backward build) or fixes this assert. #85 puts learning on the serving context (per-mind shadows), so without this PR a /learn on the 35B-A3B would hit the same DIV assert inside the serving context itself. Suggested order: this PR first, or in either order, since neither rebases onto the other.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo