[jiangkuaixue123]

Problem and behavior

Refs #53. Active acceptance: #57; historical BF16 preparation: #54.

The shared evaluator previously required a performance gain, preventing correctness-only experiments. This change adds an explicit correctness objective while preserving the default performance contract, and connects single-GPU Qwen3.8 text checks to the existing AB/BA runner.

Three scopes remain distinct: reference_self_check compares Transformers full-prefix and cached execution; native_text_smoke compares the reference with the independently implemented native engine in jiangkuaixue123/nano-vllm#1. The candidate owns its model forward and request state. native_service_functional compares that native public entrypoint with its real HTTP/SSE service, separately from reference-based numerical validation.

Delivered slice

• Authenticated immutable model intake, per-trial byte checks, distinct checkpoint/precision profiles and source/import identities.
• FP32 full-vocabulary observations, exact token/case coverage, replay and caller-supplied numerical gates.
• Official FP8 support with actual loaded FP8 tensor coverage, lossless scale normalization, native per-layer execution checks and no dequantized fallback claims.
• Offline reference kernel binding to an immutable kernel-registry revision and complete source fingerprint, checked again after execution.
• Missing/malformed/stale evidence yields INCONCLUSIVE; complete rejection of numerical gates yields FAIL.
• Explicit Slurm/cache boundaries and separate FP8 preparation tree. No new CI lane or default GPU workload.

Validation

The preceding service integration passed the complete GitHub ready lane: https://github.com/InferMatrixLab/InferRSI/actions/runs/39560867282. Independent review reran all 34 service-adapter tests and closed the lifecycle coverage, protocol-framing, provenance and cleanup findings. The preceding Qwen adapter suite passed 313 CPU tests before three additional lifecycle/length coverage regressions were added. Ruff and whitespace checks passed.

The service functional adapter reuses the shared correctness runner and records public/JSON/SSE/concurrent equivalence, cancellation and disconnect with an unaffected survivor active at the exact lifecycle boundary, state/capacity release and reuse, actual EOS and length termination, malformed/over-limit requests, shutdown/restart, and a clearly labeled harness-only post-admission error injection. All required observations and actual FP8 execution/source identities are validated. Missing coverage remains INCONCLUSIVE.

FP8 intake authenticated all 74 required files against the pinned public checkpoint in a CPU allocation. The isolated FP8 environment and pinned reference kernel cache are prepared. Hardware/reference-kernel readiness and versioned pre-experiment freezes are complete. Initial real GPU attempts remain retained and exposed native compilation/lowering/accumulation issues plus a reference exclusion-prefix collision; model and service acceptance remain incomplete. CPU tests and authenticated bytes are not model or service acceptance.

Reference loading repair

The installed Transformers exclusion matcher treats a nonexistent dense .mlp.gate alias as a prefix of the actual FP8 .mlp.gate_proj. The reference now removes only aliases independently confirmed absent from the actual meta-model tree and paired with authenticated checkpoint FP8 weights/scales. It preserves the original on-disk configuration, rejects unexpected text parameters and requires the complete FP8 module-name set.

The numerical validator independently authenticates configuration/index metadata against the frozen manifest and recomputes the adaptation contract. Missing, malformed, stale or forged identities remain INCONCLUSIVE; numerical gates are unchanged. Trial errors report a fixed safe stage instead of exposing exception text. Independent local review passed 269 tests with one real-Transformers regression skipped because Torch is absent locally. Slurm CPU 1450 then passed the complete Qwen adapter suite, including the actual installed Transformers regression, with no skips; it exited 0 and released its allocation. The current head also passed the complete ready lane: https://github.com/InferMatrixLab/InferRSI/actions/runs/39565932416.

The repaired reference loaded all 400 FP8 projections in both full-prefix and cached modes and completed both paired repetitions. The frozen evaluator returned reference_self_check FAIL because all 12 case comparisons exceeded the NRMSE gate, despite complete greedy top-1 agreement. Those artifacts remain retained; loading correctness does not imply numerical agreement. The separately frozen real-model service functional scope completed PASS in both AB/BA repetitions, with independently reviewed public/JSON/SSE equivalence, cancellation/disconnect isolation, state release/reuse, EOS/length, shutdown/restart and post-admission error recovery. Sources remained unchanged and all owned processes/resources were released. This scoped result cannot clear the numerical failure; the acceptance issue records exact identities and artifacts.

Readiness

Draft; campaign verdict INCONCLUSIVE. PR readiness, campaign acceptance and parent Issue completion are separate. The requesting owner retains acceptance and closure authority.

[Dong1017]

Reviewed head c69961b. One reproduced P2 error-classification bug is noted inline.

Integration/status notes, separate from that bug:

• Currently draft and mergeable=false; reconcile against current main and rerun applicable checks. The old exact-head ready run succeeded, but cannot qualify the eventual conflict-resolved tree. The knowledge-model-policy context is currently red.
• The PR body still calls #57 active acceptance, and evidence/qwen38-h20-fp8-text-phase1/plan/README.md:3–10 directs new freezes there. #57 is closed as superseded by #67. Label this as historical #57 workflow/evidence and route current acceptance to #67 without rewriting old gates or verdicts. This PR does not establish #67 performance/task-quality acceptance.

Validation: source inspection and one targeted CPU reproduction using the existing trace_bundle fixture. No complete local pytest run, GPU, model or service experiment.

[Dong1017]

[P2] Preserve INCONCLUSIVE for unreadable NumPy artifacts

An empty candidate NPZ with a matching recorded SHA reaches np.load at line395 and raises EOFError, which this exception handler omits. Reproduced with trace_bundle: CLI exits1, writes no validation JSON and prints a traceback. The shared correctness evaluator treats exit1 as FAIL, turning malformed evidence into a numerical rejection. Normalize archive read errors to ValueError or catch EOFError here, and assert exit2 plus INCONCLUSIVE JSON in a subprocess regression.

[amy-why-3459]

Review at head c69961bd (draft): not approving yet.

1. Merge conflict: the PR is not mergeable with main (dirty); it was opened against a much older main. Please rebase and re-run the applicable checks.
2. Open P2 from 2026-10-23 (examples/qwen38_text/validate.py, NumPy load around L395): an empty candidate NPZ with a matching recorded SHA raises EOFError, which the exception handler omits, so the CLI exits 1 with a traceback and writes no validation JSON instead of INCONCLUSIVE. No commit has addressed it. Expected: include EOFError (and other unreadable-archive errors) in the INCONCLUSIVE path and add a regression test.
3. Question: is this PR still active? If the FP8 workflows have been superseded, please say so.

[amy-why-3459]

Coverage review at head c69961bd9814beb93608f4f001a2ab3eae5cf26f (base 157f4a00563e88a606c86f7d3b5f7c0855048bee; draft). Re-checked against my earlier short-SHA note and Dong1017's review at this same head. No new commits. Not approving yet.

1. Merge conflict: mergeable_state=dirty against current main; the PR was opened against a much older main. Please rebase and re-run applicable checks on the conflict-resolved tree.
2. Open P2 from 2026-10-23 (examples/qwen38_text/validate.py, NumPy load ~L395; exception handler ~L456): an empty candidate NPZ with a matching recorded SHA raises EOFError, which the except (OSError, ValueError, KeyError, TypeError, ImportError, zipfile.BadZipFile) path omits, so the CLI exits 1 with a traceback and writes no validation JSON instead of INCONCLUSIVE. Still unaddressed at this head. Expected: include EOFError (and other unreadable-archive errors) in the INCONCLUSIVE path and add a regression test.
3. Acceptance routing: PR body still points at acceptance: Qwen3.8-27B-FP8 single-H20 text engine and HTTP service — phase 1 #57 as active acceptance; acceptance: Qwen3.8-27B-FP8 single-H20 text engine and HTTP service — phase 1 #57 is closed as superseded by acceptance: Qwen3.8 FP8 vs vLLM 0.32.0 — ≥25% TTFT or TPOT gain, ≤5% regression #67. Please label this as historical acceptance: Qwen3.8-27B-FP8 single-H20 text engine and HTTP service — phase 1 #57 workflow/evidence and route current acceptance to acceptance: Qwen3.8 FP8 vs vLLM 0.32.0 — ≥25% TTFT or TPOT gain, ≤5% regression #67 without rewriting old gates/verdicts.
4. Activity question: is this PR still active? If the FP8 workflows have been superseded, please say so.

Reviewed commit range: 157f4a00563e88a606c86f7d3b5f7c0855048bee … c69961bd9814beb93608f4f001a2ab3eae5cf26f.

[Dong1017]

@jiangkuaixue123 I prepared and tested a small fix for the unreadable-NPZ finding against current head c69961bd9814beb93608f4f001a2ab3eae5cf26f. Please apply the patch below to this PR and rerun the applicable checks.

The production change adds EOFError to the existing INCONCLUSIVE handler. The regression tests cover empty and truncated ZIP artifacts on both baseline/candidate sides with matching recorded hashes, exit code 2, JSON output, no traceback/path disclosure, and INCONCLUSIVE propagation through the shared runner. Existing complete numerical-failure tests remain unchanged.

Verified locally:

• Before the fix, the two new empty-artifact cases failed with exit code 1 and EOFError.
• python -m pytest -o addopts='' tests/test_qwen38_*.py -q -rs: 341 passed, 1 skipped. The skipped installed-Transformers regression requires Torch, which was absent. Python 3.12.14, NumPy 2.3.5, pytest 9.2.0.
• ruff check examples/qwen38_text/validate.py tests/test_qwen38_validation.py: passed (Ruff 0.17.4).
• Patch application and comparison with the tested files passed. This was CPU-only adapter testing, not full repository ready CI or model/GPU/service acceptance.

Publication status: the GitHub integration returned HTTP 403 when writing to the fork. No commit or branch update was made; the PR head is unchanged. The merge conflict and historical #57/current #67 acceptance routing remain separate follow-ups. No historical evidence, numerical gates, or campaign verdicts are changed by this patch.

Complete tested two-file patch

--- a/examples/qwen38_text/validate.py
+++ b/examples/qwen38_text/validate.py
@@ -458,7 +458,8 @@
                           args.max_nrmse, args.min_top1, args.rms_floor, args.qualification_scope,
                           args.model_id, args.fp8_kernel_revision, args.fp8_kernel_sha256,
                           args.model_path)
-    except (OSError, ValueError, KeyError, TypeError, ImportError, zipfile.BadZipFile) as exc:
+    except (OSError, EOFError, ValueError, KeyError, TypeError, ImportError,
+            zipfile.BadZipFile) as exc:
         # Do not echo exception strings, which may contain private runtime paths.
         result = {"verdict": "INCONCLUSIVE", "qualification_scope": args.qualification_scope,
                   "reasons": [f"Invalid or missing validation evidence ({type(exc).__name__})"]}
--- a/tests/test_qwen38_validation.py
+++ b/tests/test_qwen38_validation.py
@@ -95,6 +95,34 @@
         check(trace_bundle)
 
 
+@pytest.mark.parametrize("side", ["baseline", "candidate"])
+@pytest.mark.parametrize("payload", [b"", b"PK\x03\x04invalid"], ids=["empty", "truncated-zip"])
+def test_unreadable_array_writes_inconclusive(trace_bundle, side, payload):
+    run, workload, manifest = trace_bundle
+    root = run / side / "trial-001"
+    record = json.loads((root / "result.json").read_text())
+    case = record["cases"][0]
+    artifact = root / case["logits_file"]
+    artifact.write_bytes(payload)
+    case["logits_sha256"] = digest(artifact)
+    (root / "result.json").write_text(json.dumps(record))
+    output = run / "standalone-validation.json"
+    validator = Path(__file__).resolve().parents[1] / "examples/qwen38_text/validate.py"
+    completed = subprocess.run(
+        [sys.executable, str(validator), "--run-dir", str(run), "--workload", str(workload),
+         "--manifest", str(manifest), "--max-nrmse", "0.01", "--min-top1", "0.99",
+         "--rms-floor", "1e-8", "--output", str(output)],
+        capture_output=True, text=True, check=False,
+    )
+    assert completed.returncode == 2
+    result = json.loads(output.read_text())
+    assert result["verdict"] == "INCONCLUSIVE"
+    assert result["qualification_scope"] == "reference_self_check"
+    assert json.loads(completed.stdout) == result
+    assert not completed.stderr
+    assert str(run.parent) not in completed.stdout
+
+
 def test_complete_numerical_regression_is_fail(trace_bundle):
     root = trace_bundle[0] / "candidate/trial-001"
     def alter(result):
@@ -129,11 +157,20 @@
     assert any("reset/replay" in reason for reason in result["reasons"])
 
 
-def test_synthetic_traces_complete_real_subprocess_loop(trace_bundle, tmp_path):
+@pytest.mark.parametrize("empty_archive", [False, True])
+def test_synthetic_traces_complete_real_subprocess_loop(trace_bundle, tmp_path, empty_archive):
     from inferrsi.harness.runner import run_experiment
     from inferrsi.runspec import RunSpec
 
     fixture, workload, manifest = trace_bundle
+    if empty_archive:
+        root = fixture / "candidate/trial-001"
+        def empty(result):
+            case = result["cases"][0]
+            artifact = root / case["logits_file"]
+            artifact.write_bytes(b"")
+            case["logits_sha256"] = digest(artifact)
+        mutate_case(trace_bundle, empty)
     repo = tmp_path / "source"
     repo.mkdir()
     emit = (
@@ -161,11 +198,12 @@
         snapshot=lambda *_: {"available": True, "snapshot_hash": "synthetic-source"},
     )
     evaluation = json.loads((result_dir / "evaluation.json").read_text())
-    assert evaluation["verdict"] == "PASS"
+    assert evaluation["verdict"] == ("INCONCLUSIVE" if empty_archive else "PASS")
     assert evaluation["improvement_pct"] is None
     assert (result_dir / "report.html").exists()
     quality = json.loads((result_dir / "validation/result.json").read_text())
     assert quality["qualification_scope"] == "reference_self_check"
+    assert quality["verdict"] == ("INCONCLUSIVE" if empty_archive else "PASS")
 
 
 @pytest.fixture