feat(skilleval): bounded cloud generation and isolated artifact replay

.github/workflows/skill-eval-cloud-replay.yml

@@ -0,0 +1,55 @@
+name: SkillEval isolated cloud replay (manual)
+
+on:
+  workflow_dispatch:
+    inputs:
+      evidence_commit:
+        description: Exact repository commit containing the fixed-path two-parser bundle
+        required: true
+        type: string
+      bundle_sha256:
+        description: SHA256 of exact bundle bytes
+        required: true
+        type: string
+
+permissions: {}
+
+concurrency:
+  group: skill-eval-cloud-replay
+  cancel-in-progress: false
+
+jobs:
+  replay:
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-22.04
+    timeout-minutes: 10
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+        with:
+          ref: ${{ github.sha }}
+          persist-credentials: false
+      - name: Install distribution bubblewrap, without changing security policy
+        run: |
+          sudo apt-get update
+          sudo apt-get install --yes --no-install-recommends bubblewrap
+      - name: Qualify then replay the pinned data-only bundle
+        shell: bash
+        env:
+          EVIDENCE_COMMIT: ${{ inputs.evidence_commit }}
+          BUNDLE_SHA256: ${{ inputs.bundle_sha256 }}
+        run: |
+          env -i PATH=/usr/bin:/bin LANG=C.UTF-8 PYTHONPATH=src \
+            /usr/bin/python3 -m tools.skill_eval_cloud_replay \
+            --output "$RUNNER_TEMP/cloud-replay" \
+            --evidence-commit "$EVIDENCE_COMMIT" --bundle-sha256 "$BUNDLE_SHA256" \
+            --revision "$GITHUB_SHA" --runner-image "${ImageOS:-unknown}/${ImageVersion:-unknown}"
+      - name: Retain replay receipt, including unavailable isolation
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: cloud-replay-${{ github.run_id }}-${{ github.run_attempt }}
+          path: ${{ runner.temp }}/cloud-replay/replay.json
+          if-no-files-found: error
+          retention-days: 14

docs/operations.md

@@ -29,7 +29,9 @@ Effect classes used below:
 | Adapt existing Flare delivery evidence | [Offline qualification adapter](flare-qualification.md), `python -m inferrsi.model.validators.flare` | Read-only | Existing raw packets and a separate receipt; one HTTP delivery metric, no model/server/device execution |
 | Evaluate selected skill revisions | [Bounded skill-evaluation driver](skill-evaluation.md) | Experiment-executing | Explicitly authorized real Codex calls, independent task verdicts and private evidence; separate from numeric engine-performance scoring |
 | Compare skill additions or updates | [Configured skill comparison](skill-eval-comparison.md), `python -m tools.skill_eval_compare` | Read-only preparation / experiment-executing / report-generating | Fixed Git snapshots and one shared exam; explicit plan authorization for execution; JSON + Markdown regeneration makes no model calls |
+| Export a cloud-generated parser pair | [Generation-only protocol](skill-eval-generation-only.md), `python -m tools.skill_eval_cloud_export` | Local evidence-writing; no execution or publication | Validates frozen inputs and retained sources; writes one bounded inert JSON bundle, never executes parser source |
 | Qualify an artifact verifier host | [Manual reviewer qualification](skill-eval-reviewer-qualification.md), `python -m tools.skill_eval_qualify` | Trusted-control execution / report-generating | Manual main-only workflow; fixed software controls, no model or external artifact input; unavailable isolation fails rather than skips |
+| Replay a cloud-generated parser pair | [Isolated cloud replay](skill-eval-cloud-replay.md), `python -m tools.skill_eval_cloud_replay` | Untrusted-artifact execution / report-generating | Manual pinned data ingress; same-host controls before bubblewrap-only replay; no models or credentials |
 | Validate one harness-improvement step | [Offline improvement](harness-improvement.md), Python `tools.skill_eval_improve.prepare` / `run_offline` | Snapshot-writing / offline experiment-executing | Supplied declarative INV/DOM edits, shared total limits, rerun and one holdout gate; trusted fixed-output clients only; no real-model or Store execution |
 | Plan and report skill PR evaluations | [Skill PR integration](skill-eval-pr.md), `python -m tools.skill_eval_pr` | Read-only planning / authorized experiment-executing / PR-publishing | Trusted mapping, exact operator approval, consumed execution identity and credential-separated publication from retained evidence; no automatic paid trials |
 | Recover goal progress and the next proposal | [Offline goal checkpoints](goal-checkpoints.md), `python -m tools.goal_checkpoint` | Read-only | Explicit scope/criteria, pinned owner decisions and existing comparison receipts; a disposable view, never execution or acceptance authority |

docs/skill-eval-cloud-replay.md

@@ -0,0 +1,54 @@
+# Isolated replay of a cloud-generated parser pair
+
+The manual `skill-eval-cloud-replay.yml` workflow removes any requirement that
+an author host the verifier. It performs no model calls and receives no model
+credentials. Generation is separate; neither remote replay nor green CI proves
+producer isolation, causal comparability, generalization, or campaign acceptance.
+
+## Data contract and transport
+
+An operator first reviews and publishes the **data**, not executable controller
+code, at `artifacts/skill-eval-cloud/bundle.json` in this repository. Publication
+must be authorized and checked for private data. The dispatch takes only the
+exact 40-character evidence commit and SHA256 of the complete bundle bytes.
+The trusted controller is checked out from the dispatch's exact main revision,
+not the evidence commit. Its unauthenticated HTTPS read uses one fixed repository
+and path, rejects redirects, and reads at most 1 MiB plus one rejection byte.
+There are no archive extraction, arbitrary URLs, user paths, custom graders,
+configuration commands, or imports of uploaded Python.
+
+The UTF-8 JSON object has exactly these keys (duplicate keys are rejected):
+
+- `schema_version`: `engineering-cloud-replay-v1`.
+- `task_id`: 1–128 ASCII letters, numbers, underscores, dots or hyphens; an opaque
+  provenance identifier, not a path or executable task selector.
+- `request_sha256`: SHA256 of the trusted `engineering.REQUEST` UTF-8 bytes.
+- `grader_sha256`: SHA256 of the trusted `skill_eval_engineering.py` file bytes.
+- `inputs_sha256`, `protocol_sha256`: lowercase 64-character provenance digests.
+  These are retained, **not independently authenticated by this receipt**.
+- `artifacts`: exactly two objects, ordered baseline then candidate, each with
+  exactly `arm`, `source`, and `sha256`. Each source is nonempty UTF-8, at most
+  65,536 bytes, and must match its digest. Both arms are validated before either
+  is executed. JSON escaping does not change the source-byte hash.
+
+## Execution and interpretation
+
+The controller first requalifies the reviewed good and bad controls on this
+same runner. If qualification is not PASS, neither submitted artifact runs.
+Each parser then executes only through the existing bubblewrap verifier under
+its filesystem, network and resource bounds. Missing isolation never falls back
+to host execution. The shell clears inherited environment variables and never
+passes its repository token to parser execution. No host security policy is
+weakened. Model source and outputs are not echoed into workflow logs.
+
+`replay.json` retains the qualification, controller/evidence identities, bundle
+and source hashes, and separate per-arm checks. Its top-level PASS means **both
+replays were delivered**, not that either parser passed or that a skill helped;
+negative per-arm results are valid evidence. Invalid ingress, infrastructure
+failure or lost isolation yields INCONCLUSIVE and a nonzero job. Original
+producer/capture failures remain unchanged; reconcile the receipt against the
+frozen experiment protocol and original evidence before any comparison claim.
+
+The workflow is main-only and manual, independent of normal PR/merge CI. It does
+not comment, accept, close or merge anything. A reviewer can independently
+regenerate the bounded replay from the retained commit and bundle digest.

docs/skill-eval-generation-only.md

@@ -0,0 +1,63 @@
+# Engineering parser generation-only mode
+
+This opt-in #253 transport lets a provider produce parser source without running
+model-produced code on the controller. It is **not** an iterative workspace-agent
+experiment and does not replace the Linux/bubblewrap verifier.
+
+In a trusted comparison configuration set both:
+
+```json
+{
+  "adapter": "engineering-artifact-v1",
+  "execution_mode": "provider-generation-only-v1"
+}
+```
+
+The adapter name must match the current engineering adapter constant. The client
+must be the existing `anthropic-messages-full-context-v1` client, with reviewed
+model, pricing and spend limits. Both arms use the same configuration. Preparation
+freezes task, starter, skill snapshots, schema and execution mode before calls.
+No default PR mapping or provider configuration is changed by this feature.
+
+The provider receives the frozen task, starter and its assigned skill only. Its
+sole tool returns `artifact`, `summary`, and `artifact_source`; it cannot execute
+commands or modify files. A trusted writer accepts exactly one `parser.py` source
+string of 1–65,536 UTF-8 bytes and writes those bytes only after checking that the
+starter is unchanged. Path alternatives, extra fields and malformed or oversized
+payloads fail closed. Source is retained as data; it is not imported or executed
+by the writer. Summary claims are not grading evidence.
+
+Execution always requires the existing request authorization broker, even if a
+caller omits the general authorization flag. A stub is usable only in offline
+unit tests. This does not authorize paid calls: current provider pricing,
+per-request reservations, PR lifetime/day accounting and the precise experiment
+identity must be independently authorized and frozen. A new accounting PR must
+not be represented as the historical skill revisions being compared.
+
+Generation never invokes the artifact verifier, even on a capable host. The
+original trial remains `INCONCLUSIVE` with `artifact_verification_deferred`. Its
+capture report is also non-executing. A separately qualified Linux
+reviewer may replay the retained artifact after frozen-input/source integrity
+checks. The report preserves that original result and separately exposes
+`artifact_verdict`, the 19 artifact checks and `replayed_artifact_outcome`.
+Tampered inputs/source, malformed output and other execution errors are not
+eligible for this deferred replay. Existing workspace-agent execution is
+unchanged. A passing artifact comparison is not evidence of author-agent
+isolation, general skill efficacy or causal improvement from a single pair.
+
+Offline regression coverage: `tests/test_skill_eval_generation.py`. Real sandbox
+qualification, artifact transport and paid trial execution are separate steps;
+these unit tests do not establish that a particular host is qualified.
+
+After the authorized two-call capture, export only retained source and provenance:
+
+```sh
+python -m tools.skill_eval_cloud_export --run /path/to/frozen-run \
+  --protocol /path/to/prospective-protocol.json --output /path/to/bundle.json
+```
+
+The exporter refuses incomplete usage, modified inputs, mismatched receipts or
+source, additional pairs and a changed public request. It does not upload the
+bundle or establish the timing of protocol approval; the execution admission
+controller must bind that protocol hash before calls. Treat both sources as
+untrusted until independent qualified replay.

src/inferrsi/harness/skill_eval_engineering.py

@@ -18,6 +18,7 @@
 from pathlib import Path
 
 ADAPTER = "engineering-artifact-v1"
+GENERATION_MODE = "provider-generation-only-v1"
 MAX_SOURCE_BYTES = 65536
 STARTER = (
     'def evaluate(status, frames, requested_tokens):\n'
@@ -38,6 +39,47 @@
     "commits or publication. Return artifact='parser.py' and a concise summary, not a verdict. "
     "The controller retains the actual file and independently checks its behavior."
 )
+GENERATION_SCHEMA = {
+    **SCHEMA,
+    "required": ["artifact", "summary", "artifact_source"],
+    "properties": {**SCHEMA["properties"], "artifact_source": {"type": "string"}},
+}
+GENERATION_PROMPT = (
+    "Read the supplied task.json, parser.py starter and assigned knowledge. Generate the "
+    "complete repaired parser.py source as artifact_source in submit_answer. "
+    "Implement evaluate(status, frames, requested_tokens) under the public task contract. "
+    "You have no shell, execution, file-writing, network or test tool. Do not claim to "
+    "have run tests. Return artifact='parser.py', a concise summary and artifact_source; "
+    "do not return a verdict. A trusted controller writes only parser.py from the source "
+    "string and an independent isolated verifier checks its behavior later."
+)
+
+
+def materialize_generated_artifact(workspace: Path, answer: dict) -> dict:
+    """Convert a bounded provider response to one artifact; never execute its source.
+
+    The provider is a structured-output service, not a process with workspace access.
+    Fresh workspace ownership and the unchanged starter are required before writing.
+    """
+    if (not isinstance(answer, dict)
+            or set(answer) != {"artifact", "summary", "artifact_source"}
+            or answer["artifact"] != "parser.py"
+            or not isinstance(answer["summary"], str)
+            or not isinstance(answer["artifact_source"], str)):
+        raise ValueError("Invalid generated artifact answer")
+    try:
+        source = answer["artifact_source"].encode("utf-8")
+    except UnicodeError:
+        raise ValueError("Generated artifact must be UTF-8") from None
+    if not 0 < len(source) <= MAX_SOURCE_BYTES:
+        raise ValueError("Generated artifact exceeds source bounds")
+    artifact = workspace / "parser.py"
+    if read_regular(artifact, MAX_SOURCE_BYTES) != STARTER.encode("utf-8"):
+        raise ValueError("Generated artifact requires the unchanged frozen starter")
+    artifact.write_bytes(source)
+    return {"artifact": "parser.py", "summary": answer["summary"]}
+
+
 REQUEST = (
     "Repair parser.py: evaluate(status, frames, requested_tokens) returns exactly "
     "{success: bool, reason: str}. Inputs are an integer HTTP status, decoded frames, "

src/inferrsi/harness/skill_evaluation.py

@@ -189,7 +189,10 @@ def run_trial(
     *,
     timeout_seconds: float | None = None,
     engineering_artifact: bool = False,
+    generation_only: bool = False,
 ) -> dict:
+    if generation_only and not engineering_artifact:
+        raise ValueError("Generation-only requires the engineering artifact adapter")
     trial_dir.mkdir(parents=True)
     final = trial_dir / "answer.json"
     before = engineering.input_hashes(workspace) if engineering_artifact else {
@@ -233,11 +236,18 @@ def run_trial(
             evaluation_error = "missing_output"
         else:
             try:
-                answer_bytes = (engineering.read_regular(final, 65536) if engineering_artifact
+                answer_limit = 1048576 if generation_only else 65536
+                answer_bytes = (engineering.read_regular(final, answer_limit)
+                                if engineering_artifact
                                 else final.read_bytes())
                 output = json.loads(answer_bytes.decode("utf-8"))
             except (ValueError, UnicodeError):
                 evaluation_error = "invalid_output_json"
+    if evaluation_error is None and generation_only:
+        try:
+            output = engineering.materialize_generated_artifact(workspace, output)
+        except (ValueError, OSError):
+            evaluation_error = "invalid_generated_artifact"
     if evaluation_error is None and engineering_artifact:
         artifact = workspace / "parser.py"
         if not isinstance(output, dict) or set(output) != {"artifact", "summary"}:
@@ -263,14 +273,19 @@ def run_trial(
                 verification_dir.mkdir()
                 (verification_dir / "stdout.log").write_text("", encoding="utf-8")
                 (verification_dir / "stderr.log").write_text("", encoding="utf-8")
-                try:
-                    checks = engineering.verify(task, output)
-                except engineering.SandboxUnavailable:
-                    evaluation_error = "artifact_sandbox_unavailable"
+                if generation_only:
+                    # The generation process may carry provider credentials. Even
+                    # a capable host must not execute generated source in this phase.
+                    evaluation_error = "artifact_verification_deferred"
                 else:
-                    atomic_write_json(verification_dir / "result.json", checks)
-                    verification = {"status": "completed" if checks["passed"] else "failed",
-                                    "returncode": 0 if checks["passed"] else 1}
+                    try:
+                        checks = engineering.verify(task, output)
+                    except engineering.SandboxUnavailable:
+                        evaluation_error = "artifact_sandbox_unavailable"
+                    else:
+                        atomic_write_json(verification_dir / "result.json", checks)
+                        verification = {"status": "completed" if checks["passed"] else "failed",
+                                        "returncode": 0 if checks["passed"] else 1}
     if evaluation_error is None and not engineering_artifact:
         verifier_environment = getattr(client, "verifier_environment", None)
         options = {"environment": verifier_environment()} if verifier_environment else {}
@@ -333,5 +348,7 @@ def run_trial(
         record["artifact_sha256"] = (sha256(retained) if retained.is_file()
                                      and not retained.is_symlink() else None)
         record["artifact_kind"] = engineering.ADAPTER
+        if generation_only:
+            record["execution_mode"] = engineering.GENERATION_MODE
     atomic_write_json(trial_dir / "result.json", record)
     return record