Skip to content

Run semantics: replayable event reducer and fail-closed required-gate verdicts #43

Open

Description

@PhysShell

Classification

  • Change type: correctness fix + execution infrastructure improvement
  • Priority: P0
  • Expected value: very high — prevents false-green runs and makes run records independently verifiable
  • Estimated effort: medium
  • Implementation risk: medium-high — this reducer defines execution truth
  • Recommendation: YES, mandatory improvement before loops/retries depend on current verdict semantics
  • Related: Run evidence v2: action-level records linked to findings, costs and verifier outcomes #42 (action-level run evidence; deliberately separate scope)

Problem

o7 run currently stores a useful forensic archive, but the archive is not a replayable execution model. RunMeta contains an already-reduced verdict and per-step records; there is no canonical event sequence from which the state and verdict can be recomputed independently.

The current reducer also has a correctness hole:

ERROR wins
else required FAIL wins
else PASS

WARN, BLOCKED, and NOT_APPLICABLE are ignored. Today a Windows-only gate may be skipped and represented outside the effective required set, allowing a run to become PASS although a required environment-specific check never executed. A green verdict must mean every required obligation was actually discharged, not merely that no executed command returned a bad exit code.

Goal

Introduce a pure, versioned run-state reducer driven by an append-only event stream, plus a command that replays a stored run and independently verifies its verdict and artifact integrity.

Production execution remains effectful; verdict semantics become pure and replayable.

Proposed event contract

Names are reviewable, but the semantic distinctions must survive:

RunStarted
WorktreeCreated
AgentStarted
AgentExited
PatchCaptured
PolicyChecked
GateStarted
GateFinished
SandboxEvidenceCaptured
RunSealed

Each event should carry:

event_id
schema_version
run_id
sequence
previous_event_digest
event_digest
timestamp
payload
artifact references + digests where applicable

The event chain must detect post-run modification, truncation, reordering, and substitution of referenced task/diff/log artifacts.

Reducer requirements

The reducer should be a pure function equivalent to:

fn reduce(state: RunState, event: RunEvent)
    -> Result<RunState, InvalidTransition>;

Required semantics:

  1. A required gate that did not execute is not PASS.
  2. required + NOT_APPLICABLE reduces to BLOCKED unless the task contract explicitly declared that gate optional for this environment before execution.
  3. required + unsupported environment reduces to BLOCKED.
  4. required + missing required sandbox evidence reduces to BLOCKED once a policy requires confinement evidence.
  5. ERROR remains distinguishable from domain FAIL and policy/environment BLOCKED.
  6. Duplicate, out-of-order, impossible, or post-seal events fail loudly.
  7. Replaying the same event stream produces byte-stable normalized state and the same verdict.

CLI surface

A narrow first surface is sufficient:

o7 replay runs/<target>/<run-id>

It should:

  • validate event-chain continuity and digests;
  • validate referenced artifact digests;
  • recompute changed-file and gate summaries where derivable;
  • recompute the overall verdict;
  • compare the recomputed result with stored summaries;
  • emit a machine-readable replay report;
  • exit non-zero on any unexplained mismatch.

Suggested artifact:

runs/<target>/<run-id>/
  events.jsonl
  replay.json

Existing files (meta.json, diff.patch, gate/*.log) remain usable and are referenced by digest rather than duplicated into events.

Required controls

  1. All required gates execute and pass → PASS.
  2. Required Windows gate on a Linux-only runner → BLOCKED, never PASS.
  3. Optional environment-specific gate is skipped by an explicit pre-run contract → allowed and visible.
  4. Required gate returns non-zero → FAIL.
  5. Gate process cannot start or event sequence is malformed → ERROR.
  6. Required sandbox report is absent or digest-invalid → BLOCKED/ERROR according to the contract, never silent success.
  7. diff.patch changes after sealing → replay fails.
  8. A gate log changes after sealing → replay fails.
  9. Event deletion/reordering/insertion → chain verification fails.
  10. Legacy run without events remains readable with an explicit legacy/non-replayable limitation; it is not silently presented as replay-verified.

Acceptance

  • Versioned RunEvent and RunState contracts are documented and implemented.
  • Verdict reduction is pure and covered by exhaustive transition/table tests.
  • Required-but-unexecuted gates cannot produce PASS.
  • o7 replay independently reproduces the stored verdict and validates artifact digests.
  • Event-chain tampering and impossible transitions fail loudly.
  • New runs emit replayable events without removing existing forensic artifacts.
  • Legacy records remain readable with explicit limitations.
  • Run evidence v2: action-level records linked to findings, costs and verifier outcomes #42 can reference replay-verified action/verifier evidence without duplicating the event reducer.

Non-goals

  • No retry/repair loop in this issue.
  • No scheduler, queue, CFR, bandit, or model-selection policy.
  • No new sandbox implementation; only consume/validate its evidence contract.
  • No OwnAudit domain triage inside 007.
  • No inference of missing events from unstructured agent stdout.
  • No claim that a tamper-evident local chain is remote attestation.

Activity

  1. PhysShell commented on Nov 20, 2026

    @PhysShell
    OwnerAuthor

    Billy B-Assistant: revision fencing and rescue-baseline consistency (20.11.2026)

    Research page · update state machine · recovery.

    Comparison donor, no new runtime: Billy separates immutable staged release environments from a stable bootstrap supervisor/rescue baseline. selection.json is authoritative; current-source symlink is only for navigation. It uses atomic state writes, SHA-256 lock tracking, distinct config-validation vs worker-health errors, probation and rollback. This suggests negative fixtures for the existing replayable event reducer: old generation worker or callback must not overwrite a new selected generation; interrupted transition must yield one recoverable selected version; fallback must not mask incompatible state or treat runtime failure as bad configuration.

    Keep this separate from Golem's per-effect oplog replay and do not infer that a lock digest proves release provenance/signature. Under BILLY-02 / LAST-KNOWN-GOOD-UPDATE / KILL-FIRST, compare with existing atomic snapshot + immutable evidence approach only on synthetic local state; inject interruption before/after selection commit, crashed health check, concurrent supervisor and stale completed job. GO only on independently confirmed missing invariant; otherwise STOP. No product updater rebuild or mandatory CI gate. Upstream is noncommercial licensed.

  2. PhysShell commented on Nov 20, 2026

    @PhysShell
    OwnerAuthor

    Restate donor: RESTATE-02 keyed Virtual Object concurrency and identity (20.11.2026)

    Research page · Go context contracts · Restate v1.8.1 notes.

    RESTATE-02 / KEYED-ENTITY-CONCURRENCY / KILL-FIRST as a small conformance fixture for replayable event reducer / fail-closed verdicts. Restate Virtual Object gives serialized mutation within a keyed Restate-owned state context, but not an automatic transaction spanning external PostgreSQL nor owner checks. Compare the same 2–3 overlapping mutations to SQL unique/optimistic concurrency/SELECT FOR UPDATE baseline. Controls: same key retries, different keys aliasing same SQL row, concurrent completion, external SQL commit fails after Restate state progresses, old worker completion after generation switch, restart and read-after-recovery.

    Restate v1.8.1 adds canonical invocation identifiers containing log-creation position to distinguish invocation incarnations. RESTATE-03 / INVOCATION-IDENTITY / DEFER: test logical run ID reuse vs specific execution identity in existing exact-SHA receipts, only if collision/reused generation is demonstrated. Require independent observed state and first divergence, not only journal evidence.

    GO only if concurrency invariant is materially simpler and provable without dual sources of truth; STOP if existing reducer and SQL constraints already work. No new identity infrastructure or mandatory CI.

  3. PhysShell commented on Nov 20, 2026

    @PhysShell
    OwnerAuthor

    TESTed donor: TESTED-02 / conflict-aware execution-unit planner (20.11.2026)

    Research page · planning implementation.

    TESTed's execution planner groups compatible test contexts and splits on conflicting stdin, strict workdir requirements, input-file contents, output-file paths and explicit exit checks. TESTED-02 / CONFLICT-AWARE-EXECUTION-PLANNER / KILL-FIRST: compare A) fully isolated test contexts, B) naive shared process, C) conflicts-aware groups over 10–15 repeatable tests with seeded order. Extend TESTed's file-centric conflict model with shared DB rows, tenant state, ports, process globals, effects and environment variables for 007/ScenarioBox. Keep an independent state/effect ledger, exact fixture SHA, run attempt ids and verdicts; specifically verify order independence, no leakage through prior case, and no false reduction in coverage. Measure runtime savings vs isolation cost.

    GO only if faster while preserving invariant outcomes, reproducibility and error attribution; STOP if isolated execution is already cheap enough. Does NOT alter current replay reducer semantics or justify building a generic multi-language sandbox. The judge's workdir/timeout is not a security boundary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions