Skip to content

Run evidence v2: action-level records linked to findings, costs and verifier outcomes #42

Open

Description

@PhysShell

Classification

  • Change type: enhancement / run evidence contract
  • Priority: P1
  • Expected value: high — provides the action/outcome data needed by OwnAudit triage and later scheduling research
  • Implementation risk: medium-high — incomplete records create misleading training and audit data
  • Repository owner: 007 execution/orchestration layer
  • Consumes identity/event contracts from: PhysShell/Own.NET#266 and PhysShell/Own.NET#268
  • Production status: recommended improvement after the identity contract is stable

Problem

The current 007 run record correctly captures run-level metadata, task text, agent stdout, diff and per-step gate verdicts. That answers:

Did this isolated run pass its gates?

It does not yet answer:

What action was attempted?
Which finding(s) motivated it?
What did it cost?
Which verifier established the outcome?
Was the result confirmed, rejected, inconclusive or a regression?

Without action-level records, future triage/scheduling work would have to infer intent and outcome from prose/stdout. That is slower than a sedated sloth and less trustworthy than it sounds.

Goal

Add a versioned, append-friendly ActionRecord contract under each run record.

007 remains the executor and evidence producer. OwnAudit remains the consumer that interprets evidence and ranks/triages findings.

Proposed record shape

Equivalent fields:

struct ActionRecord {
    schema: u32,
    action_id: String,
    run_id: String,
    parent_action_id: Option<String>,

    target_repo: String,
    base_commit: String,
    target_pattern_ids: Vec<String>,
    target_occurrence_ids: Vec<String>,
    target_lineage_ids: Vec<String>,

    policy_id: String,
    action_kind: ActionKind,
    input_digest: String,

    started_at: String,
    finished_at: String,
    cost: ActionCost,

    outcome: ActionOutcome,
    verifier_results: Vec<VerifierResult>,
    artifact_refs: Vec<ArtifactRef>,
}

Exact Rust types are reviewable. The information must remain explicit.

Action kinds

Initial vocabulary:

HumanReviewRequested
JudgeRequested
DeepAnalyzerRun
RuntimeScenarioRun
TestGenerationRequested
FixProposed
FixApplied
FixRejected
FindingSuppressionProposed
FindingSuppressionApplied
NoAction

NoAction is important for unbiased evaluation: actions considered but skipped by policy must not disappear from the dataset.

Outcomes

Initial vocabulary:

Confirmed
Rejected
Fixed
NoChange
IntroducedRegression
TimedOut
Cancelled
PolicyDenied
OperationalError
Inconclusive

A successful process exit is not automatically Confirmed or Fixed. Outcome requires verifier evidence.

Verifier result

Each verifier result should record:

verifier_kind
verifier_name/version
command or policy step identity
input/artifact digest
verdict
exit code
started/finished timestamps
log/artifact refs

Examples:

  • re-audit reports target occurrence absent with comparable config;
  • regression tests pass/fail;
  • runtime scenario confirms retention;
  • diff policy rejects unrelated changes;
  • human disposition is pending/recorded externally.

007 must record what the verifier said, not reinterpret domain truth beyond its execution contract.

Cost record

Use measured fields where available:

wall_time_ms
agent_cost_usd
input_tokens
output_tokens
cpu_time_ms
peak_rss_bytes
human_minutes   # only when supplied, never invented

Unknown values stay null/unknown. Do not convert missing cost into zero.

On-disk layout

Suggested additive layout:

runs/<target>/<run-id>/
  meta.json
  actions/
    <action-id>/
      action.json
      verifier/
        <name>.json
        <name>.log
      artifacts/
        ...

meta.json may carry only an action summary/index. The canonical detailed records live under actions/ to avoid turning one JSON file into a junk drawer.

Identity and provenance

  • action_id is stable within the stored record and collision-resistant.
  • Every action pins run_id, base commit, policy/config identity and input digest.
  • Finding links use the final pattern/occurrence/lineage contract; legacy runs may omit them explicitly.
  • Artifact references include digests and paths relative to the run record.
  • Unknown schema or digest mismatch fails loudly during read/export.
  • Sensitive agent output/secrets remain subject to the existing redaction/output-filter policy; action records must not become a new secret sink.

Export seam

Add a deterministic export suitable for OwnAudit ingestion, for example:

o7 evidence export --run <id> --format json

The export should contain normalized action/outcome/provenance data, not raw secrets or unbounded stdout.

It must be possible to export all eligible runs incrementally without duplicating already-ingested actions.

Backward compatibility

  • Existing run records remain readable.
  • Missing actions/ means legacy run-level evidence, not a parse failure.
  • RunMeta.schema is bumped only if required by a breaking change; additive action references should prefer optional/versioned fields.
  • Existing o7 run PASS/FAIL exit behavior is unchanged.

Required controls

  1. One run with two distinct actions records both independently.
  2. Two actions target the same pattern but different occurrences.
  3. Fix process exits zero but re-audit still finds the issue → NoChange, not Fixed.
  4. Tests pass but unrelated diff violates policy → verifier conflict remains visible.
  5. Action times out → timeout outcome plus partial verifier artifacts retained.
  6. Missing cost fields remain unknown.
  7. Duplicate export/ingest uses stable IDs and does not duplicate actions.
  8. Artifact digest mismatch is rejected.
  9. Legacy run without actions remains readable/exportable with explicit limitations.
  10. Redacted/filtered fields never leak through the normalized export.

Acceptance

  • Versioned ActionRecord/VerifierResult/ActionCost contracts are documented and implemented.
  • o7 run can emit one or more action records without changing current gate semantics.
  • Actions link to finding identities when supplied.
  • Outcomes are verifier-backed and distinguish process success from domain success.
  • Measured costs are recorded; missing costs are never treated as zero.
  • Normalized deterministic export exists for OwnAudit.
  • Existing run records and CLI exit behavior remain compatible.
  • Digest/provenance and secret-redaction controls are tested.

Non-goals

  • No ranking or domain triage inside 007.
  • No automatic disposition of findings.
  • No contextual bandit, CFR or RL scheduler in this issue.
  • No new sandbox/enforcement policy.
  • No migration of OwnAudit's ledger into the private 007 repository.
  • No inference of actions/outcomes from unstructured stdout when explicit records are unavailable.

Activity

  1. PhysShell commented on Jul 13, 2026

    @PhysShell
    OwnerAuthor

    Primary consumers of this export are PhysShell/Own.NET#268 (canonical Evidence Ledger) and PhysShell/OwnAudit#41/#42 (triage and later calibration evaluation). 007 should emit measured facts and verifier outcomes; it should not absorb their domain disposition policy.

  2. PhysShell commented on Jul 13, 2026

    @PhysShell
    OwnerAuthor

    Companion correctness issue: #43 defines replayable run semantics, fail-closed required-gate reduction, event/artifact digest verification, and o7 replay.

    Scope boundary:

    When both exist, ActionRecord/VerifierResult should reference the run replay report/event-chain root rather than implementing a second integrity or verdict reducer.

  3. PhysShell commented on Nov 20, 2026

    @PhysShell
    OwnerAuthor

    Restate v1.8.1: effect receipt tiers and canonical invocation identity (20.11.2026)

    Research page · release notes.

    For action-level evidence, distinguish (1) logical invocation/idempotency key, (2) one specific invocation incarnation / log position (canonical_id in Restate 1.8.1), (3) durable completed step result, (4) remote side effect actually committed and independently acknowledged. These are not interchangeable. Especially a request with lost response may have applied at remote even when durable step receipt is absent; record UNKNOWN pending reconciliation.

    Do not copy a Restate server schema or adopt a second event store. Only compare this tiered receipt model against existing exact-SHA event IDs and run-generation fences. If current IDs already disambiguate and trace external effects, STOP. Release-note identity is version-specific; older/restored invocations may carry null/placeholder canonical IDs; don't assume a non-null new-format ID on every legacy record.

  4. PhysShell commented on Nov 20, 2026

    @PhysShell
    OwnerAuthor

    TESTed donor: TESTED-03 / structured judge receipts (20.11.2026)

    Research page · evaluation · output golden tests.

    TESTed emits JSON Lines with nested start/close events for judgement, tab, context, testcase and test, and stores expected/generated result with status. It uses distinct oracles for return value, text, stderr/stdout, exception, exit code and file. A failed oracle is reported rather than silently counted as success.

    TESTED-03 / STRUCTURED-JUDGE-RECEIPTS / KILL-FIRST: map a 6-case synthetic run into current 007 action/evidence schema. Preserve run_id, attempt ID, source SHA, suite SHA, language capabilities, oracle identity/version, observable result vs expected, explicit terminal states: compile fail, runtime exception, wrong value, unsupported suite feature, timeout, oracle error, environment failure, NOT_TESTED/UNKNOWN, pass. Inject truncated JSONL and retry/replay to check idempotent ingestion and no phantom PASS; expected semantics and full per-test evidence should round trip without info loss.

    GO only if existing receipts misclassify a demonstrated case or omit causally useful evidence. STOP if current exact-SHA event log already represents all states. No new CI gate or duplicate event store. A passing subset is bounded test evidence, not proof of general correctness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions