Skip to content

Research / INCIDENT-TO-EVAL-01: source-verified AI incident cases → inert 007 regression fixtures (kill-first) #162

Open

Description

@PhysShell

Classification / status

P3 / RESEARCH / REUSE-FIRST / KILL-FIRST / TEST-ONLY / NOT ADOPTED / NO IMPLEMENTATION AUTHORIZED.

Source: Orca AI Incident Archive · SCHEMA · data quality · dataset stats · full research Page.

Research question: Can independently corroborated, publicly documented failures of AI agents be translated into bounded, harmless, deterministic regression scenarios that reveal new gaps in existing 007 security/admission controls? The archive is a discovery index, not an authoritative incident oracle or a ready-made security benchmark.

Verified source facts and important caveats

  • The published dist/stats.json snapshot has version=2026-11-08, count=417, including 177 incident, 93 research, 74 policy, 58 vulnerability, 15 report records. real_harm=true occurs in 153 records; 175 false and 89 null. Never report all 417 as actual successful attacks.
  • Each Markdown record has YAML frontmatter with kind, type[], severity, confidence, real_harm, ai_involvement, disputed, sources and event date. confidence=A is the editors' grade (359 records), not automatic proof of primary evidence.
  • The archive has a correction history and explicitly separates disputed/unsupported claims. Its CI checks schema/derived outputs and performs an advisory sample URL check; it does not validate that each cited source proves every claim.
  • Sample caveat: GitSpawn has kind=research, real_harm=false and a secondary press URL in the record despite A-grade; do not claim verified in-the-wild exploitation. A normal clone does not copy the source repository's .git/config from the remote. Replit incident also lists secondary references despite grade A. WorkOS/ARTEX includes an agent-leaving-sandbox narrative, not an independently confirmed exploit of sandbox isolation.
  • Dataset content is CC BY 4.0; include attribution and the exact archived source commit if using its compilation or summaries. Original linked source materials have their own rights. No malware, exploit code, or customer data is needed.

Ownership / no duplication

  • 007 #46: immutable source snapshots, evidence spans, claim/contradiction status and deterministic citations. This issue consumes those contracts; no parallel source/claim database.
  • 007 #74: independent admission gates and policy authority.
  • 007 #42, #43: action identities, verifier outcomes and replayable run evidence.
  • 007 #156: untrusted-repository intake and execution surfaces.
  • 007 #161: comment/tool-result contamination tests.
  • 007 #160: false facts and dynamic planner.
  • OwnAudit #45: supply-chain evidence adapters. No direct implementation of their adapters here.

This issue owns only: incident selection → independent source corroboration → one security invariant per case → candidate inert fixture and acceptance oracle → optional bounded regression evaluation.

Gate 0: desk review / STOP-NO-GAP first

  • Pin a single archive commit and read dist/incidents.json, SCHEMA.md, relevant source Markdown and docs/data-quality.md as untrusted research input. Do not auto-fetch all 795 source URLs or copy the whole corpus into 007.
  • Predeclare 10–15 cases selected across incident/research/vulnerability, real_harm, ai_involvement, confidence, and distinct classes (rogue destructive action, prompt injection, repo preflight, credential access, supply-chain, sandbox policy, tool confusion). Include negative and disputed controls, not only sensational/critical records.
  • For each candidate: stable archive record ID, exact archive SHA, event/disclosure dates, link and content digest of genuinely first-hand evidence when available, supporting/contradicting exact spans, attribution and what remains UNKNOWN.
  • Flag source-grade discrepancies, unsupported impact figures, evidence after corrections, ambiguous AI participation, duplicates and missing primary sources. Reject a record as a factual anchor when no adequate source exists, but retain the rejection in the research packet.
  • Compare the hypothesized invariant to existing 007 tests/issues. Mark ALREADY-COVERED / UNIQUE-GAP / NON-REPRODUCIBLE / UNKNOWN before proposing implementation. STOP if no net-new cases survive.

Case-to-test candidate examples (not executable scripts)

Archive case Candidate independent invariant Existing owner
Replit DB deletion An ordinary coding task cannot authorize destructive production-like DB actions; simulate with inert fake DB #74
GitSpawn Inspecting a crafted local repo must not execute unapproved pre-model hooks; use inert Git configuration fixtures only #156
WorkOS/ARTEX Outbound network and API scopes remain bounded; test on fake endpoints and deny-by-default policy, do not assume a real sandbox escape was proven #74/#156
LiteLLM compromised releases An unexpected artifact hash/provider version is rejected against pinned provenance; synthetic manifest only OwnAudit #45

Gate 1: proposed inert regression harness, separate GO required

  • Design a 1:1 trace: record ID → verified first-party claim → trust boundary / required property → synthetic fixture → independent oracle → expected PASS/FAIL/INCONCLUSIVE → 007 action/verifier reference. A record's severity or confidence does not supply the oracle.
  • Per scenario, add one failure fixture and one benign control; evaluate against existing mock/policy/sandbox APIs without real exploits, target hosts, production credentials or client code.
  • Intentionally break an evidence span, swap source SHA, inject a disputed claim, and leave an unsupported source unavailable. The test MUST report UNKNOWN/INCONCLUSIVE, never silently accept a favorable result.
  • Define independent evidence for actual effects vs simulated capability. A model's interpretation of a case or its self-certified success is not authoritative.
  • Capture tool/model version, input/fixture/source digests, selected scope, measured wall-clock and resource cost, emitted actions, rejected policy attempts, verifier results, false positives, unknowns and reviewer effort. Never infer a zero cost from missing usage.
  • Classify reproducibility: VERIFIED-FIXTURE, DESIGN-ONLY, NON-COMPARABLE, REJECTED-EVIDENCE and ALREADY-COVERED, with reason.

Decision / kill first

  • GO-NARROW: adequate first-party evidence, at least one independently reproducible net-new gap beyond existing 007 tests, bounded runtime and no dangerous effects; authorize only a separate design-review / fixture PR if later explicitly approved.
  • KEEP IDEAS / CHANGE: retain typed incident distinctions, correction log, incident-to-invariant trace and negative examples while using existing tests; avoid new framework or data pipeline.
  • PARK / KILL: unreliable source attribution, no feasible independent oracle, duplicated baseline coverage, or more work than expected value.

Hard stop: this issue grants no permission to clone and execute untrusted projects, install malicious packages, run exploit proofs, scan third-party systems, auto-download source pages, embed a whole incident database, create CI gates, change policy, deploy, merge, or alter products. Keep proprietary/customer material and credentials out of fixtures and reports.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions