Skip to content

Define and verify the canonical provider-preflight artifact #24

Open

Description

@Punisheroot

Context

Live commands emit related but different preflight reports. Their passed
flags and caller-supplied counters are not independently trustworthy. Before a
native collector or execution gate can be shared, Needle needs one bounded
artifact schema and a pure verifier that performs no process launch.

Outcome

Define canonical expectation, observed-evidence, artifact, and
verification-result contracts for provider preflight. Add an offline verifier
that recomputes identity and rejects drift, unsafe bounds, or any observed
provider model turn. Native transport collection is a separate dependent issue.

Scope

  • Define versioned bounded schemas with unknown-field rejection.
  • Bind source snapshot/repository SHA, platform, Codex binary/version identity,
    compatibility fixture, model/reasoning/tier, pricing, execution limits,
    output-root identity/containment, estimated budget, cleanup, and observed
    transport evidence.
  • Define which values contribute to artifact identity and freshness/expiry.
  • Derive live-call and provider-turn counters from supplied observed protocol
    events; never trust summary counters or a passed field.
  • Add a pure verifier that compares the artifact with an independently supplied
    current expectation and emits a canonical verification digest.
  • Reject credentials, raw auth state, provider responses, sealed evaluator
    material, raw secret paths, unbounded diagnostics, and unknown fields.
  • Add fixtures for pass, malformed/unknown fields, source/binary/fixture drift,
    pricing/limit mismatch, unsafe root, counter mismatch, non-zero provider
    events, expiry, and cleanup failure.

Non-goals

  • Running Codex, collecting native transport events, starting a provider turn,
    recording approval, migrating commands, or authorizing spend.
  • Executing a benchmark/evaluator or treating verification as readiness
    evidence by itself.
  • Adding a native environment probe to credential-free CI.

Acceptance criteria

  • The verifier is a deterministic offline function over canonical artifact,
    observed events, and current expectation.
  • It recomputes digests, event-derived counters, compatibility, freshness,
    bounds, path identity, and cost inputs instead of trusting summaries.
  • Any provider model event, drift, expiry, unsafe path, malformed field,
    incompatible fixture, or cleanup failure rejects the artifact.
  • The artifact cannot encode credentials, auth contents, provider response,
    sealed material, or raw secret paths.
  • Verification produces a digest suitable for later approval but never
    changes approval state or launches a process.
  • Synthetic fixtures cover every acceptance and rejection branch.

Starting points

Validation

Run focused schema, canonicalization, verifier, event-counter, compatibility,
freshness, path, budget, and adversarial fixture tests. All validation is
process-free and provider-free.

Activity

  1. added
    enhancementNew feature or request
    area: runtimeCore runtime and execution lifecycle
    area: benchmarksBenchmark corpus, runners, and statistical evidence
    on Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: benchmarksBenchmark corpus, runners, and statistical evidencearea: runtimeCore runtime and execution lifecycleenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions