Repository navigation
Description
Classification
- Change type: enhancement / research evidence workflow
- Priority: P2
- Expected value: high for architecture research, technology evaluation and decision records
- Implementation risk: medium-high — plausible prose with weak provenance would be worse than no automation
- Recommendation: YES, but deferred until the run/action evidence foundation is stable
- Repository owner: 007 orchestration and artifact layer
- Depends on / should reuse: Run evidence v2: action-level records linked to findings, costs and verifier outcomes #42 (
ActionRecord, verifier and provenance conventions) - Production status: post-MVP improvement; not required for the locked one-run MVP
Problem
007 already records isolated code-agent runs and is gaining action-level evidence. It does not yet have a strict artifact contract for research tasks such as:
Which external approaches are relevant to this project?
Which claims are supported by primary evidence?
Where do reliable sources disagree?
What remains unknown?
Which recommendation follows from the evidence rather than from model prose?
A normal “research agent” can search, summarize and produce citations, but without source snapshots, exact evidence spans and deterministic citation validation the output is still a polished trust exercise.
The missing capability is not another agent persona. It is a typed, inspectable research packet whose claims can be traced back to immutable source snapshots.
Goal
Add an opt-in research workflow that produces a reviewable Decision Evidence Packet:
research request
-> query plan
-> source collection
-> canonicalization/deduplication
-> immutable source snapshots
-> exact evidence spans
-> atomic claims
-> support/contradiction graph
-> explicit research gaps
-> compiled citations
-> decision/report artifact
The model may propose queries, claims, gaps and prose. Ordinary code remains responsible for identity, provenance, quote matching, citation resolution and packet validation.
Proposed artifacts
Suggested additive layout under a run:
runs/<target>/<run-id>/research/
request.json
run-manifest.json
queries.jsonl
sources.jsonl
snapshots/
<source-id>.<ext>
evidence.jsonl
claims.jsonl
contradictions.jsonl
gaps.jsonl
errors.jsonl
outline.json
decision.md
packet-verdict.json
Names are reviewable. The separation of raw snapshots, normalized evidence, claims and rendered prose is not.
Minimal contracts
SourceSnapshot
source_id
original_uri
canonical_uri
source_kind
publisher/domain
retrieved_at
content_type
content_digest
snapshot_path
query/action provenance
EvidenceSpan
evidence_id
source_id
snapshot_digest
start_offset/end_offset or stable structural anchor
exact_text
normalized_text_digest
extraction_method
Every quoted span must be found deterministically in the referenced snapshot after documented normalization. A syntactically valid quote that cannot be matched is rejected.
Claim
claim_id
normalized_statement
scope/time qualifiers
supporting_evidence_ids[]
contradicting_evidence_ids[]
status
confidence_class
confidence_class is evidence-oriented (single_source, corroborated, contested, unsupported, stale), not a model-generated probability.
ResearchGap
gap_id
missing_claim/aspect
reason
preferred_source_kind
follow_up_queries[]
status
Citation compiler
The report writer should emit stable markers such as:
{{cite:claim:C17}}
{{cite:evidence:E42}}
A deterministic compiler must:
- resolve every marker;
- reject unknown IDs;
- ensure the referenced evidence supports the linked claim;
- generate footnotes/references;
- report factual paragraphs with no evidence marker when strict mode is enabled.
The model must not hand-number footnotes and hope arithmetic survives generation.
Source and gap policy
The workflow should record and expose, not hide:
- primary vs secondary source;
- publication/retrieval dates;
- duplicate or syndicated content;
- source concentration by publisher/domain;
- inaccessible or failed sources;
- contradictory evidence;
- claims supported only by vendor/self-authored material;
- stale evidence relative to the research question.
A later gap-analysis step may propose follow-up queries, but its output is typed and bounded by configured iteration, source and token budgets.
Deterministic controls
- Exact quote is absent from the snapshot -> evidence rejected.
- Snapshot digest changes -> dependent evidence and claims fail validation.
- Unknown citation ID -> report compilation fails.
- Duplicate canonical URL/content digest -> one source identity plus explicit duplicate records.
- Contradictory evidence remains attached to the claim and cannot be silently dropped by the renderer.
- A source fetch failure remains in
errors.jsonl; it cannot become a clean “no evidence found” result. - Same packet inputs and normalized model outputs produce byte-stable compiled citations and indexes.
- Secrets or private source content follow existing 007 redaction/output policies.
Evaluation fixture
Commit a small offline fixture with:
- two supporting sources;
- one contradicting source;
- one duplicated/syndicated source;
- one false model-proposed quote;
- one stale source;
- one unsupported report claim;
- one failed fetch record.
The fixture must validate without network or live LLM access.
Acceptance
- Versioned
SourceSnapshot,EvidenceSpan,Claim,ResearchGapand packet-manifest contracts exist. - Source content is snapshotted and digest-pinned.
- Exact evidence spans are verified against snapshots.
- Claim support and contradiction links use stable IDs.
- Citation markers are compiled and validated deterministically.
- Gaps, failures and source concentration remain visible in the final packet.
- An offline fixture covers false quotes, unknown citations, duplicates and contradictory evidence.
- Packet metadata records model IDs, prompt/module versions, tool versions, budgets and action/run provenance.
- Existing
o7 runbehavior remains unchanged unless the research workflow is explicitly selected.
Non-goals
- No general-purpose autonomous web browser.
- No vector database in the first slice.
- No replacement for the existing run evidence or future memory layer.
- No automatic promotion of research output into policy, memory or code.
- No claim that schema-valid model output is factually true.
- No provider-specific Venice dependency; the contracts should remain provider-neutral.
- No unbounded recursive research loop.
- No storage of raw secrets or private content outside configured retention policy.
Origin
Inspired by the strongest parts of Venice's private research-agent guide: provenance-rich intermediate artifacts, gap-aware iteration and source-level evidence reduction. This issue deliberately strengthens the weak parts with exact-span verification, atomic claim identities and a deterministic citation compiler.
Activity