From 9feb9b466f461c1f37a5a746318951a079fa1410 Mon Sep 17 00:00:00 2001 From: Zhou ziheng Date: Tue, 10 Nov 2026 15:48:54 +0800 Subject: [PATCH 1/6] Record round-scoped knowledge searches and use evidence Agent-assisted-by: Codex (implementer) Signed-off-by: Zhou ziheng --- docs/dashboard.md | 71 +++++++ docs/operations.md | 1 + plugins/inferrsi/skills/inferrsi/SKILL.md | 8 + src/inferrsi/arguments.py | 17 ++ src/inferrsi/cli.py | 10 + src/inferrsi/harness/codex_search.py | 212 +++++++++++++++++++++ src/inferrsi/harness/search_attribution.py | 204 ++++++++++++++++++++ tests/test_optimization_story.py | 18 ++ tests/test_search_attribution.py | 95 +++++++++ 9 files changed, 636 insertions(+) create mode 100644 src/inferrsi/harness/codex_search.py create mode 100644 src/inferrsi/harness/search_attribution.py create mode 100644 tests/test_search_attribution.py diff --git a/docs/dashboard.md b/docs/dashboard.md index 3f581e52..2e208555 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -112,6 +112,77 @@ campaign-relative files; the recorder hashes them and does not copy them. The caller assigns IDs and must serialize campaign record updates. Neither command starts an agent or validates the experiment. +### Import one round's searches + +A search import writes `search/round-.json` inside an existing campaign. +The record can exist before a candidate or run. Read it directly with a JSON +viewer. The agent or host selects the trace for its own session and keeps the +original private. When the Desktop host exposes `CODEX_THREAD_ID`, the agent +can locate its own rollout by matching that exact thread ID in the filenames +under the local Codex sessions directory. A host can also supply the trace +path directly. The import command only opens the supplied file. + +For example, set `campaign` to the campaign directory, `trace` to the local +Codex Desktop rollout JSONL and `corpus` to the absolute skill-library path +**as it appeared in that session**. In the agent's first shell call of the +round, save the next line number. In separate shell calls, use the knowledge +directory normally. Once the completed tool events are visible in the rollout, +run the import from the agent's closing shell call: + +```sh +# Start of round 1. Save the printed value as FIRST for the closing call. +awk 'END { print NR + 1 }' "$trace" + +# Separate agent shell calls with the skill library as the tool working directory. +rg -Hn 'Operator tests passed' methods/inference-optimization-casebook/cases +cat methods/inference-optimization-casebook/cases/qwen-b300-rejected-directions.md + +# End of round 1. FIRST is the value retained at the start of this round. +LAST=$(wc -l < "$trace") +inferrsi campaign import-search --directory "$campaign" --round 1 \ + --trace "$trace" --lines "$FIRST" "$LAST" --knowledge-root "$corpus" --complete +``` + +For round 2 take a new first/last pair, even in the same session. Windows are +one-based and inclusive. If the host buffers events, import after they have +flushed. After an interruption the host can do the same handoff. Use +`--context-id ` once per `campaign capture-context` performed in this round. +These inputs can be combined with the rollout or used alone for an Indexer-only +round. Each capture is verified and its returned refs, revisions, request and +resource paths are recorded. Opening a resource and receiving a prepared context +are distinct observations. A bare `collect` leaves no such capture. + +The importer reads Desktop `CommandExecution` events. It supports direct text +or filename-match searches with `rg`, literal `cat`, `nl -ba` and `sed -n N,Mp` +reads. Use `rg -Hn` for unambiguous filenames and matching lines. File results +are corpus-relative; no source text, shell output, private rollout path or +assistant reply is copied. Unsupported search pipelines and unparseable +output leave unresolved event line numbers. Other tools and scripted/remote +retrievals require separate evidence. Review query text and paths before sharing. + +- `observed` counts recognized searches, distinct materials per search (hits), + distinct hit refs across searches, and distinct refs with supported use. + A later file read is a separate event. File paths and versioned Indexer + entries keep their original identities, with no inferred cross-tool alias. +- Supply `--complete` only when the selected inputs cover every in-scope + retrieval for the round. Otherwise only observed counts are known and + `counts` is null. A complete round without a search has zero searches. + With no inputs the command records missing coverage. A failed query or + unresolved supported command also makes totals unknown. +- Use and rejection can come from the existing optimization log when its + iteration references the exact verified capture. For a decision recorded + in an assistant message, a reviewer can pass `--used REF LINE` or + `--rejected REF LINE` with the actual material ref and the JSONL line of + the relevant message. This explicitly attributes use; the importer validates + the locator and never infers a decision from a read. Unattributed use stays + unknown. Distinct use and rejection decisions for the same ref are both retained + and it counts once as used. Candidate links carry the recorded evaluation verdict when present. +- Serialize imports along with other campaign updates. Repeating an import + rebuilds the snapshot without adding events. To attach a later decision or + run, retain the same trace, start line, capture IDs and scope, and extend the + end line if necessary before the next round starts. Separate rounds cannot + claim overlapping windows or the same retrieval. + ### Record the optimization story (campaign trace) The **Optimization story** section is the campaign-level trace a reviewer reads diff --git a/docs/operations.md b/docs/operations.md index 39abbed3..cf45ac3d 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -28,6 +28,7 @@ Effect classes used below: | Execute/inspect experiments | CLI `run`; `status`; `report` | Experiment-executing / read-only / report-generating | `run` executes caller-supplied commands and records evidence; `status` reads recorded progress; `report` generates HTML from recorded evidence | | Visualize a recorded run or campaign | CLI `dashboard`; [dashboard](dashboard.md) | Report-generating / local read-only serving | Writes an offline HTML snapshot or serves an optional live view on loopback; does not run an agent or validate a claim | | Capture a knowledge context | CLI `campaign capture-context`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Runs `collect` with a context budget and keeps the full envelope and resource hashes in a new `knowledge/.json`; proves what was prepared, not what a model read | +| Import one round of knowledge searches | CLI `campaign import-search`; [round import](dashboard.md#import-one-rounds-searches) | Local evidence-writing | Reads a caller-selected Codex rollout window and/or existing context captures, writes material and decision references under the campaign | | Record the optimization log | CLI `campaign log`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Validates and stores a new numbered campaign log version; the dashboard cross-checks it against captures and runs | | Assemble an optimization trace | CLI `campaign assemble-trace`; [offline dashboard](dashboard.md#capture-the-optimization-trace) | Local evidence-writing | Copies an explicitly reviewed context into a new bounded campaign-root snapshot; caller owns redaction and narrative accuracy | | Associate runs with a local campaign | CLI `campaign init` / `campaign record` / `campaign decide`; [offline dashboard](dashboard.md) | Local evidence-writing | Creates caller-named campaign, round and candidate records; references existing artifacts and never starts an agent | diff --git a/plugins/inferrsi/skills/inferrsi/SKILL.md b/plugins/inferrsi/skills/inferrsi/SKILL.md index fa775b42..06cf97f4 100644 --- a/plugins/inferrsi/skills/inferrsi/SKILL.md +++ b/plugins/inferrsi/skills/inferrsi/SKILL.md @@ -96,6 +96,14 @@ why, the decision and its reason. At the end, write success and failure summaries that cite `candidate:` evidence and name the failing stage. Resolve every **Narrative vs evidence** warning the dashboard shows, or explain it. +When recording search attribution, mark the first line of the current round +in the caller-selected Codex rollout before searching. At round close, use +`campaign import-search` once with that round's inclusive line range, corpus +root and any capture IDs. If the rollout is available only after interruption, +the host can import the same range later. Keep private raw traces outside the +campaign. The round-import section of `docs/dashboard.md` describes how to +distinguish observed reads, explicit use and missing capture. + For a bilingual dashboard, prepare and review a translation overlay for the recorded narrative, including the knowledge context, search space, decisions, benchmark and quality descriptions. Supply it to `dashboard --translations` as diff --git a/src/inferrsi/arguments.py b/src/inferrsi/arguments.py index 577c91a9..05dee825 100644 --- a/src/inferrsi/arguments.py +++ b/src/inferrsi/arguments.py @@ -179,6 +179,23 @@ def parser() -> argparse.ArgumentParser: capture.add_argument("--context-bytes", type=int, default=65536) capture.add_argument("--skill-root", type=Path) capture.add_argument("--store", type=Path) + search = campaign_actions.add_parser( + "import-search", help="Import the observed knowledge searches for one campaign round" + ) + search.add_argument("--directory", type=Path, required=True) + search.add_argument("--round", type=int, required=True) + search.add_argument("--trace", type=Path, help="Caller-selected Codex Desktop rollout JSONL") + search.add_argument("--lines", nargs=2, type=int, metavar=("FIRST", "LAST"), + help="Inclusive, one-based round boundaries in that rollout") + search.add_argument("--knowledge-root", type=Path, help="Absolute corpus path in the rollout") + search.add_argument("--context-id", action="append", default=[], help="Capture in this round") + search.add_argument("--stage", help="Retrieval stage if known, e.g. diagnosis") + search.add_argument("--used", action="append", nargs=2, metavar=("REF", "DECISION_LINE"), + default=[], help="Reviewed material use in a cited assistant message") + search.add_argument("--rejected", action="append", nargs=2, metavar=("REF", "DECISION_LINE"), + default=[], help="Reviewed rejection in a cited assistant message") + search.add_argument("--complete", action="store_true", + help="Caller confirms coverage of all in-scope retrievals in this round") log = campaign_actions.add_parser( "log", help="Record a new version of the campaign optimization log" ) diff --git a/src/inferrsi/cli.py b/src/inferrsi/cli.py index 6736ba47..ac4a9b3d 100644 --- a/src/inferrsi/cli.py +++ b/src/inferrsi/cli.py @@ -321,6 +321,16 @@ def main(argv: list[str] | None = None) -> int: _print({"context": path.relative_to(Path(args.directory).resolve()).as_posix(), "sha256": digest}) return 0 + if args.campaign_action == "import-search": + from .harness.search_attribution import import_search + + path = import_search(args.directory, number=args.round, trace=args.trace, + lines=args.lines, knowledge_root=args.knowledge_root, + context_ids=args.context_id, used=args.used, + rejected=args.rejected, stage=args.stage, + complete=args.complete) + _print({"record": path.relative_to(args.directory.resolve()).as_posix()}) + return 0 if args.campaign_action == "log": from .harness.optimization_log import record_log diff --git a/src/inferrsi/harness/codex_search.py b/src/inferrsi/harness/codex_search.py new file mode 100644 index 00000000..461cc9d9 --- /dev/null +++ b/src/inferrsi/harness/codex_search.py @@ -0,0 +1,212 @@ +"""Extract file search observations from a caller-selected Codex Desktop rollout window.""" + +from __future__ import annotations + +import json +import os +import re +import shlex +from pathlib import Path +from urllib.parse import unquote, urlsplit + +from .campaign_records import _identifier +from .dashboard_data import _relative_location, _text + + +def _material(cwd: Path, name: str, corpus: Path) -> str | None: + # The original tree may be offline; use the recorded working directory. + path = Path(os.path.normpath(cwd / name)) + if path.is_relative_to(corpus): + return _relative_location(path.relative_to(corpus).as_posix()) + return None + + +def _rg_args(args: list[str]) -> tuple[str, list[str], str, list[str]] | None: + """Decode common text/filename rg output; decline options that change its meaning.""" + patterns, operands, options = [], [], [] + mode = "text" + it = iter(args) + for arg in it: + if arg == "--": + operands.extend(it) + break + if arg in ("-e", "--regexp"): + pattern = next(it, None) + if pattern is None: + return None + patterns.append(pattern) + elif arg in ("-g", "--glob", "-A", "-B", "-C", "--context", "--max-count", "-m"): + value = next(it, None) + if value is None: + return None + options.extend([arg, _text(value)]) + elif arg in ("--files-with-matches", "-l"): + mode = "files" + options.append(arg) + elif arg in ("--no-heading", "--with-filename", "--line-number", "--ignore-case", + "--fixed-strings", "--word-regexp", "--smart-case", "--hidden"): + options.append(arg) + elif arg.startswith("-"): + if not re.fullmatch(r"-[inSFwHs]+", arg): + return None + options.append(arg) + else: + operands.append(arg) + if not patterns and operands: + patterns.append(operands.pop(0)) + return (" | ".join(patterns), operands or ["."], mode, options) if patterns else None + + +def _rg_hits(output, cwd, corpus, scopes, mode, options): + hits = {} + scopes = [Path(os.path.normpath(cwd / s)) for s in scopes] + has_context = any(opt in options for opt in ("-A", "-B", "-C", "--context")) + complete = True + for line in output.splitlines(): + if "tokens truncated" in line and ("…" in line or "..." in line): + complete = False + continue + if mode == "files": + path, number = line, None + else: + if has_context and (line == "--" or re.match(r"^(?:[^:]*-)?\d+-", line.split(":")[0]) + and not re.match(r"^[^:]+:\d+:", line)): + continue # context rather than a matching line + # For a single explicit file, rg -n omits the filename. Do not stat + # the live corpus: the retained rollout may outlive that checkout. + bare = re.match(r"^(\d+):", line) + if bare and len(scopes) == 1 and scopes[0] != cwd: + path, number = str(scopes[0]), int(bare[1]) + else: + match = re.match(r"^([^:]+):(?:(\d+):)?", line) + if not match: + complete = False + continue + path, number = match[1], int(match[2]) if match[2] else None + candidate = Path(os.path.normpath(cwd / path)) + if not any(candidate.is_relative_to(scope) for scope in scopes): + complete = False + continue + ref = _material(cwd, str(candidate), corpus) + if ref: + hits.setdefault(ref, number) + elif candidate.is_relative_to(corpus): + complete = False + return [{"ref": ref, "match_line": n} for ref, n in hits.items()], complete + + +def _command(item, corpus): + argv = item.get("command") + if not isinstance(argv, list) or not argv or not all(isinstance(a, str) for a in argv): + return None, False + if (Path(argv[0]).name in ("sh", "bash", "zsh", "dash") and len(argv) == 3 + and argv[1] in ("-c", "-lc", "-ic", "-ilc")): + try: + lexer = shlex.shlex(argv[2], posix=True, punctuation_chars=";&|<>") + lexer.whitespace_split = True + argv = list(lexer) + except ValueError: + return None, False + if not argv: + return None, True + tool = Path(argv[0]).name + if tool not in ("rg", "cat", "sed", "nl"): + # A pipeline or chained directory change may hide an in-scope action. + return None, not any(Path(word).name in ("rg", "cat", "sed", "nl") for word in argv) + cwd = item.get("cwd") + if isinstance(cwd, str) and cwd.startswith("file://"): + uri = urlsplit(cwd) + cwd = unquote(uri.path) if uri.netloc in ("", "localhost") else None + if not isinstance(cwd, str) or not Path(cwd).is_absolute(): + return None, False + cwd = Path(cwd) + if any(word in {"|", "||", "&&", ";", ">", ">>", "<"} for word in argv): + return None, False + code, output = item.get("exit_code"), item.get("aggregated_output") + if not isinstance(output, str) or type(code) is not int: + return None, False + if tool == "rg": + if "--files" in argv: + return None, True # inventory, not a content search + parsed = _rg_args(argv[1:]) + if parsed is None: + return None, False + query, scopes, mode, options = parsed + if any(re.search(r"[$`*?~{}\[]", value) for value in scopes): + return None, False # literal command text cannot resolve expanded operands + paths = [Path(os.path.normpath(cwd / value)) for value in scopes] + scope_refs = ["." if p == corpus or corpus.is_relative_to(p) + else _material(cwd, str(p), corpus) for p in paths] + if not any(scope_refs): + return None, True + materials, complete = [], code == 1 + if code == 0: + materials, complete = _rg_hits(output, cwd, corpus, scopes, mode, options) + event = {"action": "search", "tool": tool, "query": _text(query), + "scope": [s for s in scope_refs if s], "options": options, + "exit_code": code, "materials": materials} + return event, complete and (code != 0 or bool(materials)) + if tool == "cat": + names = argv[2:] if argv[1:2] == ["--"] else argv[1:] + if any(n.startswith("-") for n in names): + return None, False + elif len(argv) == 4 and tool == "sed" and argv[1] == "-n" and re.fullmatch( + r"\d+(?:,\d+)?p", argv[2]): + names = argv[3:] + elif tool == "nl" and len(argv) == 3 and argv[1] == "-ba": + names = argv[2:] + else: + return None, False + if any(re.search(r"[$`*?~{}\[]", n) for n in names): + return None, False + refs = [ref for n in names if (ref := _material(cwd, n, corpus))] + if not refs or code != 0: + return None, True + # Successful cat opened every literal operand. Its shared stdout cannot + # establish which file supplied a particular passage. + return {"action": "read", "tool": tool, + "materials": [{"ref": ref} for ref in dict.fromkeys(refs)]}, True + + +def read_window(trace: Path, lines: tuple[int, int], corpus: Path): + """Return only attributed events and decision locators, never the raw transcript.""" + first, last = lines + if first < 1 or last < first or not corpus.is_absolute(): + raise ValueError("expected a positive inclusive line range and absolute knowledge root") + events, decisions, seen, uncertain = [], set(), set(), [] + with trace.open("rb") as stream: + meta = json.loads(stream.readline()) + if not isinstance(meta, dict) or meta.get("type") != "session_meta": + raise ValueError("expected Codex session_meta in the first trace line") + session = _identifier(meta.get("payload", {}).get("id"), "session ID") + number = 1 + while number < last: + raw = stream.readline() + if not raw: + raise ValueError("round boundary exceeds the available trace") + number += 1 + if number < first: + continue + row = json.loads(raw) + payload = row.get("payload", {}) + if (row.get("type") == "response_item" and payload.get("type") == "message" + and payload.get("role") == "assistant" + and payload.get("phase") in ("commentary", "final_answer") + and any(isinstance(part, dict) and part.get("text") + for part in payload.get("content", []))): + decisions.add(number) + if row.get("type") != "event_msg" or payload.get("type") != "item_completed": + continue + item = payload.get("item", {}) + if item.get("type") != "CommandExecution": + continue + event_id = _identifier(item.get("id"), "command event ID") + if event_id in seen: + continue + seen.add(event_id) + event, complete = _command(item, corpus) + if event: + events.append(dict(event, trace_line=number, event_id=event_id)) + if not complete: + uncertain.append(number) + return {"session": session, "lines": [first, last]}, events, decisions, uncertain diff --git a/src/inferrsi/harness/search_attribution.py b/src/inferrsi/harness/search_attribution.py new file mode 100644 index 00000000..4510ced9 --- /dev/null +++ b/src/inferrsi/harness/search_attribution.py @@ -0,0 +1,204 @@ +"""Project one round's observed knowledge events into its local campaign.""" + +from __future__ import annotations + +import hashlib +import json +import os +from pathlib import Path +from tempfile import NamedTemporaryFile + +from .campaign_records import _identifier, _round_records +from .codex_search import read_window +from .dashboard_data import _inside, _object +from .knowledge_capture import MAX_CONTEXT_CAPTURE_BYTES, verify_capture +from .optimization_log import STAGES, read_latest_log + + +def _capture(root, campaign_id, context_id): + _identifier(context_id, "context ID") + path = root / "knowledge" / f"{context_id}.json" + if not _inside(root, path): + raise ValueError("capture escapes the campaign") + with path.open("rb") as stream: + raw = stream.read(MAX_CONTEXT_CAPTURE_BYTES + 1) + doc, state = verify_capture(raw) + if not doc or doc["campaign_id"] != campaign_id or doc["context_id"] != context_id: + raise ValueError(f"invalid or mismatched campaign capture ({state})") + resources = {c["ref"]: [r["path"] for r in c["resources"]] for c in doc["contexts"]} + if any(not isinstance(item.get("ref"), str) or not isinstance(item.get("revision"), str) + for item in doc["items"]): + raise ValueError("capture item is missing its ref or revision") + materials = [{"ref": item["ref"], "revision": item["revision"], + "resources": resources.get(item["ref"], [])} for item in doc["items"]] + return {"action": "search", "tool": "collect", "context_id": context_id, + "source": path.relative_to(root).as_posix(), "request": doc["request"], + "selection": doc.get("selection"), "materials": materials, + "sha256": hashlib.sha256(raw).hexdigest()} + + +def _use(materials, ref, status, evidence): + if ref not in materials: + raise ValueError("use evidence must name a material observed in this round") + item = materials[ref] + evidence = {"use": status, **evidence} + history = item.setdefault("evidence", []) + if evidence not in history: + history.append(evidence) + states = {entry["use"] for entry in history} + item["use"] = status if len(states) == 1 else "used_and_rejected" + + +def _log_uses(root, campaign_id, number, captures, materials): + if not captures: + return + log, versions, state = read_latest_log(root) + if state == "not_recorded": + return + if not log or log["campaign_id"] != campaign_id: + raise ValueError(f"cannot correlate campaign optimization log ({state})") + iteration = next((i for i in log["iterations"] if i["round"] == number), {}) + for index, entry in enumerate(log["knowledge"]): + key = entry["context_id"] + if key not in iteration.get("context_ids", []) or key not in captures: + continue + capture = captures[key] + if capture["sha256"] != entry["sha256"].lower(): + raise ValueError("optimization log and selected capture revisions differ") + valid = {m["ref"] for m in capture["materials"]} + source = f"{versions[-1]['path']}#/knowledge/{index}" + for status, refs in (("used", entry["used_refs"]), + ("rejected", [r["ref"] for r in entry.get("rejected_refs", [])])): + for ref in refs: + if ref not in valid: + raise ValueError("optimization log cites a material absent from its capture") + _use(materials, ref, status, {"source": source, "context_id": key, + "stage": entry["stage"]}) + + +def _write(root, record): + directory = root / "search" + if not _inside(root, directory): + raise ValueError("search directory escapes the campaign") + directory.mkdir(exist_ok=True) + target = directory / f"round-{record['round']}.json" + window, contexts = record["trace"], set(record["context_ids"]) + # As with campaign record, callers serialize writes to a campaign. + for path in directory.glob("round-*.json"): + old, state = _object(path, root) + if state != "present": + raise ValueError("existing search record is unreadable") + previous = old["trace"] + if path == target: + if previous and (not window or previous["session"] != window["session"] + or previous["lines"][0] != window["lines"][0] + or previous["lines"][1] > window["lines"][1]): + raise ValueError("reimport must preserve the original round window") + if not set(old["context_ids"]) <= contexts: + raise ValueError("reimport must preserve the round's captures") + # A closing import may add the final reply or a run to an earlier + # snapshot. Keep previously reviewed decisions for this same window. + for ref, material in old["materials"].items(): + for evidence in material.get("evidence", []): + if evidence.get("basis") != "caller_reviewed": + continue + current = record["materials"].get(ref, {}) + if evidence not in current.get("evidence", []): + _use(record["materials"], ref, evidence["use"], evidence) + record["observed"]["used"] = sum( + m["use"] in ("used", "used_and_rejected") for m in record["materials"].values() + ) + if old == record: + return target + continue + if contexts.intersection(old["context_ids"]): + raise ValueError("capture is already associated with another round") + if window and previous and window["session"] == previous["session"]: + lo, hi = previous["lines"] + if max(lo, window["lines"][0]) <= min(hi, window["lines"][1]): + raise ValueError("trace window overlaps another round") + with NamedTemporaryFile(mode="w", encoding="utf-8", dir=directory, delete=False) as stream: + temporary = Path(stream.name) + try: + json.dump(record, stream, ensure_ascii=False, indent=2, allow_nan=False) + stream.write("\n") + stream.close() + os.replace(temporary, target) + finally: + temporary.unlink(missing_ok=True) + return target + + +def _round_candidates(root, number): + rounds_path = root / "rounds.jsonl" + if not _inside(root, rounds_path): + raise ValueError("rounds file escapes the campaign") + candidates = next((r["candidates"] for r in _round_records(rounds_path) + if r["round"] == number), []) + runs = [] + for candidate in candidates: + _identifier(candidate, "candidate") + path = root / "candidates" / candidate / "candidate.json" + data, status = _object(path, root) + if (status != "present" or data.get("round") != number + or data.get("candidate_id") != candidate): + raise ValueError("invalid candidate association for the round") + run_id = _identifier(data.get("run_id"), "run ID") + evaluation_path = root / "runs" / run_id / "evaluation.json" + evaluation, result_state = _object(evaluation_path, root) + verdict = evaluation.get("verdict") + runs.append({"candidate": candidate, "run_id": run_id, + "source": path.relative_to(root).as_posix(), + "evaluation": evaluation_path.relative_to(root).as_posix() + if result_state == "present" else None, + "verdict": verdict if verdict in ("PASS", "FAIL", "INCONCLUSIVE") else None}) + return runs + + +def import_search(root: Path, *, number: int, trace=None, lines=None, knowledge_root=None, + context_ids=(), used=(), rejected=(), complete=False, stage=None): + root = root.resolve() + campaign, state = _object(root / "campaign.json", root) + if state != "present" or number < 1: + raise ValueError("expected an existing campaign and a positive round number") + campaign_id = _identifier(campaign.get("campaign_id"), "campaign ID") + trace_inputs = (trace is not None, lines is not None, knowledge_root is not None) + if any(trace_inputs) and not all(trace_inputs): + raise ValueError("--trace, --lines and --knowledge-root must be supplied together") + if stage is not None and stage not in STAGES: + raise ValueError(f"stage must be one of {', '.join(STAGES)}") + if complete and not trace and not context_ids: + raise ValueError("complete coverage needs a trace window or a capture") + window, events, decisions, uncertain = None, [], set(), [] + if trace: + window, events, decisions, uncertain = read_window(trace, lines, knowledge_root) + captures = {key: _capture(root, campaign_id, key) for key in dict.fromkeys(context_ids)} + events.extend(captures.values()) + materials = {} + hits = set() + searches = hit_count = 0 + for event in events: + refs = {m["ref"] for m in event["materials"]} + if event["action"] == "search": + searches += 1 + hit_count += len(refs) + hits.update(refs) + for ref in sorted(refs): + materials.setdefault(ref, {"use": "unknown"}) + _log_uses(root, campaign_id, number, captures, materials) + for status, entries in (("used", used), ("rejected", rejected)): + for ref, line in entries: + if int(line) not in decisions: + raise ValueError("decision line must reference an assistant message in this round") + _use(materials, ref, status, {"decision_line": int(line), "basis": "caller_reviewed"}) + observed = {"searches": searches, "hits": hit_count, "unique_hits": len(hits), + "used": sum(m["use"] in ("used", "used_and_rejected") for m in materials.values())} + coverage = "complete" if complete and not uncertain else "partial" + if not window and not captures: + coverage = "missing" + record = {"schema_version": 1, "campaign_id": campaign_id, "round": number, + "stage": stage, "trace": window, "context_ids": list(captures), "coverage": coverage, + "unresolved_command_lines": uncertain, "events": events, "materials": materials, + "observed": observed, "counts": observed if coverage == "complete" else None, + "candidates": _round_candidates(root, number)} + return _write(root, record) diff --git a/tests/test_optimization_story.py b/tests/test_optimization_story.py index 3efa3f86..92ca53de 100644 --- a/tests/test_optimization_story.py +++ b/tests/test_optimization_story.py @@ -599,3 +599,21 @@ def test_malformed_capture_envelope_is_withheld(tmp_path): retrieval = load_dashboard(tmp_path)["story"]["retrievals"][0] assert retrieval["state"] == "invalid_format" assert "assembled_text" not in retrieval + + +def test_search_import_joins_capture_use_with_failed_run(tmp_path): + from inferrsi.harness.search_attribution import import_search + + digest = campaign(tmp_path) + write_log(tmp_path, log_document(digest)) + evaluation = tmp_path / "runs/run-a/evaluation.json" + evaluation.write_text(json.dumps({"verdict": "FAIL"}), encoding="utf-8") + p = import_search(tmp_path, number=1, context_ids=["r1", "r1"], complete=True) + doc = json.loads(p.read_bytes()) + assert doc["counts"] == {"searches": 1, "hits": 2, "unique_hits": 2, "used": 1} + assert doc["materials"]["package:graphs"]["use"] == "used" + assert doc["materials"]["package:kv"]["use"] == "rejected" + assert doc["events"][0]["materials"][0]["resources"] + assert doc["candidates"][0]["verdict"] == "FAIL" + with pytest.raises(ValueError, match="another round"): + import_search(tmp_path, number=2, context_ids=["r1"]) diff --git a/tests/test_search_attribution.py b/tests/test_search_attribution.py new file mode 100644 index 00000000..4fcb4e11 --- /dev/null +++ b/tests/test_search_attribution.py @@ -0,0 +1,95 @@ +"""Round boundaries and adoption evidence must survive both public retrieval paths.""" + +import json + +import pytest + +from inferrsi.cli import main +from inferrsi.harness.campaign_records import create_campaign +from inferrsi.harness.search_attribution import import_search + + +def command(corpus, cmd, output, event_id, exit_code=0): + item = {"type": "CommandExecution", "id": event_id, "cwd": corpus.as_uri(), + "command": ["/bin/sh", "-lc", cmd], "exit_code": exit_code, + "aggregated_output": output} + return {"type": "event_msg", "payload": {"type": "item_completed", "item": item}} + + +def rollout(path, *events): + rows = [{"type": "session_meta", "payload": {"id": "example-session"}}, *events] + path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8") + + +def test_cli_distinguishes_search_read_and_decision_in_two_rounds(tmp_path): + campaign = tmp_path / "campaign" + corpus, trace = tmp_path / "knowledge", tmp_path / "rollout.jsonl" + create_campaign(campaign, "example") + search = command(corpus, "rg -n 'RoPE' .", "case.md:3:private matched text", "search") + rollout(trace, search, search, command(corpus, "cat case.md", "private body", "read"), + {"type": "response_item", "payload": {"type": "message", "role": "assistant", + "phase": "final_answer", "content": [{"type": "output_text", "text": "Apply case."}]}}, + command(corpus, "pwd", str(corpus), "other")) + args = ["campaign", "import-search", "--directory", str(campaign), "--trace", str(trace), + "--knowledge-root", str(corpus), "--complete"] + first = args + ["--round", "1", "--lines", "2", "5", "--used", "case.md", "5"] + assert main(first) == 0 + path = campaign / "search/round-1.json" + original = path.read_bytes() + assert main(first) == 0 and path.read_bytes() == original + doc = json.loads(original) + assert doc["counts"] == {"searches": 1, "hits": 1, "unique_hits": 1, "used": 1} + assert [e["action"] for e in doc["events"]] == ["search", "read"] + assert doc["events"][0]["materials"] == [{"ref": "case.md", "match_line": 3}] + assert doc["materials"]["case.md"]["evidence"][0]["decision_line"] == 5 + assert b"private" not in original and str(tmp_path).encode() not in original + assert main(args + ["--round", "2", "--lines", "6", "6"]) == 0 + assert json.loads((campaign / "search/round-2.json").read_bytes())["counts"]["searches"] == 0 + with pytest.raises(ValueError, match="overlaps"): + import_search(campaign, number=3, trace=trace, lines=[5, 6], knowledge_root=corpus) + with pytest.raises(ValueError, match="assistant message"): + import_search(campaign, number=1, trace=trace, lines=[2, 5], knowledge_root=corpus, + used=[("case.md", "2")]) + + +def test_partial_or_missing_capture_does_not_report_zero(tmp_path): + create_campaign(tmp_path, "example") + path = import_search(tmp_path, number=1) + assert json.loads(path.read_bytes())["counts"] is None + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "knowledge" + rollout(trace, command(corpus, "rg --json needle .", "unsupported", "search")) + path = import_search(tmp_path, number=1, trace=trace, lines=[1, 2], + knowledge_root=corpus, complete=True) + doc = json.loads(path.read_bytes()) + assert doc["coverage"] == "partial" and doc["counts"] is None + assert doc["unresolved_command_lines"] == [2] + with pytest.raises(ValueError, match="boundary"): + import_search(tmp_path, number=1, trace=trace, lines=[1, 3], knowledge_root=corpus) + + +def test_extend_live_window_and_keep_decision_on_retry(tmp_path): + create_campaign(tmp_path, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "knowledge" + rollout(trace, command(corpus, "rg -nH key cases", "cases/a.md:2:key", "search"), + {"type": "response_item", "payload": {"type": "message", "role": "assistant", + "phase": "final_answer", "content": [{"text": "Use cases/a.md for validation."}]}}) + inputs = {"number": 1, "trace": trace, "knowledge_root": corpus, "complete": True} + path = import_search(tmp_path, lines=[1, 2], **inputs) + assert json.loads(path.read_bytes())["materials"]["cases/a.md"]["use"] == "unknown" + import_search(tmp_path, lines=[1, 3], used=[("cases/a.md", 3)], **inputs) + original = path.read_bytes() + import_search(tmp_path, lines=[1, 3], **inputs) + assert path.read_bytes() == original + + +def test_nonmatching_rg_and_incomplete_search_are_distinct(tmp_path): + create_campaign(tmp_path, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "knowledge" + rollout(trace, command(corpus, "rg -n absent .", "", "empty", exit_code=1), + command(corpus, "rg -n key . | head -1", "case.md:2:key", "pipe")) + inputs = {"trace": trace, "knowledge_root": corpus, "complete": True} + p = import_search(tmp_path, number=1, lines=[2, 2], **inputs) + assert json.loads(p.read_bytes())["counts"] == { + "searches": 1, "hits": 0, "unique_hits": 0, "used": 0} + p = import_search(tmp_path, number=2, lines=[3, 3], **inputs) + assert json.loads(p.read_bytes())["counts"] is None From eb984bb8f23a1d0ed1e3d782542794c6dd3760fd Mon Sep 17 00:00:00 2001 From: Zhou ziheng Date: Tue, 10 Nov 2026 15:51:46 +0800 Subject: [PATCH 2/6] Decode local file URIs with platform path semantics Agent-assisted-by: Codex (implementer) Signed-off-by: Zhou ziheng --- docs/dashboard.md | 5 +++-- src/inferrsi/harness/codex_search.py | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/dashboard.md b/docs/dashboard.md index 2e208555..39647978 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -155,8 +155,9 @@ are distinct observations. A bare `collect` leaves no such capture. The importer reads Desktop `CommandExecution` events. It supports direct text or filename-match searches with `rg`, literal `cat`, `nl -ba` and `sed -n N,Mp` reads. Use `rg -Hn` for unambiguous filenames and matching lines. File results -are corpus-relative; no source text, shell output, private rollout path or -assistant reply is copied. Unsupported search pipelines and unparseable +are corpus-relative. A shell event establishes the command's recorded output; +the outer agent tool may have displayed a shorter excerpt. No source text, +shell output, private rollout path or assistant reply is copied. Unsupported search pipelines and unparseable output leave unresolved event line numbers. Other tools and scripted/remote retrievals require separate evidence. Review query text and paths before sharing. diff --git a/src/inferrsi/harness/codex_search.py b/src/inferrsi/harness/codex_search.py index 461cc9d9..b06f903e 100644 --- a/src/inferrsi/harness/codex_search.py +++ b/src/inferrsi/harness/codex_search.py @@ -7,7 +7,8 @@ import re import shlex from pathlib import Path -from urllib.parse import unquote, urlsplit +from urllib.parse import urlsplit +from urllib.request import url2pathname from .campaign_records import _identifier from .dashboard_data import _relative_location, _text @@ -116,7 +117,7 @@ def _command(item, corpus): cwd = item.get("cwd") if isinstance(cwd, str) and cwd.startswith("file://"): uri = urlsplit(cwd) - cwd = unquote(uri.path) if uri.netloc in ("", "localhost") else None + cwd = url2pathname(uri.path) if uri.netloc in ("", "localhost") else None if not isinstance(cwd, str) or not Path(cwd).is_absolute(): return None, False cwd = Path(cwd) From 49086cf74657fa89baffc74201dc1cac7c72999e Mon Sep 17 00:00:00 2001 From: Zhou ziheng Date: Tue, 10 Nov 2026 16:13:10 +0800 Subject: [PATCH 3/6] Preserve uncertainty in round search attribution Agent-assisted-by: Codex (implementer) Signed-off-by: Zhou ziheng --- docs/dashboard.md | 16 ++++++--- src/inferrsi/harness/codex_search.py | 41 ++++++++++++++++----- src/inferrsi/harness/search_attribution.py | 1 + tests/test_optimization_story.py | 17 +++++++++ tests/test_search_attribution.py | 42 ++++++++++++++++++++++ 5 files changed, 105 insertions(+), 12 deletions(-) diff --git a/docs/dashboard.md b/docs/dashboard.md index 39647978..90a422ae 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -124,8 +124,11 @@ path directly. The import command only opens the supplied file. For example, set `campaign` to the campaign directory, `trace` to the local Codex Desktop rollout JSONL and `corpus` to the absolute skill-library path -**as it appeared in that session**. In the agent's first shell call of the -round, save the next line number. In separate shell calls, use the knowledge +**as it appeared in that session**. The importer normalizes path components +such as `.` and `..` lexically. Use the root spelling recorded in the trace, +including any symlink names, so replay also works after the checkout is gone. +In the agent's first shell call of the round, save the next line number. +In separate shell calls, use the knowledge directory normally. Once the completed tool events are visible in the rollout, run the import from the agent's closing shell call: @@ -169,7 +172,10 @@ retrievals require separate evidence. Review query text and paths before sharing retrieval for the round. Otherwise only observed counts are known and `counts` is null. A complete round without a search has zero searches. With no inputs the command records missing coverage. A failed query or - unresolved supported command also makes totals unknown. + unresolved supported command also makes totals unknown. For example, + `rg -C 2` without line numbers cannot reliably separate context text from + matching filenames. A failed multi-file `cat` may have read only some + operands. These commands retain an unresolved trace locator. - Use and rejection can come from the existing optimization log when its iteration references the exact verified capture. For a decision recorded in an assistant message, a reviewer can pass `--used REF LINE` or @@ -177,7 +183,9 @@ retrievals require separate evidence. Review query text and paths before sharing the relevant message. This explicitly attributes use; the importer validates the locator and never infers a decision from a read. Unattributed use stays unknown. Distinct use and rejection decisions for the same ref are both retained - and it counts once as used. Candidate links carry the recorded evaluation verdict when present. + and it counts once as used. Candidate links carry the recorded evaluation verdict + when present. `evaluation_state` reports the source file's status, including + `missing` for an absent file and `corrupt` for invalid JSON. - Serialize imports along with other campaign updates. Repeating an import rebuilds the snapshot without adding events. To attach a later decision or run, retain the same trace, start line, capture IDs and scope, and extend the diff --git a/src/inferrsi/harness/codex_search.py b/src/inferrsi/harness/codex_search.py index b06f903e..cf81a71e 100644 --- a/src/inferrsi/harness/codex_search.py +++ b/src/inferrsi/harness/codex_search.py @@ -26,6 +26,7 @@ def _rg_args(args: list[str]) -> tuple[str, list[str], str, list[str]] | None: """Decode common text/filename rg output; decline options that change its meaning.""" patterns, operands, options = [], [], [] mode = "text" + context = numbered = False it = iter(args) for arg in it: if arg == "--": @@ -41,27 +42,38 @@ def _rg_args(args: list[str]) -> tuple[str, list[str], str, list[str]] | None: if value is None: return None options.extend([arg, _text(value)]) + if arg in ("-A", "-B", "-C", "--context"): + context = True elif arg in ("--files-with-matches", "-l"): mode = "files" options.append(arg) elif arg in ("--no-heading", "--with-filename", "--line-number", "--ignore-case", "--fixed-strings", "--word-regexp", "--smart-case", "--hidden"): options.append(arg) + if arg == "--line-number": + numbered = True elif arg.startswith("-"): if not re.fullmatch(r"-[inSFwHs]+", arg): return None options.append(arg) + if "n" in arg: + numbered = True else: operands.append(arg) if not patterns and operands: patterns.append(operands.pop(0)) + if mode == "text" and context: + mode = "numbered_context" if numbered else "unnumbered_context" return (" | ".join(patterns), operands or ["."], mode, options) if patterns else None -def _rg_hits(output, cwd, corpus, scopes, mode, options): +def _rg_hits(output, cwd, corpus, scopes, mode): + # Without numbers, "file-context: text" also looks like a match in + # "file-context". Retain the search event but withhold hit attribution. + if mode == "unnumbered_context": + return [], False hits = {} scopes = [Path(os.path.normpath(cwd / s)) for s in scopes] - has_context = any(opt in options for opt in ("-A", "-B", "-C", "--context")) complete = True for line in output.splitlines(): if "tokens truncated" in line and ("…" in line or "..." in line): @@ -70,9 +82,19 @@ def _rg_hits(output, cwd, corpus, scopes, mode, options): if mode == "files": path, number = line, None else: - if has_context and (line == "--" or re.match(r"^(?:[^:]*-)?\d+-", line.split(":")[0]) - and not re.match(r"^[^:]+:\d+:", line)): - continue # context rather than a matching line + if mode == "numbered_context": + numbered_match = re.match(r"^(?:[^:]+:)?\d+:", line) + if line == "--": + continue + if re.match(r"^(?:[^:]*-)?\d+-", line): + # Context text may itself contain ":42:", and filenames + # may contain "-2-". Neither interpretation is certain. + if numbered_match: + complete = False + continue + if not numbered_match: + complete = False + continue # For a single explicit file, rg -n omits the filename. Do not stat # the live corpus: the retained rollout may outlive that checkout. bare = re.match(r"^(\d+):", line) @@ -120,7 +142,7 @@ def _command(item, corpus): cwd = url2pathname(uri.path) if uri.netloc in ("", "localhost") else None if not isinstance(cwd, str) or not Path(cwd).is_absolute(): return None, False - cwd = Path(cwd) + cwd = Path(os.path.normpath(cwd)) if any(word in {"|", "||", "&&", ";", ">", ">>", "<"} for word in argv): return None, False code, output = item.get("exit_code"), item.get("aggregated_output") @@ -142,7 +164,7 @@ def _command(item, corpus): return None, True materials, complete = [], code == 1 if code == 0: - materials, complete = _rg_hits(output, cwd, corpus, scopes, mode, options) + materials, complete = _rg_hits(output, cwd, corpus, scopes, mode) event = {"action": "search", "tool": tool, "query": _text(query), "scope": [s for s in scope_refs if s], "options": options, "exit_code": code, "materials": materials} @@ -161,8 +183,10 @@ def _command(item, corpus): if any(re.search(r"[$`*?~{}\[]", n) for n in names): return None, False refs = [ref for n in names if (ref := _material(cwd, n, corpus))] - if not refs or code != 0: + if not refs: return None, True + if code != 0: + return None, False # Successful cat opened every literal operand. Its shared stdout cannot # establish which file supplied a particular passage. return {"action": "read", "tool": tool, @@ -174,6 +198,7 @@ def read_window(trace: Path, lines: tuple[int, int], corpus: Path): first, last = lines if first < 1 or last < first or not corpus.is_absolute(): raise ValueError("expected a positive inclusive line range and absolute knowledge root") + corpus = Path(os.path.normpath(corpus)) events, decisions, seen, uncertain = [], set(), set(), [] with trace.open("rb") as stream: meta = json.loads(stream.readline()) diff --git a/src/inferrsi/harness/search_attribution.py b/src/inferrsi/harness/search_attribution.py index 4510ced9..2ce1da7e 100644 --- a/src/inferrsi/harness/search_attribution.py +++ b/src/inferrsi/harness/search_attribution.py @@ -149,6 +149,7 @@ def _round_candidates(root, number): verdict = evaluation.get("verdict") runs.append({"candidate": candidate, "run_id": run_id, "source": path.relative_to(root).as_posix(), + "evaluation_state": result_state, "evaluation": evaluation_path.relative_to(root).as_posix() if result_state == "present" else None, "verdict": verdict if verdict in ("PASS", "FAIL", "INCONCLUSIVE") else None}) diff --git a/tests/test_optimization_story.py b/tests/test_optimization_story.py index 92ca53de..7696a5fa 100644 --- a/tests/test_optimization_story.py +++ b/tests/test_optimization_story.py @@ -615,5 +615,22 @@ def test_search_import_joins_capture_use_with_failed_run(tmp_path): assert doc["materials"]["package:kv"]["use"] == "rejected" assert doc["events"][0]["materials"][0]["resources"] assert doc["candidates"][0]["verdict"] == "FAIL" + assert doc["candidates"][0]["evaluation_state"] == "present" with pytest.raises(ValueError, match="another round"): import_search(tmp_path, number=2, context_ids=["r1"]) + + +@pytest.mark.parametrize("expected", ["corrupt", "missing"]) +def test_search_import_reports_unavailable_evaluation(tmp_path, expected): + from inferrsi.harness.search_attribution import import_search + + campaign(tmp_path) + evaluation = tmp_path / "runs/run-a/evaluation.json" + if expected == "missing": + evaluation.unlink() + else: + evaluation.write_text("{", encoding="utf-8") + path = import_search(tmp_path, number=1) + candidate = json.loads(path.read_bytes())["candidates"][0] + assert candidate["evaluation_state"] == expected + assert candidate["evaluation"] is None and candidate["verdict"] is None diff --git a/tests/test_search_attribution.py b/tests/test_search_attribution.py index 4fcb4e11..c8fa5461 100644 --- a/tests/test_search_attribution.py +++ b/tests/test_search_attribution.py @@ -93,3 +93,45 @@ def test_nonmatching_rg_and_incomplete_search_are_distinct(tmp_path): "searches": 1, "hits": 0, "unique_hits": 0, "used": 0} p = import_search(tmp_path, number=2, lines=[3, 3], **inputs) assert json.loads(p.read_bytes())["counts"] is None + + +@pytest.mark.parametrize(("cmd", "output", "exit_code", "searches"), [ + ("rg -C 2 key .", "./case.md-context: foo\n./case.md:key", 0, 1), + ("cat case.md missing.md", "body\ncat: missing.md: not found", 1, 0), +], ids=["unnumbered-context", "partial-cat"]) +def test_ambiguous_output_stays_unresolved(tmp_path, cmd, output, exit_code, searches): + create_campaign(tmp_path, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "knowledge" + rollout(trace, command(corpus, cmd, output, "example", exit_code=exit_code)) + path = import_search(tmp_path, number=1, trace=trace, lines=[2, 2], + knowledge_root=corpus, complete=True) + doc = json.loads(path.read_bytes()) + assert doc["coverage"] == "partial" and doc["counts"] is None + assert doc["unresolved_command_lines"] == [2] + assert doc["observed"]["searches"] == searches + assert doc["materials"] == {} + + +def test_numbered_context_preserves_confirmed_hits_under_normalized_root(tmp_path): + create_campaign(tmp_path, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "knowledge" + rollout(trace, command(corpus, "rg -Hn -C 1 key .", + "./case.md-1-context: foo\n./case.md:2:key", "search")) + # The historical knowledge directory need not exist on the importing machine. + path = import_search(tmp_path, number=1, trace=trace, lines=[2, 2], + knowledge_root=corpus / "old" / "..", complete=True) + doc = json.loads(path.read_bytes()) + assert doc["coverage"] == "complete" + assert doc["events"][0]["materials"] == [{"ref": "case.md", "match_line": 2}] + + +def test_ambiguous_numbered_context_cannot_invent_another_file(tmp_path): + create_campaign(tmp_path, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "knowledge" + rollout(trace, command(corpus, "rg -n -C 1 key .", + "./case.md:1:key\n./case.md-2-context:42: more", "search")) + path = import_search(tmp_path, number=1, trace=trace, lines=[2, 2], + knowledge_root=corpus, complete=True) + doc = json.loads(path.read_bytes()) + assert doc["counts"] is None and doc["unresolved_command_lines"] == [2] + assert set(doc["materials"]) == {"case.md"} From 0896f309d061f695cb884c00db74ce046e2cb7d1 Mon Sep 17 00:00:00 2001 From: Zhou ziheng Date: Tue, 10 Nov 2026 16:26:09 +0800 Subject: [PATCH 4/6] Count searches and adoption only with attributable evidence Agent-assisted-by: Codex (implementer) Signed-off-by: Zhou ziheng --- docs/dashboard.md | 14 +++++++---- src/inferrsi/harness/codex_search.py | 12 ++++++---- src/inferrsi/harness/search_attribution.py | 14 +++++++---- tests/test_optimization_story.py | 21 ++++++++++++++++ tests/test_search_attribution.py | 28 ++++++++++++++++++++++ 5 files changed, 77 insertions(+), 12 deletions(-) diff --git a/docs/dashboard.md b/docs/dashboard.md index 90a422ae..49bd2247 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -160,8 +160,9 @@ or filename-match searches with `rg`, literal `cat`, `nl -ba` and `sed -n N,Mp` reads. Use `rg -Hn` for unambiguous filenames and matching lines. File results are corpus-relative. A shell event establishes the command's recorded output; the outer agent tool may have displayed a shorter excerpt. No source text, -shell output, private rollout path or assistant reply is copied. Unsupported search pipelines and unparseable -output leave unresolved event line numbers. Other tools and scripted/remote +shell output, private rollout path or assistant reply is copied. Multiline +retrieval scripts, unsupported pipelines and unparseable output leave +unresolved event line numbers. Other tools and scripted/remote retrievals require separate evidence. Review query text and paths before sharing. - `observed` counts recognized searches, distinct materials per search (hits), @@ -176,8 +177,13 @@ retrievals require separate evidence. Review query text and paths before sharing `rg -C 2` without line numbers cannot reliably separate context text from matching filenames. A failed multi-file `cat` may have read only some operands. These commands retain an unresolved trace locator. -- Use and rejection can come from the existing optimization log when its - iteration references the exact verified capture. For a decision recorded +- Use and rejection apply to materials observed in the imported round. + A decision based only on a previous round's material is outside this + record's scope. The existing optimization log can supply evidence when + its iteration references the exact verified capture. If that log claims + use of a selected item whose text was omitted, the record retains the + claim under `conflicts`. Counting it as used requires separate evidence. + For a decision recorded in an assistant message, a reviewer can pass `--used REF LINE` or `--rejected REF LINE` with the actual material ref and the JSONL line of the relevant message. This explicitly attributes use; the importer validates diff --git a/src/inferrsi/harness/codex_search.py b/src/inferrsi/harness/codex_search.py index cf81a71e..afff785a 100644 --- a/src/inferrsi/harness/codex_search.py +++ b/src/inferrsi/harness/codex_search.py @@ -32,6 +32,8 @@ def _rg_args(args: list[str]) -> tuple[str, list[str], str, list[str]] | None: if arg == "--": operands.extend(it) break + if arg == "--files": + return "", [], "inventory", [] if arg in ("-e", "--regexp"): pattern = next(it, None) if pattern is None: @@ -122,8 +124,10 @@ def _command(item, corpus): argv = item.get("command") if not isinstance(argv, list) or not argv or not all(isinstance(a, str) for a in argv): return None, False + multiline = False if (Path(argv[0]).name in ("sh", "bash", "zsh", "dash") and len(argv) == 3 and argv[1] in ("-c", "-lc", "-ic", "-ilc")): + multiline = "\n" in argv[2].strip() try: lexer = shlex.shlex(argv[2], posix=True, punctuation_chars=";&|<>") lexer.whitespace_split = True @@ -133,8 +137,8 @@ def _command(item, corpus): if not argv: return None, True tool = Path(argv[0]).name - if tool not in ("rg", "cat", "sed", "nl"): - # A pipeline or chained directory change may hide an in-scope action. + if multiline or tool not in ("rg", "cat", "sed", "nl"): + # A compound script can run several tools against one shared stdout. return None, not any(Path(word).name in ("rg", "cat", "sed", "nl") for word in argv) cwd = item.get("cwd") if isinstance(cwd, str) and cwd.startswith("file://"): @@ -149,12 +153,12 @@ def _command(item, corpus): if not isinstance(output, str) or type(code) is not int: return None, False if tool == "rg": - if "--files" in argv: - return None, True # inventory, not a content search parsed = _rg_args(argv[1:]) if parsed is None: return None, False query, scopes, mode, options = parsed + if mode == "inventory": + return None, True if any(re.search(r"[$`*?~{}\[]", value) for value in scopes): return None, False # literal command text cannot resolve expanded operands paths = [Path(os.path.normpath(cwd / value)) for value in scopes] diff --git a/src/inferrsi/harness/search_attribution.py b/src/inferrsi/harness/search_attribution.py index 2ce1da7e..367e768b 100644 --- a/src/inferrsi/harness/search_attribution.py +++ b/src/inferrsi/harness/search_attribution.py @@ -65,15 +65,21 @@ def _log_uses(root, campaign_id, number, captures, materials): capture = captures[key] if capture["sha256"] != entry["sha256"].lower(): raise ValueError("optimization log and selected capture revisions differ") - valid = {m["ref"] for m in capture["materials"]} + resources = {m["ref"]: m["resources"] for m in capture["materials"]} source = f"{versions[-1]['path']}#/knowledge/{index}" for status, refs in (("used", entry["used_refs"]), ("rejected", [r["ref"] for r in entry.get("rejected_refs", [])])): for ref in refs: - if ref not in valid: + if ref not in resources: raise ValueError("optimization log cites a material absent from its capture") - _use(materials, ref, status, {"source": source, "context_id": key, - "stage": entry["stage"]}) + evidence = {"source": source, "context_id": key, "stage": entry["stage"]} + if status == "used" and not resources[ref]: + conflict = {**evidence, "claimed_use": status, "reason": "context_omitted"} + conflicts = materials[ref].setdefault("conflicts", []) + if conflict not in conflicts: + conflicts.append(conflict) + else: + _use(materials, ref, status, evidence) def _write(root, record): diff --git a/tests/test_optimization_story.py b/tests/test_optimization_story.py index 7696a5fa..846e299c 100644 --- a/tests/test_optimization_story.py +++ b/tests/test_optimization_story.py @@ -634,3 +634,24 @@ def test_search_import_reports_unavailable_evaluation(tmp_path, expected): candidate = json.loads(path.read_bytes())["candidates"][0] assert candidate["evaluation_state"] == expected assert candidate["evaluation"] is None and candidate["verdict"] is None + + +def test_search_import_keeps_omitted_use_claim_without_counting_it(tmp_path): + from inferrsi.harness.search_attribution import import_search + + create_campaign(tmp_path, "camp") + capture, digest = capture_context(tmp_path, context_id="r1", reader=KnowledgeReader(), + relevance="casebook Qwen B300 RoPE", limit=1, context_bytes=1) + context = json.loads(capture.read_bytes()) + ref = context["items"][0]["ref"] + assert context["contexts"] == [] + document = log_document(digest) + document["knowledge"][0].update(used_refs=[ref], rejected_refs=[]) + document["iterations"][0]["knowledge_refs"] = [ref] + write_log(tmp_path, document) + path = import_search(tmp_path, number=1, context_ids=["r1"], complete=True) + record = json.loads(path.read_bytes()) + assert record["counts"] == {"searches": 1, "hits": 1, "unique_hits": 1, "used": 0} + assert record["materials"][ref]["use"] == "unknown" + conflict, = record["materials"][ref]["conflicts"] + assert conflict["context_id"] == "r1" and conflict["reason"] == "context_omitted" diff --git a/tests/test_search_attribution.py b/tests/test_search_attribution.py index c8fa5461..ed514a5d 100644 --- a/tests/test_search_attribution.py +++ b/tests/test_search_attribution.py @@ -135,3 +135,31 @@ def test_ambiguous_numbered_context_cannot_invent_another_file(tmp_path): doc = json.loads(path.read_bytes()) assert doc["counts"] is None and doc["unresolved_command_lines"] == [2] assert set(doc["materials"]) == {"case.md"} + + +@pytest.mark.parametrize(("script", "output"), [ + ("rg -Hn key a.md\nrg -Hn key b.md", "a.md:1:key\nb.md:1:key"), + ("cat a.md\ncat b.md", "first\nsecond"), +]) +def test_multiline_shell_does_not_merge_separate_retrievals(tmp_path, script, output): + create_campaign(tmp_path, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "knowledge" + rollout(trace, command(corpus, script, output, "compound")) + path = import_search(tmp_path, number=1, trace=trace, lines=[2, 2], + knowledge_root=corpus, complete=True) + doc = json.loads(path.read_bytes()) + assert doc["counts"] is None and doc["unresolved_command_lines"] == [2] + assert doc["materials"] == {} + + +@pytest.mark.parametrize("script", ["rg -n -- --files .", "rg -n -e --files ."]) +def test_literal_files_option_is_a_search(tmp_path, script): + create_campaign(tmp_path, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "knowledge" + rollout(trace, command(corpus, "rg --files .", "case.md", "inventory"), + command(corpus, script, "case.md:1:rg --files", "search")) + path = import_search(tmp_path, number=1, trace=trace, lines=[2, 3], + knowledge_root=corpus, complete=True) + doc = json.loads(path.read_bytes()) + assert doc["counts"] == {"searches": 1, "hits": 1, "unique_hits": 1, "used": 0} + assert doc["events"][0]["query"] == "--files" From e9624197bb89b3fb99b78475634e75ffb404829d Mon Sep 17 00:00:00 2001 From: Zhou ziheng Date: Tue, 10 Nov 2026 21:02:22 +0800 Subject: [PATCH 5/6] Archive delivered search passages and reviewed round decisions Agent-assisted-by: Codex (implementer) Agent-assisted-by: Codex / gpt-6-luna effort=max (trace delivery) Agent-assisted-by: Codex / gpt-6-luna effort=max (archive attribution) Agent-assisted-by: Codex / gpt-6-luna effort=max (Qwen evidence replay) Signed-off-by: Zhou ziheng --- docs/dashboard.md | 96 +- docs/operations.md | 2 +- evidence/README.md | 6 + .../README.md | 29 + .../results/summary.json | 15 + .../search/round-1.json | 1491 +++++++++++++++++ plugins/inferrsi/skills/inferrsi/SKILL.md | 4 + src/inferrsi/arguments.py | 2 + src/inferrsi/cli.py | 2 +- src/inferrsi/harness/capture_delivery.py | 184 ++ src/inferrsi/harness/codex_search.py | 564 ++++++- src/inferrsi/harness/codex_tool_returns.py | 261 +++ src/inferrsi/harness/search_archive.py | 154 ++ src/inferrsi/harness/search_attribution.py | 242 ++- tests/test_codex_search_delivery.py | 413 +++++ tests/test_evidence_inventory.py | 2 +- tests/test_search_archive.py | 308 ++++ tests/test_search_capture_delivery.py | 37 + tools/evidence_inventory.json | 4 + 19 files changed, 3682 insertions(+), 134 deletions(-) create mode 100644 evidence/search-attribution-qwen-s2-b1-20261110/README.md create mode 100644 evidence/search-attribution-qwen-s2-b1-20261110/results/summary.json create mode 100644 evidence/search-attribution-qwen-s2-b1-20261110/search/round-1.json create mode 100644 src/inferrsi/harness/capture_delivery.py create mode 100644 src/inferrsi/harness/codex_tool_returns.py create mode 100644 src/inferrsi/harness/search_archive.py create mode 100644 tests/test_codex_search_delivery.py create mode 100644 tests/test_search_archive.py create mode 100644 tests/test_search_capture_delivery.py diff --git a/docs/dashboard.md b/docs/dashboard.md index 49bd2247..e8c4d8af 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -159,8 +159,8 @@ The importer reads Desktop `CommandExecution` events. It supports direct text or filename-match searches with `rg`, literal `cat`, `nl -ba` and `sed -n N,Mp` reads. Use `rg -Hn` for unambiguous filenames and matching lines. File results are corpus-relative. A shell event establishes the command's recorded output; -the outer agent tool may have displayed a shorter excerpt. No source text, -shell output, private rollout path or assistant reply is copied. Multiline +the outer agent tool may have displayed a shorter excerpt. The default local +record keeps locators and omits source text, output and assistant replies. Multiline retrieval scripts, unsupported pipelines and unparseable output leave unresolved event line numbers. Other tools and scripted/remote retrievals require separate evidence. Review query text and paths before sharing. @@ -198,6 +198,98 @@ retrievals require separate evidence. Review query text and paths before sharing end line if necessary before the next round starts. Separate rounds cannot claim overlapping windows or the same retrieval. +#### Archive evidence for readers outside the session + +Use `--archive-review review.json` on the same import when a campaign calls +for independently readable evidence. Use the historical knowledge directory +as `--knowledge-root`. In the review file, identify the repository, revision +and repository-relative directory from the experiment's frozen inputs. These +are caller-declared source coordinates. The importer obtains the passages +from the recorded tool return, so it can also handle a corpus that the host +copied into a temporary directory. + +For example, this reviewer input links a visible match to a passage of the +agent's actual decision. The line numbers for the event and decision locate +records in the private input. The `passage_line` field locates a line in the material. +Use a verbatim, contiguous decision excerpt. The importer checks the cited +event, passage and message before recording the reviewed relation. + +```json +{ + "task": "Decide what to check before combining three Qwen Ascend optimizations.", + "corpus": { + "repository": "InferMatrixLab/InferRSI", + "revision": "f6d200775d554d08fb73673833611dd6ac02d14b", + "root": "src/inferrsi/data/skill_library" + }, + "decisions": [{ + "use": "used", + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "event_line": 73, + "passage_line": 18, + "decision_line": 108, + "decision_quote": "报告称 **RoPE + VAE fastpath + 融合 attention 在初始化时 OOM**，并建议先盘点初始化驻留。因此这是这次组合的首要风险，应先做资源检查，再决定是否进入组合试验。" + }] +} +``` + +```sh +inferrsi campaign import-search --directory "$campaign" --round 1 \ + --trace "$trace" --lines "$FIRST" "$LAST" --knowledge-root "$corpus" \ + --archive-review review.json --complete +``` + +The command keeps `search/round-1.json` as the local window record and +returns the portable file `search/archive/round-1.json`. The latter embeds +the evidence selected by this review. Supply all intended decisions on each +archive import. Existing `--used`, `--rejected` and optimization-log claims +retain their local meaning. In the portable record they appear under +`declarations`. Reviewed passage-to-decision links appear under +`materials[ref].evidence` and contribute to its adopted-material count. + +In the portable file, inspect the model-visible tool return and its +passages before the adoption entry. A search can match three lines of one +file and still count as one material hit. A reviewed decision can use one +of those lines. Hit counts include references the command matched or the Indexer +selected. Use each event's `delivery` to see which of those references can +be placed in the agent's return. A material marked `ref_visible` has a +confirmed reference but no archived content excerpt for that material. A zero `observed.used` means there were no +reviewed positive uses. Each unattributed material retains `use: unknown`. +Runtime truncation describes what the agent received. Archival omissions +describe what the reader can inspect. An ambiguous host +return leaves delivery unknown. A verified Indexer capture establishes what +the Indexer prepared. Its campaign-relative source is the evidence for that +preparation. When a tool actually returned its rendered context, the review +file can also supply `"capture_returns": {"r1-diagnosis": 52}`, where 52 is +the line of the model-facing tool response in the round. Pass the corresponding +`--context-id r1-diagnosis`. Only complete resource blocks with matching +owner, revision, path and body count as visibly delivered. A return that just +names the capture file or prints its digest leaves delivery unknown. +To review the use of a delivered Indexer passage, use its return line as +`event_line`, the entry ID as `ref`, and its capture resource path as +`passage_path`. The one-based `passage_line` selects a line within that +resource. Cite the later assistant decision in the same way as for a file. + +To include a round in repository evidence, copy the portable file to +`evidence//search/round-1.json`. Retain the campaign-relative +paths to cited `knowledge/`, candidate and run artifacts, or document which +are unavailable in the published bundle. Review the generated text, +commands and referenced artifacts under the [privacy rules](privacy-security.md). +The raw Codex rollout stays local. A missing candidate is expected for a +decision-only exercise. See the [Qwen round archive](../evidence/search-attribution-qwen-s2-b1-20261110/README.md) +for the generated evidence and its interpretation. To follow one material across +events and then see its reviewed uses, set `record` to the portable JSON: + +```sh +ref=methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md +jq --arg ref "$ref" '{ + returns: [.events[] | {event: .id, action, query, + passages: [.delivery.passages[]? | select(.ref == $ref)]} + | select(.passages | length > 0)], + decisions: .materials[$ref].evidence +}' "$record" +``` + ### Record the optimization story (campaign trace) The **Optimization story** section is the campaign-level trace a reviewer reads diff --git a/docs/operations.md b/docs/operations.md index cf45ac3d..e6e1f9da 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -28,7 +28,7 @@ Effect classes used below: | Execute/inspect experiments | CLI `run`; `status`; `report` | Experiment-executing / read-only / report-generating | `run` executes caller-supplied commands and records evidence; `status` reads recorded progress; `report` generates HTML from recorded evidence | | Visualize a recorded run or campaign | CLI `dashboard`; [dashboard](dashboard.md) | Report-generating / local read-only serving | Writes an offline HTML snapshot or serves an optional live view on loopback; does not run an agent or validate a claim | | Capture a knowledge context | CLI `campaign capture-context`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Runs `collect` with a context budget and keeps the full envelope and resource hashes in a new `knowledge/.json`; proves what was prepared, not what a model read | -| Import one round of knowledge searches | CLI `campaign import-search`; [round import](dashboard.md#import-one-rounds-searches) | Local evidence-writing | Reads a caller-selected Codex rollout window and/or existing context captures, writes material and decision references under the campaign | +| Import one round of knowledge searches | CLI `campaign import-search`; [round import](dashboard.md#import-one-rounds-searches) | Local evidence-writing | Reads a Codex rollout window or context captures. With `--archive-review`, retains excerpts of attributable tool returns and reviewed decisions for readers of the evidence bundle. | | Record the optimization log | CLI `campaign log`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Validates and stores a new numbered campaign log version; the dashboard cross-checks it against captures and runs | | Assemble an optimization trace | CLI `campaign assemble-trace`; [offline dashboard](dashboard.md#capture-the-optimization-trace) | Local evidence-writing | Copies an explicitly reviewed context into a new bounded campaign-root snapshot; caller owns redaction and narrative accuracy | | Associate runs with a local campaign | CLI `campaign init` / `campaign record` / `campaign decide`; [offline dashboard](dashboard.md) | Local evidence-writing | Creates caller-named campaign, round and candidate records; references existing artifacts and never starts an agent | diff --git a/evidence/README.md b/evidence/README.md index 35da1c08..662775a2 100644 --- a/evidence/README.md +++ b/evidence/README.md @@ -21,3 +21,9 @@ evidence/ ``` Populate per acceptance issue. This scaffold keeps only `.gitkeep` placeholders—no campaign artifacts or pinned SHAs in the template. + +When the experiment studies knowledge retrieval or adoption, archive its +reviewed `search/round-.json` and the campaign-relative artifacts needed +to audit its claims. The [round import guide](../docs/dashboard.md#archive-evidence-for-readers-outside-the-session) +describes how to retain visible passages and decision evidence while keeping +the original session private. diff --git a/evidence/search-attribution-qwen-s2-b1-20261110/README.md b/evidence/search-attribution-qwen-s2-b1-20261110/README.md new file mode 100644 index 00000000..0fdca895 --- /dev/null +++ b/evidence/search-attribution-qwen-s2-b1-20261110/README.md @@ -0,0 +1,29 @@ +# Qwen S2-B1 search and decision record + +In the [Qwen material-discovery exercise](https://github.com/InferMatrixLab/InferRSI/issues/380), +the agent considered combining RoPE, the VAE fastpath and fusion attention on +Ascend. It recommended checking initialization memory before attempting the +combination. The round produced a recommendation and had no candidate or new +device measurement. + +Open the [generated round record](search/round-1.json) to trace that decision. + +1. In `events`, `file-39` records the initial read of the Qwen case. The later + search `file-73` records the query, scope and tool response. Under + `delivery.passages` are three returned lines from the case. Line 18 describes + the reported initialization OOM. Line 19 describes a resource reservation + failure on a different host with a different combination. +2. In `materials`, find `qwen-ascend-rejected-directions.md`. Its `evidence` + links line 18 to the assistant's actual recommendation. Line 16 also has a + reviewed decision. Line 19 has no recorded adoption or rejection. +3. In `unresolved_details`, the broad search and the filename-filtering pipeline + explain the partial coverage. `counts` is null. The individual verified + passages and decisions remain available in the same file. + +The [round outcome](results/summary.json) summarizes the decision and its limits. +The corpus was frozen at [InferMatrixLab/InferRSI revision f6d2007](https://github.com/InferMatrixLab/InferRSI/tree/f6d200775d554d08fb73673833611dd6ac02d14b/src/inferrsi/data/skill_library). +The reviewed passages come from three public sources. + +- Cohort, fusion attention and initialization residency. [Qwen Ascend case, lines 6, 16 and 18](https://github.com/InferMatrixLab/InferRSI/blob/f6d200775d554d08fb73673833611dd6ac02d14b/src/inferrsi/data/skill_library/methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md#L6) +- Applicability conditions and evidence limits. [Typed case, lines 15 and 17](https://github.com/InferMatrixLab/InferRSI/blob/f6d200775d554d08fb73673833611dd6ac02d14b/src/inferrsi/data/skill_library/methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json#L15) +- Conditions for a retry and guidance on adapting a precedent. [Casebook procedure, lines 96 and 103](https://github.com/InferMatrixLab/InferRSI/blob/f6d200775d554d08fb73673833611dd6ac02d14b/src/inferrsi/data/skill_library/methods/inference-optimization-casebook/SKILL.md#L96) diff --git a/evidence/search-attribution-qwen-s2-b1-20261110/results/summary.json b/evidence/search-attribution-qwen-s2-b1-20261110/results/summary.json new file mode 100644 index 00000000..f7026795 --- /dev/null +++ b/evidence/search-attribution-qwen-s2-b1-20261110/results/summary.json @@ -0,0 +1,15 @@ +{ + "title": "Qwen S2-B1 search archive", + "campaign_id": "search-attribution-qwen-s2-b1-20261110", + "record_path": "search/round-1.json", + "decision": "The historical answer treated an author-reported triple-stack initialization OOM as a reason to check initialization residency before deciding whether to combine the optimizations.", + "outcome": { + "candidate_count": 0, + "new_device_measurement": "none" + }, + "coverage": { + "state": "partial", + "counts": null, + "reason": "The broad-search return was truncated. The filename filter has verified path excerpts and an unknown hit count." + } +} diff --git a/evidence/search-attribution-qwen-s2-b1-20261110/search/round-1.json b/evidence/search-attribution-qwen-s2-b1-20261110/search/round-1.json new file mode 100644 index 00000000..308df524 --- /dev/null +++ b/evidence/search-attribution-qwen-s2-b1-20261110/search/round-1.json @@ -0,0 +1,1491 @@ +{ + "schema_version": 2, + "campaign_id": "search-attribution-qwen-s2-b1-20261110", + "round": 1, + "stage": null, + "task": "Qwen S2-B1: Decide what to check before combining three Qwen Ascend optimizations.", + "corpus": { + "repository": "InferMatrixLab/InferRSI", + "revision": "f6d200775d554d08fb73673833611dd6ac02d14b", + "root": "src/inferrsi/data/skill_library" + }, + "trace": { + "lines": [ + 16, + 108 + ], + "archive": true + }, + "context_ids": [], + "coverage": "partial", + "unresolved_command_lines": [ + 25, + 31 + ], + "unresolved_details": [ + { + "trace_line": 25, + "reason": "command_output_incomplete", + "action": "search", + "description": "Output is incomplete; cite only retained lines.", + "id": "file-25", + "operation": { + "tool": "rg", + "action": "search", + "scope": [ + ".", + "frameworks/mlx/lazy-evaluation-stream-lifetimes/SKILL.md", + "skill-index.json", + "frameworks/mlx/lazy-evaluation-stream-lifetimes/references/source-map.md", + "frameworks/diffusers/diffusers-invocation-state/SKILL.md", + "hardware/README.md", + "README.md", + "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/SKILL.md", + "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/SKILL.md", + "frameworks/pytorch/pytorch/references/repo-provenance.md", + "frameworks/pytorch/pytorch/sub-skills/training-and-data/SKILL.md", + "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/references/troubleshooting.md", + "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/references/workflows.md", + "workflows/inference-engine-research/SKILL.md", + "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/references/troubleshooting.md", + "methods/weight-loading-optimization/SKILL.md", + "frameworks/pytorch/pytorch/sub-skills/training-and-data/references/troubleshooting.md", + "frameworks/pytorch/pytorch/sub-skills/custom-operators-and-extension/SKILL.md", + "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/references/workflows.md", + "workflows/inference-engine-research/references/model-aware-preparation.md", + "workflows/rsi-evidence/references/omni-experiments.md", + "methods/weight-loading-optimization/metadata.json", + "frameworks/pytorch/pytorch/sub-skills/training-and-data/references/workflows.md", + "frameworks/pytorch/pytorch/references/troubleshooting.md", + "hardware/cuda/reason-hopper-kernels/references/sources.md", + "frameworks/pytorch/pytorch/sub-skills/distributed-and-checkpoint/SKILL.md", + "frameworks/pytorch/pytorch/sub-skills/installation-and-build/SKILL.md", + "frameworks/pytorch/pytorch/SKILL.md", + "workflows/inference-engine-research/references/repository-source-scope.md", + "repositories/lithos-metal/sub-skills/compile-fusion/SKILL.md", + "frameworks/torch-npu/torch-npu/sub-skills/memory-streams-and-events/metadata.json", + "repositories/lithos-metal/sub-skills/compile-fusion/metadata.json", + "frameworks/pytorch/pytorch/sub-skills/custom-operators-and-extension/references/workflows.md", + "repositories/vllm-ascend/references/sources.md", + "repositories/h3-c/audit-h3-runtime-residency/SKILL.md", + "repositories/h3-metal/audit-h3-metal-fasth3-semantics/SKILL.md", + "repositories/vllm-metal/references/scope-and-notices.md", + "repositories/h3-c/audit-h3-runtime-residency/metadata.json", + "repositories/h3-c/audit-h3-runtime-residency/references/minimax-h3-apple.md", + "repositories/vllm-metal/sub-skills/platform-config/references/configuration.md", + "repositories/h3-c/audit-h3-runtime-residency/references/source-notes.md", + "repositories/vllm/diagnose-vllm-admission-pressure/references/papers-20261015/notes/orca.md", + "repositories/h3-c/audit-h3-runtime-residency/references/phase-residency.md", + "repositories/vllm-metal/metadata.json", + "repositories/h3-c/audit-h3-runtime-residency/references/validation.md", + "repositories/h3-c/audit-h3-runtime-residency/references/precision-and-streaming.md", + "repositories/strata/sub-skills/design-space/references/design-space.md", + "repositories/vllm/diagnose-vllm-admission-pressure/references/papers-20261015/references/llumnix-readset.md", + "repositories/strata/references/design-space-analysis.md", + "repositories/sglang/SKILL.md", + "repositories/sglang/audit-sglang-diffusion-lifetimes/SKILL.md", + "repositories/sglang/audit-sglang-diffusion-lifetimes/metadata.json", + "repositories/lily/sub-skills/model-runtime/SKILL.md", + "repositories/sglang/audit-sglang-diffusion-lifetimes/references/source-notes.md", + "repositories/project-maya/sub-skills/model-runtime/references/notices.md", + "repositories/sglang-omni/sub-skills/sglang-omni-serving/references/recipes-and-tuning.md" + ] + }, + "excerpt": { + "ref": "frameworks/mlx/lazy-evaluation-stream-lifetimes/SKILL.md", + "line": 8, + "text": "Use for an MLX workload with misleading submission timings, missing cross-stream dependencies, unexpected retained memory, or a result read before completion. Do not use as an LLM request scheduler, a model KV-cache policy, a compiler/fusion guide, or proof that a backend is installed." + } + }, + { + "trace_line": 31, + "reason": "unsupported_or_compound_command", + "action": "unknown", + "description": "Samples are filenames only; no content or match count is attributed.", + "id": "file-31", + "operation": { + "tools": [ + "rg", + "rg" + ], + "action": "file_list_filter", + "scope": [ + "." + ], + "compound": true + }, + "command_excerpt": "rg --files | rg -i '(qwen|ascend.*(optimization|video|casebook)|diffusion.*optimization|inference-optimization-casebook)'", + "excerpt": [ + { + "ref": "methods/kernel-writing-guide/cases/hf-qwen35-gdn-binding.md", + "line": null, + "text": "/methods/kernel-writing-guide/cases/hf-qwen35-gdn-binding.md", + "root_normalized": true + }, + { + "ref": "methods/kernel-writing-guide/references/hf-qwen35-gdn-binding.md", + "line": null, + "text": "/methods/kernel-writing-guide/references/hf-qwen35-gdn-binding.md", + "root_normalized": true + }, + { + "ref": "methods/kernel-writing-guide/references/hf-qwen35-gdn-source.json", + "line": null, + "text": "/methods/kernel-writing-guide/references/hf-qwen35-gdn-source.json", + "root_normalized": true + }, + { + "ref": "methods/video-diffusion-optimization/assets/campaign.template.json", + "line": null, + "text": "/methods/video-diffusion-optimization/assets/campaign.template.json", + "root_normalized": true + } + ], + "return_line": 33, + "runtime_truncated": false, + "command_root_normalized": true, + "archive_omitted": true, + "omitted_excerpt_count": 18 + } + ], + "events": [ + { + "action": "search", + "tool": "rg", + "query": "qwen.?image|910b1|ascend|interleav(ed|ing).*rope|npu.*rope|vae fast|fastpath|fused attention|fusion|eager|native eight|eight.step|8.step|turbo", + "scope": [ + "." + ], + "options": [ + "-n", + "-i" + ], + "exit_code": 0, + "materials": [ + { + "ref": "frameworks/mlx/lazy-evaluation-stream-lifetimes/SKILL.md", + "match_line": 8 + }, + { + "ref": "skill-index.json", + "match_line": 4 + }, + { + "ref": "frameworks/mlx/lazy-evaluation-stream-lifetimes/references/source-map.md", + "match_line": 18 + }, + { + "ref": "frameworks/diffusers/diffusers-invocation-state/SKILL.md", + "match_line": 19 + }, + { + "ref": "hardware/README.md", + "match_line": 3 + }, + { + "ref": "README.md", + "match_line": 11 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/SKILL.md", + "match_line": 16 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/SKILL.md", + "match_line": 16 + }, + { + "ref": "frameworks/pytorch/pytorch/references/repo-provenance.md", + "match_line": 15 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/training-and-data/SKILL.md", + "match_line": 16 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/references/troubleshooting.md", + "match_line": 8 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/references/workflows.md", + "match_line": 8 + }, + { + "ref": "workflows/inference-engine-research/SKILL.md", + "match_line": 22 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/references/troubleshooting.md", + "match_line": 7 + }, + { + "ref": "methods/weight-loading-optimization/SKILL.md", + "match_line": 12 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/training-and-data/references/troubleshooting.md", + "match_line": 10 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/custom-operators-and-extension/SKILL.md", + "match_line": 16 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/references/workflows.md", + "match_line": 7 + }, + { + "ref": "workflows/inference-engine-research/references/model-aware-preparation.md", + "match_line": 14 + }, + { + "ref": "workflows/rsi-evidence/references/omni-experiments.md", + "match_line": 24 + }, + { + "ref": "methods/weight-loading-optimization/metadata.json", + "match_line": 108 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/training-and-data/references/workflows.md", + "match_line": 17 + }, + { + "ref": "frameworks/pytorch/pytorch/references/troubleshooting.md", + "match_line": 8 + }, + { + "ref": "hardware/cuda/reason-hopper-kernels/references/sources.md", + "match_line": 59 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/distributed-and-checkpoint/SKILL.md", + "match_line": 16 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/installation-and-build/SKILL.md", + "match_line": 16 + }, + { + "ref": "frameworks/pytorch/pytorch/SKILL.md", + "match_line": 48 + }, + { + "ref": "workflows/inference-engine-research/references/repository-source-scope.md", + "match_line": 19 + }, + { + "ref": "repositories/lithos-metal/sub-skills/compile-fusion/SKILL.md", + "match_line": 2 + }, + { + "ref": "frameworks/torch-npu/torch-npu/sub-skills/memory-streams-and-events/metadata.json", + "match_line": 17 + }, + { + "ref": "repositories/lithos-metal/sub-skills/compile-fusion/metadata.json", + "match_line": 14 + }, + { + "ref": "frameworks/pytorch/pytorch/sub-skills/custom-operators-and-extension/references/workflows.md", + "match_line": 22 + }, + { + "ref": "repositories/vllm-ascend/references/sources.md", + "match_line": 40 + }, + { + "ref": "repositories/h3-c/audit-h3-runtime-residency/SKILL.md", + "match_line": 57 + }, + { + "ref": "repositories/h3-metal/audit-h3-metal-fasth3-semantics/SKILL.md", + "match_line": 51 + }, + { + "ref": "repositories/vllm-metal/references/scope-and-notices.md", + "match_line": 11 + }, + { + "ref": "repositories/h3-c/audit-h3-runtime-residency/metadata.json", + "match_line": 16 + }, + { + "ref": "repositories/h3-c/audit-h3-runtime-residency/references/minimax-h3-apple.md", + "match_line": 18 + }, + { + "ref": "repositories/vllm-metal/sub-skills/platform-config/references/configuration.md", + "match_line": 17 + }, + { + "ref": "repositories/h3-c/audit-h3-runtime-residency/references/source-notes.md", + "match_line": 48 + }, + { + "ref": "repositories/vllm/diagnose-vllm-admission-pressure/references/papers-20261015/notes/orca.md", + "match_line": 22 + }, + { + "ref": "repositories/h3-c/audit-h3-runtime-residency/references/phase-residency.md", + "match_line": 5 + }, + { + "ref": "repositories/vllm-metal/metadata.json", + "match_line": 102 + }, + { + "ref": "repositories/h3-c/audit-h3-runtime-residency/references/validation.md", + "match_line": 30 + }, + { + "ref": "repositories/h3-c/audit-h3-runtime-residency/references/precision-and-streaming.md", + "match_line": 21 + }, + { + "ref": "repositories/strata/sub-skills/design-space/references/design-space.md", + "match_line": 42 + }, + { + "ref": "repositories/vllm/diagnose-vllm-admission-pressure/references/papers-20261015/references/llumnix-readset.md", + "match_line": 20 + }, + { + "ref": "repositories/strata/references/design-space-analysis.md", + "match_line": 14 + }, + { + "ref": "repositories/sglang/SKILL.md", + "match_line": 11 + }, + { + "ref": "repositories/sglang/audit-sglang-diffusion-lifetimes/SKILL.md", + "match_line": 2 + }, + { + "ref": "repositories/sglang/audit-sglang-diffusion-lifetimes/metadata.json", + "match_line": 4 + }, + { + "ref": "repositories/lily/sub-skills/model-runtime/SKILL.md", + "match_line": 12 + }, + { + "ref": "repositories/sglang/audit-sglang-diffusion-lifetimes/references/source-notes.md", + "match_line": 1 + }, + { + "ref": "repositories/project-maya/sub-skills/model-runtime/references/notices.md", + "match_line": 45 + }, + { + "ref": "repositories/sglang-omni/sub-skills/sglang-omni-serving/references/recipes-and-tuning.md", + "match_line": 26 + } + ], + "trace_line": 25, + "event_id": "file-25", + "id": "file-25", + "delivery": { + "state": "visible", + "runtime_truncated": true, + "archive_omitted": true, + "passages": [ + { + "ref": "frameworks/mlx/lazy-evaluation-stream-lifetimes/SKILL.md", + "line": 8, + "text": "Use for an MLX workload with misleading submission timings, missing cross-stream dependencies, unexpected retained memory, or a result read before completion. Do not use as an LLM request scheduler, a model KV-cache policy, a compiler/fusion guide, or proof that a backend is installed." + }, + { + "ref": "frameworks/mlx/lazy-evaluation-stream-lifetimes/SKILL.md", + "line": 29, + "text": "Do not claim a speedup from lazy graph construction time, infer fusion from lazy evaluation, infer zero data-movement cost from unified memory, or transfer Metal callback details to CUDA. A framework test does not establish model quality or serving behavior." + }, + { + "ref": "skill-index.json", + "line": 4, + "text": " \"ascend-aclgraph-capture-replay\": {" + }, + { + "ref": "skill-index.json", + "line": 5, + "text": " \"path\": \"hardware/ascend/ascend-aclgraph-capture-replay\"," + }, + { + "ref": "skill-index.json", + "line": 6, + "text": " \"backend\": \"ascend\"" + }, + { + "ref": "skill-index.json", + "line": 8, + "text": " \"ascend-device-health\": {" + }, + { + "ref": "skill-index.json", + "line": 9, + "text": " \"path\": \"hardware/ascend/ascend-device-health\"," + }, + { + "ref": "skill-index.json", + "line": 10, + "text": " \"backend\": \"ascend\"" + }, + { + "ref": "skill-index.json", + "line": 12, + "text": " \"ascend-distributed-parallel\": {" + }, + { + "ref": "skill-index.json", + "line": 13, + "text": " \"path\": \"hardware/ascend/ascend-distributed-parallel\"," + }, + { + "ref": "skill-index.json", + "line": 14, + "text": " \"backend\": \"ascend\"" + }, + { + "ref": "skill-index.json", + "line": 16, + "text": " \"ascend-graph-mode\": {" + } + ], + "visible_refs": [ + "frameworks/mlx/lazy-evaluation-stream-lifetimes/SKILL.md", + "skill-index.json", + "frameworks/mlx/lazy-evaluation-stream-lifetimes/references/source-map.md", + "frameworks/diffusers/diffusers-invocation-state/SKILL.md", + "hardware/README.md", + "README.md", + "repositories/vllm-ascend/references/sources.md", + "repositories/h3-c/audit-h3-runtime-residency/SKILL.md", + "repositories/h3-metal/audit-h3-metal-fasth3-semantics/SKILL.md", + "repositories/vllm-metal/references/scope-and-notices.md", + "repositories/h3-c/audit-h3-runtime-residency/metadata.json", + "repositories/h3-c/audit-h3-runtime-residency/references/minimax-h3-apple.md", + "repositories/vllm-metal/sub-skills/platform-config/references/configuration.md", + "repositories/h3-c/audit-h3-runtime-residency/references/source-notes.md", + "repositories/vllm/diagnose-vllm-admission-pressure/references/papers-20261015/notes/orca.md", + "repositories/h3-c/audit-h3-runtime-residency/references/phase-residency.md", + "repositories/vllm-metal/metadata.json", + "repositories/h3-c/audit-h3-runtime-residency/references/validation.md", + "repositories/h3-c/audit-h3-runtime-residency/references/precision-and-streaming.md", + "repositories/strata/sub-skills/design-space/references/design-space.md", + "repositories/vllm/diagnose-vllm-admission-pressure/references/papers-20261015/references/llumnix-readset.md", + "repositories/strata/references/design-space-analysis.md", + "repositories/sglang/SKILL.md", + "repositories/sglang/audit-sglang-diffusion-lifetimes/SKILL.md", + "repositories/sglang/audit-sglang-diffusion-lifetimes/metadata.json", + "repositories/lily/sub-skills/model-runtime/SKILL.md", + "repositories/sglang/audit-sglang-diffusion-lifetimes/references/source-notes.md", + "repositories/project-maya/sub-skills/model-runtime/references/notices.md", + "repositories/sglang-omni/sub-skills/sglang-omni-serving/references/recipes-and-tuning.md" + ], + "return_line": 26, + "omitted_passages": 128 + } + }, + { + "action": "read", + "tool": "sed", + "materials": [ + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md" + } + ], + "trace_line": 39, + "event_id": "file-39", + "id": "file-39", + "delivery": { + "state": "visible", + "runtime_truncated": false, + "archive_omitted": true, + "passages": [ + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 1, + "text": "# Qwen Ascend: reported rejected directions" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 3, + "text": "These observations come from the [pinned Ascend guide]([redacted-url]", + "redacted": true + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 4, + "text": "which attributes them to amy-why-3459 at the upstream head recorded in the" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 5, + "text": "typed record below." + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 6, + "text": "They apply to its one-910B1, BF16, eager, 1280x704, native eight-sigma cohort." + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 7, + "text": "They are author-reported, `locally_verified=false`, with InferRSI acceptance" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 8, + "text": "INCONCLUSIVE. No raw timings, images or frozen validator were imported." + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 9, + "text": "Kind, origin, state and source identities belong to the" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 10, + "text": "[typed source and scope record](qwen-ascend-rejected-directions.json)." + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 12, + "text": "| Direction | Recorded observation and disposition | Suggested revisit condition and first check |" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 13, + "text": "| --- | --- | --- |" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 14, + "text": "| Cache eager RoPE cos/sin or expanded SDPA masks | Pixel-identical, but discarded as slower in the source trials. | Profile current recomputation and retained-tensor costs; qualify a matched timing cohort before attributing a regression. |" + } + ], + "visible_refs": [ + "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md" + ], + "return_line": 40, + "omitted_passages": 9, + "redacted_passages": 1 + } + }, + { + "action": "read", + "tool": "cat", + "materials": [ + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json" + } + ], + "trace_line": 45, + "event_id": "file-45", + "id": "file-45", + "delivery": { + "state": "visible", + "runtime_truncated": false, + "archive_omitted": true, + "passages": [ + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 1, + "text": "{" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 2, + "text": " \"id\": \"case.qwen-ascend-rejected-directions\"," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 3, + "text": " \"kind\": \"case\"," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 4, + "text": " \"origin\": [\"external\"]," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 5, + "text": " \"state\": \"author_reported\"," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 6, + "text": " \"content\": \"qwen-ascend-rejected-directions.md\"," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 7, + "text": " \"required_resources\": [\"qwen-ascend-rejected-directions.json\"]," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 8, + "text": " \"sources\": [{" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 9, + "text": " \"type\": \"git\"," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 10, + "text": " \"url\": \"[redacted-url]", + "redacted": true + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 11, + "text": " \"commit\": \"d51c941dc57807c1e16ea698cd9291f414110c45\"," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 12, + "text": " \"paths\": [\"docs/qwen-image-2.1-turbo-ascend.md\"]," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 15, + "text": " \"applicability\": [\"Author-reported single Ascend 910B1 Qwen-Image-2.1-Turbo cohort\", \"BF16, eager, TORCH_SDPA, CFG=1, 1280x704, native eight sigmas\"]," + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json", + "line": 17, + "text": " \"limitations\": [\"Imported author report; locally_verified=false\", \"Raw request timings, RGBA outputs and frozen validator are absent from this import\", \"InferRSI acceptance remains INCONCLUSIVE; the source 30 percent target was not met\", \"No cross-device or HTTP-serving qualification\"]," + } + ], + "visible_refs": [ + "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json" + ], + "return_line": 47, + "omitted_passages": 5, + "redacted_passages": 1 + } + }, + { + "action": "read", + "tool": "sed", + "materials": [ + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md" + } + ], + "trace_line": 52, + "event_id": "file-52", + "id": "file-52", + "delivery": { + "state": "visible", + "runtime_truncated": false, + "archive_omitted": true, + "passages": [ + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 1, + "text": "---" + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 2, + "text": "name: ascend-optimization-casebook" + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 3, + "text": "description: 'Casebook of NPU optimization cases for Ascend inference systems: symptom → diagnosis → action → evidence records for memory-bound, compute-bound, communication-exposed, graph-compilation, and AI-CPU-fallback bottleneck classes. Use when a bottleneck has been classified and you need prior art for what to try next. Complement to inference-optimization-casebook (which covers the caseboo", + "truncated": true + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 5, + "text": " '" + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 6, + "text": "---" + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 8, + "text": "# Ascend NPU Optimization Casebook" + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 10, + "text": "Status: candidate. Cases grounded in MindIE-SD @ 8481d39b and vllm-ascend" + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 11, + "text": "@ bc8115ed source evidence. All cases are source-grounded or inferred from" + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 12, + "text": "observed patterns; none have been independently re-run on a local NPU." + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 13, + "text": "Mark each applied case as `locally_verified: false` until you run it." + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 15, + "text": "## How to use this casebook" + }, + { + "ref": "hardware/ascend/ascend-optimization-casebook/SKILL.md", + "line": 17, + "text": "1. Classify the bottleneck first (see `ascend-performance-analysis` skill)." + } + ], + "visible_refs": [ + "hardware/ascend/ascend-optimization-casebook/SKILL.md" + ], + "return_line": 54, + "omitted_passages": 145, + "truncated_passages": 1 + } + }, + { + "action": "read", + "tool": "sed", + "materials": [ + { + "ref": "methods/inference-optimization-casebook/SKILL.md" + } + ], + "trace_line": 60, + "event_id": "file-60", + "id": "file-60", + "delivery": { + "state": "visible", + "runtime_truncated": false, + "archive_omitted": true, + "passages": [ + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 1, + "text": "---" + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 2, + "text": "name: inference-optimization-casebook" + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 3, + "text": "description: 'How to structure and use a casebook of past optimization decisions as a decision-support tool. Extracts the casebook pattern from TensorRT-LLM''s perf-optimization-casebook skill. Use when building a record of optimization experiments or when deciding which optimization to try next based on prior art." + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 5, + "text": " '" + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 6, + "text": "---" + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 8, + "text": "# Inference Optimization Casebook Pattern" + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 10, + "text": "Status: candidate. Source-grounded from TensorRT-LLM @ f33e0e5f," + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 11, + "text": "`perf-optimization-casebook/SKILL.md`." + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 13, + "text": "## What a casebook is" + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 15, + "text": "A casebook is a queryable record of past optimization decisions: what worked," + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 16, + "text": "what failed, under what conditions, at what cost, and with what evidence." + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 17, + "text": "It is reference material, not a coordinator. It does not run profiling or" + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 96, + "text": "Before retrying, state which condition changed or which uncertainty a bounded" + }, + { + "ref": "methods/inference-optimization-casebook/SKILL.md", + "line": 103, + "text": "A case is a precedent, not a guaranteed config. The procedure and expected" + } + ], + "visible_refs": [ + "methods/inference-optimization-casebook/SKILL.md" + ], + "return_line": 61, + "omitted_passages": 101 + } + }, + { + "action": "read", + "tool": "sed", + "materials": [ + { + "ref": "methods/video-diffusion-optimization/SKILL.md" + } + ], + "trace_line": 66, + "event_id": "file-66", + "id": "file-66", + "delivery": { + "state": "visible", + "runtime_truncated": false, + "archive_omitted": true, + "passages": [ + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 1, + "text": "---" + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 2, + "text": "name: video-diffusion-optimization" + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 3, + "text": "description: 'Run and verify goal-driven video diffusion optimization, backend-agnostic. Use when the user asks for latency, RTF, throughput, or memory targets across VLM, DiT, MoE, attention, or VAE, on any accelerator. Not for text-only LLMs.'" + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 4, + "text": "---" + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 6, + "text": "# Video Diffusion Optimization" + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 8, + "text": "Status: candidate workflow guidance, backend-agnostic. The procedure is first-hand and has not been independently validated. Keep model identifiers, campaign revisions, and numeric gates out of this file; they belong to an acceptance issue and its frozen evidence." + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 10, + "text": "Own the performance objective through implementation and final verification. Do not stop after profiling, proposing changes, or showing a faster microbenchmark. Continue the measure-change-verify loop until the target is met, the authorized budget is exhausted, or an external blocker prevents execution." + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 12, + "text": "This skeleton is backend-agnostic. Operator names, memory layouts, and profiler references are backend-specific. The `ascend-video-diffusion-optimization` skill under `hardware/ascend/` specializes this skeleton with Ascend annotations (`TransData`, `FRACTAL_NZ`, `CANN`) and first-hand field notes; re-derive those per backend before following the candidate-selection sections below." + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 14, + "text": "Verify every stage against the packaged [benchmark design](..[redacted-path] and [evidence discipline](..[redacted-path] A microbenchmark or operator parity check is not end-to-end evidence.", + "redacted": true + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 16, + "text": "## 1. Freeze The Goal Contract" + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 18, + "text": "Write the contract before editing code:" + }, + { + "ref": "methods/video-diffusion-optimization/SKILL.md", + "line": 20, + "text": "- Primary metric and direction, such as a latency, RTF, throughput, or peak-memory bar." + } + ], + "visible_refs": [ + "methods/video-diffusion-optimization/SKILL.md" + ], + "return_line": 68, + "omitted_passages": 116, + "redacted_passages": 1 + } + }, + { + "action": "search", + "tool": "rg", + "query": "interleav.*rope|rope.*interleav|initialization OOM|initialization residency|RoPE \\+ VAE|VAE fastpath|fusion attention", + "scope": [ + "methods/inference-optimization-casebook", + "hardware/ascend/ascend-video-diffusion-optimization" + ], + "options": [ + "-n", + "-i" + ], + "exit_code": 0, + "materials": [ + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "match_line": 16 + } + ], + "trace_line": 73, + "event_id": "file-73", + "id": "file-73", + "delivery": { + "state": "visible", + "runtime_truncated": false, + "archive_omitted": false, + "passages": [ + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 16, + "text": "| Fusion attention, including precise=2/0 and mask-free steps | No gain or a gain below the source keep criterion. | Check exact mask, dtype, shapes and device support; run operator quality then complete pipeline timing. Do not reuse the source keep threshold as a repository default. |" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 18, + "text": "| RoPE + VAE fastpath + fusion attention | Initialization OOM for this triple stack. | Inventory initialization residency before attempting a new combination. |" + }, + { + "ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md", + "line": 19, + "text": "| Separate kept RoPE + VAE stack on a later busy host | Later requests failed resource reservation; this is a different stack/host cohort from the triple-stack initialization failure. | Qualify current headroom and outside contention before retrying; do not attribute this resource blockage to fusion attention. |" + } + ], + "visible_refs": [ + "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md" + ], + "return_line": 75 + } + }, + { + "action": "read", + "tool": "sed", + "materials": [ + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md" + } + ], + "trace_line": 81, + "event_id": "file-81", + "id": "file-81", + "delivery": { + "state": "visible", + "runtime_truncated": false, + "archive_omitted": true, + "passages": [ + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 1, + "text": "---" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 2, + "text": "name: ascend-video-diffusion-optimization" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 3, + "text": "description: 'Run and verify goal-driven Ascend video diffusion optimization. Specializes the backend-agnostic video-diffusion-optimization skeleton with Ascend NPU operator, layout and profiler references. Use when the user asks for latency, RTF, throughput, or HBM targets across VLM, DiT, MoE, attention, or VAE on Ascend NPU. Not for text-only LLMs.'" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 4, + "text": "---" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 6, + "text": "# Ascend Video Diffusion Optimization" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 8, + "text": "Status: candidate workflow guidance, Ascend-specific. The procedure is first-hand and has not been independently validated beyond the [field notes](references/field-notes-video-diffusion.md). Keep model identifiers, campaign revisions, and numeric gates out of this file; they belong to an acceptance issue and its frozen evidence." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 10, + "text": "This skill is the Ascend specialization of the backend-agnostic [video diffusion optimization skeleton](..[redacted-path] Follow that skeleton for the goal contract, baseline discipline, optimization loop, candidate-selection strategy, correctness protection, final acceptance, and final report. This file adds the Ascend-specific operator, layout, and profiler references; it does not repeat the ful", + "redacted": true, + "truncated": true + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 12, + "text": "Own the performance objective through implementation and final verification. Do not stop after profiling, proposing changes, or showing a faster microbenchmark. Continue the measure-change-verify loop until the target is met, the authorized budget is exhausted, or an external blocker prevents execution." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 14, + "text": "Read the field notes only for prior failures and starting hypotheses. Treat every observation there as untraceable and non-reproducible. Derive thresholds, chunk sizes, precision policy, and sparsity from the target model's own measurements." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 16, + "text": "Verify every stage against the packaged [benchmark design](..[redacted-path] and [evidence discipline](..[redacted-path] A microbenchmark or operator parity check is not end-to-end evidence.", + "redacted": true + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 18, + "text": "## Ascend-Specific Layout And Profiler References" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md", + "line": 20, + "text": "Where the common skeleton says \"operator names, memory layouts, and profiler references are backend-specific\", apply these Ascend facts:" + } + ], + "visible_refs": [ + "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md" + ], + "return_line": 82, + "omitted_passages": 6, + "truncated_passages": 1, + "redacted_passages": 2 + } + }, + { + "action": "read", + "tool": "sed", + "materials": [ + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md" + } + ], + "trace_line": 87, + "event_id": "file-87", + "id": "file-87", + "delivery": { + "state": "visible", + "runtime_truncated": false, + "archive_omitted": true, + "passages": [ + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 1, + "text": "# Field Notes: Ascend Video Diffusion Optimization" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 3, + "text": "First-hand observations from video-editing diffusion campaigns on Ascend NPU. Use them as starting hypotheses and as a list of failures to avoid, never as defaults." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 5, + "text": "Provenance and limits:" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 7, + "text": "- The observations cover two variants of one video-editing model family; both were edited for text and both used a vision-language condition encoder." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 8, + "text": "- Exact measurements were removed or coarsened. Remaining figures are order-of-magnitude recollections, not reproducible evidence." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 9, + "text": "- No campaign under a published evidence contract produced these notes. Treat them as `unverified` and non-reproducible." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 10, + "text": "- A redacted artifact cannot turn an unverified observation into local validation. Re-measure on the target model before relying on any figure below." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 12, + "text": "## Variants" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 14, + "text": "- **Variant A.** OCR-oriented vision-language condition encoder; short-video text editing; a sparse-attention path that only becomes eligible above a video-token threshold." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 15, + "text": "- **Variant B.** Vision-language encoder plus a byte-level text encoder; video DiT with routed and shared MoE experts; temporal video VAE." + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 17, + "text": "## Findings That Transferred Across Both Variants" + }, + { + "ref": "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md", + "line": 19, + "text": "1. Stage profiling repeatedly changed the optimization order. Condition encoding, DiT, and VAE dominated at different points; no single optimization stayed dominant through a campaign." + } + ], + "visible_refs": [ + "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md" + ], + "return_line": 89, + "omitted_passages": 61 + } + } + ], + "materials": { + "README.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "frameworks/diffusers/diffusers-invocation-state/SKILL.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "frameworks/mlx/lazy-evaluation-stream-lifetimes/SKILL.md": { + "use": "unknown", + "delivery_state": "visible" + }, + "frameworks/mlx/lazy-evaluation-stream-lifetimes/references/source-map.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "frameworks/pytorch/pytorch/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/references/repo-provenance.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/references/troubleshooting.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/custom-operators-and-extension/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/custom-operators-and-extension/references/workflows.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/distributed-and-checkpoint/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/installation-and-build/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/references/troubleshooting.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/memory-streams-and-graphs/references/workflows.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/references/troubleshooting.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/tensor-dispatch-autograd/references/workflows.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/training-and-data/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/training-and-data/references/troubleshooting.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/pytorch/pytorch/sub-skills/training-and-data/references/workflows.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "frameworks/torch-npu/torch-npu/sub-skills/memory-streams-and-events/metadata.json": { + "use": "unknown", + "delivery_state": "unknown" + }, + "hardware/README.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "hardware/cuda/reason-hopper-kernels/references/sources.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "methods/weight-loading-optimization/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "methods/weight-loading-optimization/metadata.json": { + "use": "unknown", + "delivery_state": "unknown" + }, + "repositories/h3-c/audit-h3-runtime-residency/SKILL.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/h3-c/audit-h3-runtime-residency/metadata.json": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/h3-c/audit-h3-runtime-residency/references/minimax-h3-apple.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/h3-c/audit-h3-runtime-residency/references/phase-residency.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/h3-c/audit-h3-runtime-residency/references/precision-and-streaming.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/h3-c/audit-h3-runtime-residency/references/source-notes.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/h3-c/audit-h3-runtime-residency/references/validation.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/h3-metal/audit-h3-metal-fasth3-semantics/SKILL.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/lily/sub-skills/model-runtime/SKILL.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/lithos-metal/sub-skills/compile-fusion/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "repositories/lithos-metal/sub-skills/compile-fusion/metadata.json": { + "use": "unknown", + "delivery_state": "unknown" + }, + "repositories/project-maya/sub-skills/model-runtime/references/notices.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/sglang-omni/sub-skills/sglang-omni-serving/references/recipes-and-tuning.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/sglang/SKILL.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/sglang/audit-sglang-diffusion-lifetimes/SKILL.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/sglang/audit-sglang-diffusion-lifetimes/metadata.json": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/sglang/audit-sglang-diffusion-lifetimes/references/source-notes.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/strata/references/design-space-analysis.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/strata/sub-skills/design-space/references/design-space.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/vllm-ascend/references/sources.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/vllm-metal/metadata.json": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/vllm-metal/references/scope-and-notices.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/vllm-metal/sub-skills/platform-config/references/configuration.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/vllm/diagnose-vllm-admission-pressure/references/papers-20261015/notes/orca.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "repositories/vllm/diagnose-vllm-admission-pressure/references/papers-20261015/references/llumnix-readset.md": { + "use": "unknown", + "delivery_state": "ref_visible" + }, + "skill-index.json": { + "use": "unknown", + "delivery_state": "visible" + }, + "workflows/inference-engine-research/SKILL.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "workflows/inference-engine-research/references/model-aware-preparation.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "workflows/inference-engine-research/references/repository-source-scope.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "workflows/rsi-evidence/references/omni-experiments.md": { + "use": "unknown", + "delivery_state": "unknown" + }, + "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md": { + "use": "used", + "delivery_state": "visible", + "evidence": [ + { + "use": "used", + "basis": "archive_reviewed", + "event_id": "file-39", + "passage_line": 6, + "passage_text": "They apply to its one-910B1, BF16, eager, 1280x704, native eight-sigma cohort.", + "decision_line": 108, + "decision_quote": "已有 Qwen Ascend 记录正好覆盖单卡 910B1、BF16、eager、1280×704、原生八步这一队列，" + }, + { + "use": "used", + "basis": "archive_reviewed", + "event_id": "file-73", + "passage_line": 18, + "passage_text": "| RoPE + VAE fastpath + fusion attention | Initialization OOM for this triple stack. | Inventory initialization residency before attempting a new combination. |", + "decision_line": 108, + "decision_quote": "报告称 **RoPE + VAE fastpath + 融合 attention 在初始化时 OOM**，并建议先盘点初始化驻留。因此这是这次组合的首要风险，应先做资源检查，再决定是否进入组合试验。" + }, + { + "use": "used", + "basis": "archive_reviewed", + "event_id": "file-73", + "passage_line": 16, + "passage_text": "| Fusion attention, including precise=2/0 and mask-free steps | No gain or a gain below the source keep criterion. | Check exact mask, dtype, shapes and device support; run operator quality then complete pipeline timing. Do not reuse the source keep threshold as a repository default. |", + "decision_line": 108, + "decision_quote": "若 profile 未显示 attention 是明显成本，不应优先投入。" + } + ] + }, + "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json": { + "use": "used", + "delivery_state": "visible", + "evidence": [ + { + "use": "used", + "basis": "archive_reviewed", + "event_id": "file-45", + "passage_line": 15, + "passage_text": " \"applicability\": [\"Author-reported single Ascend 910B1 Qwen-Image-2.1-Turbo cohort\", \"BF16, eager, TORCH_SDPA, CFG=1, 1280x704, native eight sigmas\"],", + "decision_line": 108, + "decision_quote": "记录还指定了 `CFG=1` 和 `TORCH_SDPA`；题目没有提供这两项以及当前代码版本，需先核对适用条件。" + }, + { + "use": "used", + "basis": "archive_reviewed", + "event_id": "file-45", + "passage_line": 17, + "passage_text": " \"limitations\": [\"Imported author report; locally_verified=false\", \"Raw request timings, RGBA outputs and frozen validator are absent from this import\", \"InferRSI acceptance remains INCONCLUSIVE; the source 30 percent target was not met\", \"No cross-device or HTTP-serving qualification\"],", + "decision_line": 108, + "decision_quote": "这条先例是作者报告，`locally_verified=false`，缺原始计时、图像和冻结的质量验证器，验收状态为 `INCONCLUSIVE`。" + } + ] + }, + "hardware/ascend/ascend-optimization-casebook/SKILL.md": { + "use": "unknown", + "delivery_state": "visible" + }, + "methods/inference-optimization-casebook/SKILL.md": { + "use": "used", + "delivery_state": "visible", + "evidence": [ + { + "use": "used", + "basis": "archive_reviewed", + "event_id": "file-60", + "passage_line": 96, + "passage_text": "Before retrying, state which condition changed or which uncertainty a bounded", + "decision_line": 108, + "decision_quote": "“Before retrying”段要求说明条件变化或待解的不确定性，保留原结果，做未改变的 control，并选择成本最低的区分性检查；" + }, + { + "use": "used", + "basis": "archive_reviewed", + "event_id": "file-60", + "passage_line": 103, + "passage_text": "A case is a precedent, not a guaranteed config. The procedure and expected", + "decision_line": 108, + "decision_quote": "“Adapt, do not copy”段说明先例不是保证有效的配置，应按目标模型和硬件重新推导并测量。" + } + ] + }, + "methods/video-diffusion-optimization/SKILL.md": { + "use": "unknown", + "delivery_state": "visible" + }, + "hardware/ascend/ascend-video-diffusion-optimization/SKILL.md": { + "use": "unknown", + "delivery_state": "visible" + }, + "hardware/ascend/ascend-video-diffusion-optimization/references/field-notes-video-diffusion.md": { + "use": "unknown", + "delivery_state": "visible" + } + }, + "declarations": [], + "observed": { + "searches": 2, + "hits": 56, + "unique_hits": 56, + "used": 3 + }, + "counts": null, + "candidates": [], + "result_note": "Historical decision exercise for #380. It produced no candidate and no new device measurement; this record documents retrieval and decision attribution only, not device performance or a candidate result.", + "result_note_basis": "caller_reviewed" +} diff --git a/plugins/inferrsi/skills/inferrsi/SKILL.md b/plugins/inferrsi/skills/inferrsi/SKILL.md index 06cf97f4..f2588f54 100644 --- a/plugins/inferrsi/skills/inferrsi/SKILL.md +++ b/plugins/inferrsi/skills/inferrsi/SKILL.md @@ -103,6 +103,10 @@ root and any capture IDs. If the rollout is available only after interruption, the host can import the same range later. Keep private raw traces outside the campaign. The round-import section of `docs/dashboard.md` describes how to distinguish observed reads, explicit use and missing capture. +For a round intended for repository evidence, supply the reviewed corpus +version and decision references with `--archive-review`. Check that the +published passages, decision excerpts and linked campaign artifacts let a +reader verify the claim without the private rollout. For a bilingual dashboard, prepare and review a translation overlay for the recorded narrative, including the knowledge context, search space, decisions, diff --git a/src/inferrsi/arguments.py b/src/inferrsi/arguments.py index 05dee825..5fdfe403 100644 --- a/src/inferrsi/arguments.py +++ b/src/inferrsi/arguments.py @@ -189,6 +189,8 @@ def parser() -> argparse.ArgumentParser: help="Inclusive, one-based round boundaries in that rollout") search.add_argument("--knowledge-root", type=Path, help="Absolute corpus path in the rollout") search.add_argument("--context-id", action="append", default=[], help="Capture in this round") + search.add_argument("--archive-review", type=Path, metavar="FILE", + help="Build a round archive from reviewed passage and decision references") search.add_argument("--stage", help="Retrieval stage if known, e.g. diagnosis") search.add_argument("--used", action="append", nargs=2, metavar=("REF", "DECISION_LINE"), default=[], help="Reviewed material use in a cited assistant message") diff --git a/src/inferrsi/cli.py b/src/inferrsi/cli.py index ac4a9b3d..ae5c029d 100644 --- a/src/inferrsi/cli.py +++ b/src/inferrsi/cli.py @@ -328,7 +328,7 @@ def main(argv: list[str] | None = None) -> int: lines=args.lines, knowledge_root=args.knowledge_root, context_ids=args.context_id, used=args.used, rejected=args.rejected, stage=args.stage, - complete=args.complete) + complete=args.complete, archive_review=args.archive_review) _print({"record": path.relative_to(args.directory.resolve()).as_posix()}) return 0 if args.campaign_action == "log": diff --git a/src/inferrsi/harness/capture_delivery.py b/src/inferrsi/harness/capture_delivery.py new file mode 100644 index 00000000..10dd8e43 --- /dev/null +++ b/src/inferrsi/harness/capture_delivery.py @@ -0,0 +1,184 @@ +"""Check which prepared Indexer resources also appear in a reviewed tool return.""" + +import hashlib +import json + +from .campaign_records import _identifier +from .dashboard_data import _inside, _relative_location, _text +from .knowledge_capture import MAX_CONTEXT_CAPTURE_BYTES, assemble_context_text, verify_capture + + +def capture_return_texts(trace, lines, references, context_ids): + """Read only tool-return text at the caller's line references in this round.""" + if not isinstance(references, dict): + raise ValueError("capture_returns must map capture IDs to tool-return lines") + if not references: + return {} + if trace is None or lines is None or not set(references).issubset(context_ids): + raise ValueError("capture_returns requires a trace and a selected capture for each ID") + first, last = lines + wanted = set() + for number in references.values(): + if type(number) is not int or not first <= number <= last: + raise ValueError("capture return line is outside the round") + wanted.add(number) + texts = {} + with trace.open("rb") as stream: + for number, raw in enumerate(stream, 1): + if number not in wanted: + continue + row = json.loads(raw) + payload = row.get("payload", {}) + if row.get("type") != "response_item" or payload.get("type") not in ( + "function_call_output", "custom_tool_call_output" + ): + raise ValueError("capture return must reference a model-facing tool response") + output = payload.get("output") + if isinstance(output, str): + texts[number] = [output] + elif isinstance(output, list): + texts[number] = [ + part["text"] for part in output if isinstance(part, dict) + and part.get("type") == "input_text" and isinstance(part.get("text"), str) + ] + else: + texts[number] = [] + if wanted == texts.keys(): + break + if wanted != texts.keys(): + raise ValueError("capture return line is missing from the trace") + return {key: texts[number] for key, number in references.items()} + + +def visible_capture_resources(doc, texts): + """Require matching owner, revision, resource path and body in the visible text.""" + visible = [] + if not texts: + return visible + chunks = ["\n" + text + "\n" for text in texts] + for context in doc["contexts"]: + for resource in context["resources"]: + block = assemble_context_text([{ + **context, "resources": [resource], "omissions": [], + }]) + if any("\n" + block in chunk for chunk in chunks): + visible.append({"ref": context["ref"], "path": resource["path"]}) + return visible + + +def _archive_capture_metadata(doc): + request = doc.get("request", {}) + safe_request = {} + for key in ("query", "relevance", "relevance_method", "kind", "repo", "tier", "backend", + "scenario"): + value = request.get(key) + if isinstance(value, str): + safe_request[key] = _text(value, "", limit=len(value)) + ids = request.get("ids") + if isinstance(ids, list): + safe_request["ids"] = [value for value in ids if isinstance(value, str) + and _relative_location(value) == value] + sources = request.get("sources") + if isinstance(sources, list): + safe_request["sources"] = [_text(value, "", limit=len(value)) + for value in sources if isinstance(value, str)] + for key in ("limit", "context_bytes"): + value = request.get(key) + if type(value) is int and value >= 0: + safe_request[key] = value + selection = doc.get("selection") + safe_selection = {} + if isinstance(selection, dict): + for key in ("method", "cue", "scope"): + value = selection.get(key) + if isinstance(value, str): + safe_selection[key] = _text(value, "", limit=len(value)) + if type(selection.get("ranking_applied")) is bool: + safe_selection["ranking_applied"] = selection["ranking_applied"] + return safe_request, safe_selection + + +def load_campaign_capture(root, campaign_id, context_id): + _identifier(context_id, "context ID") + path = root / "knowledge" / f"{context_id}.json" + if not _inside(root, path): + raise ValueError("capture escapes the campaign") + with path.open("rb") as stream: + raw = stream.read(MAX_CONTEXT_CAPTURE_BYTES + 1) + doc, state = verify_capture(raw) + if not doc or doc["campaign_id"] != campaign_id or doc["context_id"] != context_id: + raise ValueError(f"invalid or mismatched campaign capture ({state})") + return doc, path.relative_to(root).as_posix(), hashlib.sha256(raw).hexdigest() + + +def _capture_passages(doc, selections, visible): + locators = dict.fromkeys((item["ref"], item["passage_path"], item["passage_line"]) + for item in selections) + passages = [] + for ref, path, line in locators: + if (ref, path) not in visible: + raise ValueError("Indexer review needs a matching visible captured resource") + if line is None: + raise ValueError("Indexer review needs a positive source line") + resources = [resource for context in doc["contexts"] if context["ref"] == ref + for resource in context["resources"] if resource["path"] == path] + if len(resources) != 1: + raise ValueError("Indexer review must name one exact captured resource") + lines = resources[0]["content"].splitlines() + if line > len(lines) or not lines[line - 1]: + raise ValueError("Indexer review source line is missing or empty") + text = lines[line - 1] + if _text(text, "", limit=len(text)) != text: + raise ValueError("Indexer review source passage is not safe to publish verbatim") + passages.append({"ref": ref, "path": path, "line": line, "text": text}) + return passages + + +def capture_event(context_id, captured, *, archive=False, return_chunks=None, + return_line=None, selected_passages=()): + doc, source, digest = captured + resources = {c["ref"]: [r["path"] for r in c["resources"]] for c in doc["contexts"]} + materials = [] + for item in doc["items"]: + ref, revision = item.get("ref"), item.get("revision") + if not isinstance(ref, str) or not isinstance(revision, str): + raise ValueError("capture item is missing its ref or revision") + materials.append({"ref": ref, "revision": revision, "resources": resources.get(ref, [])}) + event = {"action": "search", "tool": "collect", "context_id": context_id, + "source": source, "sha256": digest, "request": doc["request"], + "selection": doc.get("selection"), "materials": materials} + if not archive: + return event + + omissions = {o["ref"]: o.get("reason") for o in + [*doc.get("omissions", []), *doc.get("context_omissions", [])] + if isinstance(o, dict) and isinstance(o.get("ref"), str)} + visible = [r for r in visible_capture_resources(doc, return_chunks) + if _relative_location(r["ref"]) == r["ref"] + and _relative_location(r["path"]) == r["path"]] + visible_keys = {(r["ref"], r["path"]) for r in visible} + for material in materials: + ref, revision = material["ref"], material["revision"] + if _relative_location(ref) != ref or _text(revision, "", len(revision)) != revision: + raise ValueError("capture contains a nonportable material ref or revision") + paths = material["resources"] + material["preparation_state"] = "prepared" if paths else "not_recorded" + if ref in omissions: + if not paths: + material["preparation_state"] = "omitted" + material["omission_reason"] = _text(omissions[ref]) + material["resources"] = [p for p in paths if _relative_location(p) == p] + material["delivery_state"] = ("visible" if any((ref, p) in visible_keys for p in paths) + else "unknown") + event["id"] = f"capture:{context_id}" + event["request"], event["selection"] = _archive_capture_metadata(doc) + event["capture"] = {"state": "verified", "source": source, "sha256": digest} + event["delivery"] = {"state": "visible" if visible else "unknown", + "output_state": "verified" if return_line is not None else "unknown", + "visible_resources": visible, + "passages": _capture_passages(doc, selected_passages, visible_keys)} + if return_line is not None: + event["trace_line"] = return_line + if visible: + event["delivery"]["return_line"] = return_line + return event diff --git a/src/inferrsi/harness/codex_search.py b/src/inferrsi/harness/codex_search.py index afff785a..0fa0daa0 100644 --- a/src/inferrsi/harness/codex_search.py +++ b/src/inferrsi/harness/codex_search.py @@ -1,4 +1,4 @@ -"""Extract file search observations from a caller-selected Codex Desktop rollout window.""" +"""Extract bounded, attributable file-search evidence from Codex rollouts.""" from __future__ import annotations @@ -7,12 +7,18 @@ import re import shlex from pathlib import Path -from urllib.parse import urlsplit -from urllib.request import url2pathname from .campaign_records import _identifier +from .codex_tool_returns import _path, correlate_window from .dashboard_data import _relative_location, _text +_SHELLS = ("sh", "bash", "zsh", "dash") +_TOOLS = ("rg", "cat", "sed", "nl") +_MAX_PASSAGES = 12 +_MAX_PASSAGE_CHARS = 400 +_MAX_VISIBLE_REFS = 64 +_MAX_UNRESOLVED_PATH_EXCERPTS = 4 + def _material(cwd: Path, name: str, corpus: Path) -> str | None: # The original tree may be offline; use the recorded working directory. @@ -69,20 +75,20 @@ def _rg_args(args: list[str]) -> tuple[str, list[str], str, list[str]] | None: return (" | ".join(patterns), operands or ["."], mode, options) if patterns else None -def _rg_hits(output, cwd, corpus, scopes, mode): - # Without numbers, "file-context: text" also looks like a match in - # "file-context". Retain the search event but withhold hit attribution. +def _rg_observations(output, cwd, corpus, scopes, mode): + """Return parsed rg rows with their aggregate-output line indexes.""" if mode == "unnumbered_context": return [], False - hits = {} - scopes = [Path(os.path.normpath(cwd / s)) for s in scopes] + observations = [] + scope_paths = [Path(os.path.normpath(cwd / scope)) for scope in scopes] complete = True - for line in output.splitlines(): + for index, raw_line in enumerate(output.splitlines()): + line = raw_line.rstrip("\r") if "tokens truncated" in line and ("…" in line or "..." in line): complete = False continue if mode == "files": - path, number = line, None + path, number, text = line, None, None else: if mode == "numbered_context": numbered_match = re.match(r"^(?:[^:]+:)?\d+:", line) @@ -97,35 +103,47 @@ def _rg_hits(output, cwd, corpus, scopes, mode): if not numbered_match: complete = False continue - # For a single explicit file, rg -n omits the filename. Do not stat - # the live corpus: the retained rollout may outlive that checkout. - bare = re.match(r"^(\d+):", line) - if bare and len(scopes) == 1 and scopes[0] != cwd: - path, number = str(scopes[0]), int(bare[1]) + bare = re.match(r"^(\d+):(.*)$", line) + if bare and len(scope_paths) == 1 and scope_paths[0] != cwd: + path, number, text = str(scope_paths[0]), int(bare[1]), bare[2] else: - match = re.match(r"^([^:]+):(?:(\d+):)?", line) + match = re.match(r"^([^:]+):(?:(\d+):)?(.*)$", line) if not match: complete = False continue - path, number = match[1], int(match[2]) if match[2] else None + path = match[1] + number = int(match[2]) if match[2] else None + text = match[3] candidate = Path(os.path.normpath(cwd / path)) - if not any(candidate.is_relative_to(scope) for scope in scopes): + if not any(candidate.is_relative_to(scope) for scope in scope_paths): complete = False continue ref = _material(cwd, str(candidate), corpus) if ref: - hits.setdefault(ref, number) + observations.append({"ref": ref, "line": number, "text": text, + "output_line": index}) elif candidate.is_relative_to(corpus): complete = False - return [{"ref": ref, "match_line": n} for ref, n in hits.items()], complete + return observations, complete + + +def _rg_hits(output, cwd, corpus, scopes, mode): + observations, complete = _rg_observations(output, cwd, corpus, scopes, mode) + hits = {} + for observation in observations: + hits.setdefault(observation["ref"], observation["line"]) + return [{"ref": ref, "match_line": line} for ref, line in hits.items()], complete def _command(item, corpus): - argv = item.get("command") - if not isinstance(argv, list) or not argv or not all(isinstance(a, str) for a in argv): - return None, False + """Decode one simple file command without changing its legacy event fields.""" + original_argv = item.get("command") + if (not isinstance(original_argv, list) or not original_argv + or not all(isinstance(arg, str) for arg in original_argv)): + return None, False, None + argv = original_argv multiline = False - if (Path(argv[0]).name in ("sh", "bash", "zsh", "dash") and len(argv) == 3 + if (Path(argv[0]).name in _SHELLS and len(argv) == 3 and argv[1] in ("-c", "-lc", "-ic", "-ilc")): multiline = "\n" in argv[2].strip() try: @@ -133,82 +151,397 @@ def _command(item, corpus): lexer.whitespace_split = True argv = list(lexer) except ValueError: - return None, False + return None, False, None if not argv: - return None, True + return None, True, None tool = Path(argv[0]).name - if multiline or tool not in ("rg", "cat", "sed", "nl"): + if multiline or tool not in _TOOLS: # A compound script can run several tools against one shared stdout. - return None, not any(Path(word).name in ("rg", "cat", "sed", "nl") for word in argv) - cwd = item.get("cwd") - if isinstance(cwd, str) and cwd.startswith("file://"): - uri = urlsplit(cwd) - cwd = url2pathname(uri.path) if uri.netloc in ("", "localhost") else None - if not isinstance(cwd, str) or not Path(cwd).is_absolute(): - return None, False - cwd = Path(os.path.normpath(cwd)) + return None, not any(Path(word).name in _TOOLS for word in argv), None + cwd = _path(item.get("cwd")) + if cwd is None: + return None, False, None if any(word in {"|", "||", "&&", ";", ">", ">>", "<"} for word in argv): - return None, False + return None, False, None code, output = item.get("exit_code"), item.get("aggregated_output") if not isinstance(output, str) or type(code) is not int: - return None, False + return None, False, None if tool == "rg": parsed = _rg_args(argv[1:]) if parsed is None: - return None, False + return None, False, None query, scopes, mode, options = parsed if mode == "inventory": - return None, True - if any(re.search(r"[$`*?~{}\[]", value) for value in scopes): - return None, False # literal command text cannot resolve expanded operands + return None, True, None + if any(re.search(r"[$\x60*?~{}\[]", value) for value in scopes): + return None, False, None paths = [Path(os.path.normpath(cwd / value)) for value in scopes] - scope_refs = ["." if p == corpus or corpus.is_relative_to(p) - else _material(cwd, str(p), corpus) for p in paths] + scope_refs = ["." if path == corpus or corpus.is_relative_to(path) + else _material(cwd, str(path), corpus) for path in paths] if not any(scope_refs): - return None, True + return None, True, None materials, complete = [], code == 1 if code == 0: materials, complete = _rg_hits(output, cwd, corpus, scopes, mode) event = {"action": "search", "tool": tool, "query": _text(query), - "scope": [s for s in scope_refs if s], "options": options, + "scope": [scope for scope in scope_refs if scope], "options": options, "exit_code": code, "materials": materials} - return event, complete and (code != 0 or bool(materials)) + spec = {"kind": "search", "tool": tool, "cwd": cwd, "scopes": scopes, + "mode": mode} + return event, complete and (code != 0 or bool(materials)), spec if tool == "cat": names = argv[2:] if argv[1:2] == ["--"] else argv[1:] - if any(n.startswith("-") for n in names): - return None, False + if any(name.startswith("-") for name in names): + return None, False, None elif len(argv) == 4 and tool == "sed" and argv[1] == "-n" and re.fullmatch( r"\d+(?:,\d+)?p", argv[2]): names = argv[3:] elif tool == "nl" and len(argv) == 3 and argv[1] == "-ba": names = argv[2:] else: - return None, False - if any(re.search(r"[$`*?~{}\[]", n) for n in names): - return None, False - refs = [ref for n in names if (ref := _material(cwd, n, corpus))] + return None, False, None + if any(re.search(r"[$\x60*?~{}\[]", name) for name in names): + return None, False, None + refs = [ref for name in names if (ref := _material(cwd, name, corpus))] if not refs: - return None, True + return None, True, None if code != 0: - return None, False - # Successful cat opened every literal operand. Its shared stdout cannot - # establish which file supplied a particular passage. - return {"action": "read", "tool": tool, - "materials": [{"ref": ref} for ref in dict.fromkeys(refs)]}, True + return None, False, None + # Successful cat of multiple files has one joined stdout, so no excerpt + # can be assigned to an individual source. + event = {"action": "read", "tool": tool, + "materials": [{"ref": ref} for ref in dict.fromkeys(refs)]} + spec = {"kind": "read", "tool": tool, "refs": list(dict.fromkeys(refs))} + if len(names) == 1 and len(refs) == 1: + spec["source_ref"] = refs[0] + if tool == "sed": + selected = argv[2][:-1].split(",") + spec["line_start"] = int(selected[0]) + spec["line_end"] = int(selected[-1]) + elif tool == "cat": + spec["line_start"] = 1 + else: + spec["line_mode"] = "numbered" + return event, True, spec + + +def _archive_text(value, corpus, limit=_MAX_PASSAGE_CHARS): + text = value if isinstance(value, str) else "" + normalized_root = False + placeholder = "NORMALIZEDKNOWLEDGEROOTTOKEN" + roots = (corpus.as_posix(), str(corpus)) + for root in dict.fromkeys(roots): + if root and root in text: + text = text.replace(root, placeholder) + normalized_root = True + preview = _text(text, "", limit + 1) + redacted = preview != text[:limit + 1] + truncated = len(preview) > limit + fields = {"text": preview[:limit].replace(placeholder, "")} + if redacted: + fields["redacted"] = True + if normalized_root: + fields["root_normalized"] = True + if truncated: + fields["truncated"] = True + return fields + + +def _passage(ref, line, text, corpus): + safe = _archive_text(text, corpus) + return {"ref": ref, "line": line, **safe} + +def _unique_candidates(candidates): + result, seen = [], set() + for candidate in candidates: + key = (candidate["ref"], candidate["line"], candidate["text"]) + if key not in seen: + seen.add(key) + result.append(candidate) + return result -def read_window(trace: Path, lines: tuple[int, int], corpus: Path): - """Return only attributed events and decision locators, never the raw transcript.""" + +def _select_passages(candidates, event_line, selections): + candidates = _unique_candidates(candidates) + requested = set(selections.get(event_line, ())) + selected = [] + found = set() + for index, candidate in enumerate(candidates): + locator = (candidate["ref"], candidate["line"]) + if locator in requested: + found.add(locator) + if index < _MAX_PASSAGES or locator in requested: + selected.append(candidate) + missing = [key for key in selections.get(event_line, ()) if key not in found] + return selected, len(candidates) - len(selected), missing + + +def _delivery(event, spec, join, corpus, event_line, selections): + base = {"state": "unknown", "runtime_truncated": join.get("runtime_truncated"), + "archive_omitted": True, "passages": [], "visible_refs": []} + issues = [] + if join.get("state") != "visible": + return base, join.get("reason", "output_not_verified"), issues + if spec["kind"] == "read" and "source_ref" not in spec: + base["runtime_truncated"] = join.get("runtime_truncated") + return base, "read_sources_not_separable", issues + aggregate_lines = join["aggregate_lines"] + visible_indexes = set(join["line_indexes"]) + candidates, refs = [], [] + if spec["kind"] == "search": + observations, _ = _rg_observations( + "".join(aggregate_lines), spec["cwd"], corpus, spec["scopes"], spec["mode"] + ) + for observation in observations: + if observation["output_line"] not in visible_indexes: + continue + if observation["ref"] not in refs: + refs.append(observation["ref"]) + if observation["text"] is None: + continue + candidates.append({"ref": observation["ref"], "line": observation["line"], + "text": observation["text"]}) + else: + ref = spec["source_ref"] + refs.append(ref) + marker_indexes = [ + index for index, output_line in enumerate(aggregate_lines) + if ("tokens truncated" in output_line + and ("…" in output_line or "..." in output_line)) + ] + truncation_index = min(marker_indexes) if marker_indexes else None + for index in sorted(visible_indexes): + if index >= len(aggregate_lines): + continue + # A cut marker in a read stream means indexes after it no longer + # establish source line numbers. + if truncation_index is not None and index >= truncation_index: + continue + text = aggregate_lines[index].rstrip("\r\n") + if spec.get("line_mode") == "numbered": + match = re.match(r"^\s*(\d+)\t(.*)$", text) + if not match: + continue + line_number, text = int(match[1]), match[2] + else: + line_number = spec["line_start"] + index + candidates.append({"ref": ref, "line": line_number, "text": text}) + passages, omitted, unmatched = _select_passages(candidates, event_line, selections) + safe_passages = [_passage(item["ref"], item["line"], item["text"], corpus) + for item in passages] + visible_refs = refs[:_MAX_VISIBLE_REFS] + omitted_refs = max(0, len(refs) - len(visible_refs)) + truncated_passages = sum(item.get("truncated", False) for item in safe_passages) + redacted_passages = sum(item.get("redacted", False) for item in safe_passages) + normalized_passages = sum(item.get("root_normalized", False) for item in safe_passages) + base.update({ + "state": "visible", + "return_line": join["return_line"], + "archive_omitted": bool(omitted or omitted_refs or truncated_passages), + "passages": safe_passages, + "visible_refs": visible_refs, + }) + for key, count in ( + ("omitted_passages", omitted), ("omitted_refs", omitted_refs), + ("truncated_passages", truncated_passages), + ("redacted_passages", redacted_passages), + ("root_normalized_passages", normalized_passages), + ): + if count: + base[key] = count + for ref, line in unmatched: + issues.append({"reason": "requested_passage_not_visible", + "selection": {"ref": ref, "line": line}}) + return base, None, issues + + +def _rough_action(item): + argv = item.get("command") + if not isinstance(argv, list) or not argv: + return "unknown", False + tokens = argv + if (isinstance(argv[0], str) and Path(argv[0]).name in _SHELLS + and len(argv) == 3 and argv[1] in ("-c", "-lc", "-ic", "-ilc")): + try: + lexer = shlex.shlex(argv[2], posix=True, punctuation_chars=";&|<>") + lexer.whitespace_split = True + tokens = list(lexer) + except ValueError: + return "unknown", False + found = [Path(word).name for word in tokens if isinstance(word, str) + and Path(word).name in _TOOLS] + compound = any(word in {"|", "||", "&&", ";", ">", ">>", "<"} for word in tokens) + if compound: + return "unknown", True + if found: + return ("search" if found[0] == "rg" else "read"), False + return "unknown", False + + +def _safe_file_list_pipeline(item, corpus, join): + """Retain samples only for a simple scoped rg file-list/filter pipeline.""" + if join.get("state") != "visible": + return None + argv = item.get("command") + if (not isinstance(argv, list) or len(argv) != 3 + or Path(argv[0]).name not in ("sh", "bash", "zsh", "dash") + or argv[1] not in ("-c", "-lc", "-ic", "-ilc") + or "\n" in argv[2]): + return None + try: + lexer = shlex.shlex(argv[2], posix=True, punctuation_chars=";&|<>") + lexer.whitespace_split = True + tokens = list(lexer) + except ValueError: + return None + if (len(tokens) == 6 and tokens[:2] == ["rg", "--files"] + and tokens[3:5] == ["|", "rg"]): + pattern, ignore_case = tokens[5], False + elif (len(tokens) == 7 and tokens[:2] == ["rg", "--files"] + and tokens[3:5] == ["|", "rg"] and tokens[5] in ("-i", "--ignore-case")): + pattern, ignore_case = tokens[6], True + else: + return None + if not pattern or pattern.startswith("-") or len(pattern) > 120: + return None + cwd = _path(item.get("cwd")) + if cwd is None: + return None + operand = tokens[2] + if re.search(r"[$\x60*?~{}\[]", operand): + return None + scope = Path(os.path.normpath(cwd / operand)) + if not scope.is_relative_to(corpus): + return None + scope_ref = "." if scope == corpus else _material(cwd, str(scope), corpus) + if scope_ref is None: + return None + pattern_text = _text(pattern, "", 120) + if not pattern_text: + return None + display_scope = "" + (f"/{scope_ref}" if scope_ref != "." else "") + flag = " -i" if ignore_case else "" + rendered_command = ( + f"rg --files {display_scope} | rg{flag} {shlex.quote(pattern_text)}" + ) + command_truncated = len(rendered_command) > 240 + command_excerpt = rendered_command[:237] + "..." if command_truncated else rendered_command + lines = join["aggregate_lines"] + visible_indexes = set(join["line_indexes"]) + paths, seen = [], set() + for index in sorted(visible_indexes): + if index >= len(lines): + continue + text = lines[index].rstrip("\r\n") + if not text or ("tokens truncated" in text and ("…" in text or "..." in text)): + continue + candidate_path = _path(text) or Path(os.path.normpath(cwd / text)) + if not candidate_path.is_relative_to(scope): + continue + ref = _material(cwd, str(candidate_path), corpus) + if ref is None or ref in seen: + continue + seen.add(ref) + paths.append({"ref": ref, "line": None, "text": text}) + excerpts = [_passage(item["ref"], None, item["text"], corpus) + for item in paths[:_MAX_UNRESOLVED_PATH_EXCERPTS]] + omitted = max(0, len(paths) - len(excerpts)) + return { + "scope": scope_ref, "command_excerpt": command_excerpt, + "excerpts": excerpts, "omitted": omitted, + "pattern_redacted": pattern_text != pattern, + "scope_normalized": Path(operand).is_absolute(), + "command_truncated": command_truncated, + } + + +def _operation_skeleton(item, action, event=None): + if event: + refs = list(event.get("scope", [])) + refs.extend(material.get("ref") for material in event.get("materials", [])) + return {"tool": event.get("tool"), "action": action, + "scope": list(dict.fromkeys(refs))[:_MAX_VISIBLE_REFS]} + argv = item.get("command") + if not isinstance(argv, list) or not argv: + return {"action": action} + tokens = argv + if (isinstance(argv[0], str) and Path(argv[0]).name in _SHELLS + and len(argv) == 3 and argv[1] in ("-c", "-lc", "-ic", "-ilc")): + try: + lexer = shlex.shlex(argv[2], posix=True, punctuation_chars=";&|<>") + lexer.whitespace_split = True + tokens = list(lexer) + except ValueError: + return {"action": action} + tools = list(dict.fromkeys( + Path(word).name for word in tokens + if isinstance(word, str) and Path(word).name in _TOOLS + )) + compound = any(word in {"|", "||", "&&", ";", ">", ">>", "<"} for word in tokens) + if compound: + return {"tools": tools, "action": "unknown", "compound": True} + if tools: + return {"tool": tools[0], "action": action, "scope": []} + return {"action": action} + + +def _detail(event_id, trace_line, action, reason, event=None, excerpt=None, selection=None, + command_item=None): + descriptions = { + "trace_scan_incomplete": "Trace scan incomplete; verify the selected lines manually.", + "command_not_unique": "Multiple same-key commands are covered; narrow the selected window.", + "outer_call_not_unique": "No unique enclosing call; narrow the selected window.", + "outer_result_not_unique": "No unique result; inspect tool-return rows in the window.", + "outer_result_has_no_text": "No text is available to verify an excerpt.", + "output_missing_or_too_large": "Output is absent or exceeds the comparison cap.", + "output_has_too_many_lines": "Output exceeds the line comparison cap.", + "output_bytes_do_not_match": "Output differs from the command record; don't quote it.", + "no_exact_output_lines": "No exact returned lines match the command record.", + "read_sources_not_separable": "Files were concatenated; source paths cannot be assigned.", + "requested_passage_not_visible": "Requested line is absent; choose a returned passage.", + "unsupported_or_compound_command": "Inspect the operation and return rows before citing.", + "command_output_incomplete": "Output is incomplete; cite only retained lines.", + } + detail = {"trace_line": trace_line, "reason": reason, + "action": action, "description": descriptions.get( + reason, "File output could not be attributed safely; inspect the selected trace." + )} + if event_id: + detail["id"] = event_id + detail["operation"] = _operation_skeleton(command_item or {}, action, event) + if excerpt: + detail["excerpt"] = excerpt + if selection: + detail["selection"] = selection + return detail + + +def read_window(trace: Path, lines: tuple[int, int], corpus: Path, *, archive=False, + selections=None): + """Return window, events, full transient decisions, uncertainty and detail. + + In archive mode, selections maps a command event's trace line to + [(corpus_relative_ref, source_line_or_None), ...]. Requested lines are + retained alongside the default first 12 passages per event. Each selected + line must be present in the model-facing tool return. + """ first, last = lines if first < 1 or last < first or not corpus.is_absolute(): raise ValueError("expected a positive inclusive line range and absolute knowledge root") corpus = Path(os.path.normpath(corpus)) - events, decisions, seen, uncertain = [], set(), set(), [] + selected_passages = selections or {} + events, decisions, uncertain, unresolved_details = [], {}, [], [] + commands, seen = [], set() with trace.open("rb") as stream: meta = json.loads(stream.readline()) if not isinstance(meta, dict) or meta.get("type") != "session_meta": raise ValueError("expected Codex session_meta in the first trace line") - session = _identifier(meta.get("payload", {}).get("id"), "session ID") + meta_payload = meta.get("payload", {}) + if not isinstance(meta_payload, dict): + raise ValueError("expected Codex session metadata payload") + session = _identifier(meta_payload.get("id"), "session ID") + meta_cwd = _path(meta_payload.get("cwd")) number = 1 while number < last: raw = stream.readline() @@ -219,24 +552,115 @@ def read_window(trace: Path, lines: tuple[int, int], corpus: Path): continue row = json.loads(raw) payload = row.get("payload", {}) + if not isinstance(payload, dict): + continue if (row.get("type") == "response_item" and payload.get("type") == "message" and payload.get("role") == "assistant" - and payload.get("phase") in ("commentary", "final_answer") - and any(isinstance(part, dict) and part.get("text") - for part in payload.get("content", []))): - decisions.add(number) + and payload.get("phase") in ("commentary", "final_answer")): + content = payload.get("content", []) + text = "".join(part.get("text", "") for part in content + if isinstance(part, dict) and isinstance(part.get("text"), str)) + if text: + # Full strings are transient quote-validation input. Callers + # must select a reviewed quote before serialization. + decisions[number] = text if row.get("type") != "event_msg" or payload.get("type") != "item_completed": continue item = payload.get("item", {}) - if item.get("type") != "CommandExecution": + if not isinstance(item, dict) or item.get("type") != "CommandExecution": continue event_id = _identifier(item.get("id"), "command event ID") if event_id in seen: continue seen.add(event_id) - event, complete = _command(item, corpus) - if event: + event, complete, spec = _command(item, corpus) + commands.append((number, item, event_id, event, complete, spec)) + if event is not None: events.append(dict(event, trace_line=number, event_id=event_id)) if not complete: uncertain.append(number) - return {"session": session, "lines": [first, last]}, events, decisions, uncertain + window = {"session": session, "lines": [first, last]} + if archive: + window["archive"] = True + joinable = [(line, item, event_id) for line, item, event_id, _, _, _ in commands] + joins = correlate_window(trace, joinable, meta_cwd, (first, last)) + event_by_line = {event["trace_line"]: event for event in events} + for line, item, event_id, event, complete, spec in commands: + if event is None: + if not complete: + action, compound = _rough_action(item) + detail = _detail(f"file-{line}", line, action, + "unsupported_or_compound_command", + command_item=item) + if compound: + detail["description"] = ( + "Compound file operation; review its call and result before citing." + ) + safe_list = _safe_file_list_pipeline( + item, corpus, joins.get(event_id, {"state": "unknown"}) + ) + if safe_list: + detail["description"] = ( + "Samples are filenames only; no content or match count is attributed." + ) + detail["operation"] = { + "tools": ["rg", "rg"], "action": "file_list_filter", + "scope": [safe_list["scope"]], "compound": True, + } + detail["command_excerpt"] = safe_list["command_excerpt"] + detail["excerpt"] = safe_list["excerpts"] + detail["return_line"] = joins[event_id]["return_line"] + detail["runtime_truncated"] = joins[event_id]["runtime_truncated"] + if safe_list["pattern_redacted"]: + detail["command_redacted"] = True + if safe_list["scope_normalized"]: + detail["command_root_normalized"] = True + if safe_list["command_truncated"]: + detail["command_excerpt_truncated"] = True + excerpt_truncated = sum( + passage.get("truncated", False) for passage in safe_list["excerpts"] + ) + if safe_list["omitted"] or excerpt_truncated: + detail["archive_omitted"] = True + unresolved_details.append(detail) + continue + public_event = event_by_line[line] + public_id = f"file-{line}" + public_event["id"] = public_id + public_event["event_id"] = public_id + join = joins.get(event_id, {"state": "unknown", "runtime_truncated": None, + "reason": "outer_call_not_unique"}) + delivery, problem, selection_issues = _delivery( + public_event, spec, join, corpus, line, selected_passages + ) + public_event["delivery"] = delivery + for issue in selection_issues: + unresolved_details.append(_detail( + public_id, line, public_event["action"], issue["reason"], public_event, + selection=issue.get("selection"), + )) + reason = None + if not complete: + reason = "command_output_incomplete" + elif problem: + reason = problem + elif delivery["runtime_truncated"]: + reason = "command_output_incomplete" + elif delivery["state"] == "unknown": + reason = join.get("reason", "output_missing_or_too_large") + if (delivery["state"] == "unknown" or not complete + or delivery["runtime_truncated"]): + if line not in uncertain: + uncertain.append(line) + if reason: + excerpt = delivery["passages"][0] if delivery["passages"] else None + unresolved_details.append(_detail( + public_id, line, public_event["action"], reason, public_event, excerpt + )) + event_lines = set(event_by_line) + for event_line, requested in selected_passages.items(): + if event_line not in event_lines and requested: + unresolved_details.append(_detail( + None, event_line, "unknown", "requested_passage_not_visible" + )) + return window, events, decisions, uncertain, unresolved_details diff --git a/src/inferrsi/harness/codex_tool_returns.py b/src/inferrsi/harness/codex_tool_returns.py new file mode 100644 index 00000000..bbf765de --- /dev/null +++ b/src/inferrsi/harness/codex_tool_returns.py @@ -0,0 +1,261 @@ +"""Correlate Codex command events with exact outer tool call/result envelopes.""" + +from __future__ import annotations + +import json +import os +import re +import shlex +from collections import Counter +from pathlib import Path +from urllib.parse import urlsplit +from urllib.request import url2pathname + +_SHELLS = ("sh", "bash", "zsh", "dash") +_MAX_JOIN_CHARS = 1_000_000 +_MAX_JOIN_LINES = 8_000 + + +def _path(value: object) -> Path | None: + if not isinstance(value, str) or not value: + return None + if value.startswith("file://"): + uri = urlsplit(value) + if uri.netloc not in ("", "localhost"): + return None + value = url2pathname(uri.path) + path = Path(os.path.normpath(value)) + return path if path.is_absolute() else None + + +def _command_key(item): + argv = item.get("command") + if not isinstance(argv, list) or not argv or not all(isinstance(arg, str) for arg in argv): + return None + if (Path(argv[0]).name in _SHELLS and len(argv) == 3 + and argv[1] in ("-c", "-lc", "-ic", "-ilc")): + return "shell", argv[2] + return "argv", tuple(argv) + + +def _signature(item): + cwd = _path(item.get("cwd")) + key = _command_key(item) + code, output = item.get("exit_code"), item.get("aggregated_output") + if cwd is None or key is None or type(code) is not int or not isinstance(output, str): + return None + return cwd.as_posix(), key, code, output + + +def _js_string_property(source, name): + pattern = (r"(?:\{\s*|,\s*)" + re.escape(name) + + r"\s*:\s*(\"(?:\\.|[^\"\\])*\")") + matches = re.findall(pattern, source) + if len(matches) != 1: + return None, False + try: + value = json.loads(matches[0]) + except json.JSONDecodeError: + return None, False + return (value, True) if isinstance(value, str) else (None, False) + + +def _outer_call_spec(row, meta_cwd): + payload = row.get("payload", {}) + if not isinstance(payload, dict): + return None + payload_type, name = payload.get("type"), payload.get("name") + command = workdir = None + if payload_type == "custom_tool_call" and name in ("exec", "exec_command"): + source = payload.get("input") + if not isinstance(source, str): + return None + command, valid = _js_string_property(source, "cmd") + if not valid: + command, valid = _js_string_property(source, "command") + if not valid: + return None + workdir, workdir_set = _js_string_property(source, "workdir") + if not workdir_set and re.search(r"(?:\{\s*|,\s*)workdir\s*:", source): + return None + if not workdir_set: + workdir = None + elif payload_type == "function_call" and name in ("exec", "exec_command"): + arguments = payload.get("arguments", payload.get("input")) + if isinstance(arguments, str): + try: + arguments = json.loads(arguments) + except json.JSONDecodeError: + return None + if not isinstance(arguments, dict): + return None + command = arguments.get("cmd", arguments.get("command")) + workdir = arguments.get("workdir", arguments.get("cwd")) + if not isinstance(command, str): + if (isinstance(command, list) and command + and all(isinstance(arg, str) for arg in command)): + command = shlex.join(command) + else: + return None + else: + return None + cwd = _path(workdir) if workdir is not None else meta_cwd + call_id = payload.get("call_id") + if cwd is None or not isinstance(command, str) or not isinstance(call_id, str) or not call_id: + return None + keys = {("shell", command)} + try: + keys.add(("argv", tuple(shlex.split(command)))) + except ValueError: + pass + return {"call_id": call_id, "cwd": cwd.as_posix(), "keys": keys} + + +def _response_text(row): + payload = row.get("payload", {}) + value = payload.get("output") if isinstance(payload, dict) else None + if isinstance(value, str): + joined = value + elif isinstance(value, list): + joined = "".join( + part.get("text", "") for part in value + if isinstance(part, dict) and part.get("type") == "input_text" + and isinstance(part.get("text"), str) + ) + else: + return None + return joined + + +def _truncation_marker(aggregate, response): + for line in aggregate.splitlines(): + if "tokens truncated" in line and ("…" in line or "..." in line): + return True + aggregate_lines = set(aggregate.splitlines()) + for line in response.splitlines(): + if (re.match(r"^\s*(?:Warning:\s*)?truncated output\b", line, re.I) + and line not in aggregate_lines): + return True + if re.match(r"^\s*\[output truncated\b", line, re.I): + return True + return False + + +def _match_output_lines(aggregate, response): + """Map only exact aggregate lines present in the outer call result.""" + # Some producers prepend a tool header. Only remove a known envelope, + # and leave an exact source body containing its own "Output:" intact. + if response != aggregate: + header = re.match(r"^(?:Script completed\nWall time[^\n]*\n)?Output:\n", response) + if header: + response = response[header.end():] + if max(len(aggregate), len(response)) > _MAX_JOIN_CHARS: + return {}, None, "output_missing_or_too_large" + source = aggregate.splitlines(keepends=True) + returned = response.splitlines(keepends=True) + if max(len(source), len(returned)) > _MAX_JOIN_LINES: + return {}, None, "output_has_too_many_lines" + truncated = _truncation_marker(aggregate, response) + if aggregate == response: + return {i: i for i, line in enumerate(source) if line.strip()}, truncated, None + if not truncated: + return {}, None, "output_bytes_do_not_match" + + # A truncated read can contain identical paragraphs from different + # positions. Only a unique complete row can identify its source line. + source_counts, return_counts = Counter(source), Counter(returned) + positions = {line: i for i, line in enumerate(returned) if return_counts[line] == 1} + indices = {i: positions[line] for i, line in enumerate(source) + if line.strip() and source_counts[line] == 1 and line in positions} + return indices, True, None if indices else "no_exact_output_lines" + + +def correlate_window(trace, commands, meta_cwd, window_lines): + """Join each command to one exact enclosing call/result pair in this window.""" + first, last = window_lines + command_rows, calls, outputs = [], [], {} + scan_complete = True + with trace.open("rb") as stream: + stream.readline() + for line_number, raw in enumerate(stream, 2): + if line_number < first: + continue + if line_number > last: + break + try: + row = json.loads(raw) + except (json.JSONDecodeError, UnicodeDecodeError): + scan_complete = False + continue + payload = row.get("payload", {}) + if (row.get("type") == "event_msg" and isinstance(payload, dict) + and payload.get("type") == "item_completed"): + item = payload.get("item", {}) + if isinstance(item, dict) and item.get("type") == "CommandExecution": + command_rows.append((line_number, item)) + if row.get("type") != "response_item" or not isinstance(payload, dict): + continue + call = _outer_call_spec(row, meta_cwd) + if call: + call["line"] = line_number + calls.append(call) + outputs.setdefault(call["call_id"], []) + if (payload.get("type") in ("custom_tool_call_output", "function_call_output") + and payload.get("call_id") in outputs): + outputs[payload["call_id"]].append((line_number, row)) + calls_by_id = {} + for call in calls: + calls_by_id.setdefault(call["call_id"], []).append(call) + joins = {} + for event_line, item, event_id in commands: + signature = _signature(item) + if signature is None or not scan_complete: + joins[event_id] = {"state": "unknown", "runtime_truncated": None, + "reason": "trace_scan_incomplete"} + continue + matching, ambiguous_command = [], False + for call in calls: + if (call["line"] > event_line or call["cwd"] != signature[0] + or signature[1] not in call["keys"]): + continue + if len(calls_by_id.get(call["call_id"], [])) != 1: + continue + call_outputs = outputs.get(call["call_id"], []) + if len(call_outputs) != 1: + continue + return_line, output_row = call_outputs[0] + if return_line < event_line: + continue + covered = [] + for line, candidate in command_rows: + cwd = _path(candidate.get("cwd")) + if (call["line"] <= line <= return_line and cwd is not None + and cwd.as_posix() == signature[0] + and _command_key(candidate) == signature[1]): + covered.append(line) + if len(covered) == 1 and covered[0] == event_line: + matching.append((return_line, output_row)) + elif event_line in covered: + ambiguous_command = True + if len(matching) != 1: + reason = "command_not_unique" if ambiguous_command else "outer_call_not_unique" + joins[event_id] = {"state": "unknown", "runtime_truncated": None, + "reason": reason} + continue + return_line, output_row = matching[0] + response = _response_text(output_row) + if response is None: + joins[event_id] = {"state": "unknown", "runtime_truncated": None, + "reason": "outer_result_has_no_text"} + continue + indexes, truncated, reason = _match_output_lines(signature[3], response) + if reason is not None: + joins[event_id] = {"state": "unknown", "runtime_truncated": truncated, + "reason": reason} + continue + joins[event_id] = { + "state": "visible", "runtime_truncated": truncated, + "line_indexes": indexes, "aggregate_lines": signature[3].splitlines(keepends=True), + "return_line": return_line, + } + return joins diff --git a/src/inferrsi/harness/search_archive.py b/src/inferrsi/harness/search_archive.py new file mode 100644 index 00000000..58f85760 --- /dev/null +++ b/src/inferrsi/harness/search_archive.py @@ -0,0 +1,154 @@ +"""Validate the portable review input and its source-to-decision associations.""" + +import json +import re +from pathlib import Path + +from .campaign_records import _identifier +from .dashboard_data import _relative_location, _text +from .optimization_log import _unique as _unique_pairs + +_REVISION = re.compile(r"[A-Fa-f0-9]{40}|[A-Fa-f0-9]{64}") + + +def read_archive_review(path): + """Read and validate the caller-reviewed portable attribution input.""" + path = Path(path) + try: + raw = path.read_bytes() + except OSError as exc: + raise ValueError("archive review is unavailable") from exc + try: + document = json.loads(raw.decode("utf-8"), object_pairs_hook=_unique_pairs) + except (UnicodeError, ValueError, RecursionError) as exc: + raise ValueError("archive review must be valid UTF-8 JSON") from exc + if not isinstance(document, dict): + raise ValueError("archive review must be a JSON object") + required = {"task", "corpus", "decisions"} + allowed = required | {"capture_returns", "result_note"} + if required - set(document) or set(document) - allowed: + raise ValueError("archive review is missing required fields or contains unknown fields") + task = document["task"] + if not isinstance(task, str) or not task.strip(): + raise ValueError("archive review task must be nonempty text") + # The task is caller-reviewed prose, but still redact values forbidden by + # the repository's existing display sanitizer before making it portable. + task = _text(task, "Not recorded", limit=len(task)) + corpus = document["corpus"] + if not isinstance(corpus, dict) or set(corpus) != {"repository", "revision", "root"}: + raise ValueError("archive review corpus must declare repository, revision and root") + repository, revision, corpus_root = (corpus.get("repository"), corpus.get("revision"), + corpus.get("root")) + repo_parts = repository.split("/") if isinstance(repository, str) else [] + if len(repo_parts) != 2: + raise ValueError("archive review corpus.repository must be owner/repository") + try: + _identifier(repo_parts[0], "archive review repository owner") + _identifier(repo_parts[1], "archive review repository name") + except ValueError as exc: + raise ValueError("archive review corpus.repository must be owner/repository") from exc + if not isinstance(revision, str) or not _REVISION.fullmatch(revision): + raise ValueError("archive review corpus.revision must be a full commit SHA") + if (not isinstance(corpus_root, str) or not corpus_root + or (corpus_root != "." and _relative_location(corpus_root) != corpus_root)): + raise ValueError("archive review corpus.root must be a portable repository-relative path") + corpus = {"repository": repository, "revision": revision.lower(), "root": corpus_root} + + raw_decisions = document["decisions"] + if not isinstance(raw_decisions, list): + raise ValueError("archive review decisions must be a list") + decision_fields = {"use", "ref", "event_line", "passage_line", "decision_line", + "decision_quote"} + for index, item in enumerate(raw_decisions): + if not isinstance(item, dict) or set(item) not in ( + decision_fields, decision_fields | {"passage_path"}): + raise ValueError(f"archive review decisions[{index}] has unsupported fields") + if item["use"] not in ("used", "rejected"): + raise ValueError(f"decisions[{index}].use must be used or rejected") + ref = item["ref"] + if not isinstance(ref, str) or _relative_location(ref) != ref: + raise ValueError(f"decisions[{index}].ref must be a portable source ref") + for key in ("event_line", "decision_line"): + if type(item[key]) is not int or item[key] < 1: + raise ValueError(f"decisions[{index}].{key} must be a positive line") + passage_line = item["passage_line"] + if passage_line is not None and (type(passage_line) is not int or passage_line < 1): + raise ValueError(f"decisions[{index}].passage_line must be positive or null") + quote = item["decision_quote"] + if not isinstance(quote, str) or not quote.strip(): + raise ValueError(f"decisions[{index}].decision_quote must be nonempty text") + # A quote is an exact evidence locator. If the archive sanitizer would + # alter it, it cannot be presented as an exact quote in a public record. + if _text(quote, "", limit=len(quote)) != quote: + raise ValueError(f"decisions[{index}].decision_quote is not portable") + if "passage_path" in item: + passage_path = item["passage_path"] + if (not isinstance(passage_path, str) + or _relative_location(passage_path) != passage_path): + raise ValueError(f"decisions[{index}].passage_path must be portable") + + capture_returns = document.get("capture_returns", {}) + if not isinstance(capture_returns, dict): + raise ValueError("archive review capture_returns must map context IDs to return lines") + for context_id, line in capture_returns.items(): + _identifier(context_id, "archive review context ID") + if type(line) is not int or line < 1: + raise ValueError("archive review capture return lines must be positive integers") + result_note = document.get("result_note") + if result_note is not None: + if not isinstance(result_note, str) or not result_note.strip(): + raise ValueError("archive review result_note must be nonempty text") + result_note = _text(result_note, "Not recorded", limit=len(result_note)) + return {"task": task, "corpus": corpus, "decisions": raw_decisions, + "result_note": result_note, + "capture_returns": capture_returns} + + +def reviewed_uses(review, events, decisions): + """Yield the material, disposition and portable evidence for each reviewed association.""" + by_line = {event.get("trace_line"): event for event in events + if type(event.get("trace_line")) is int} + for item in review["decisions"]: + event = by_line.get(item["event_line"]) + if event is None: + raise ValueError("event_line must reference an observed event in this round") + delivery = event.get("delivery") + if not isinstance(delivery, dict) or delivery.get("state") != "visible": + raise ValueError("archive review must cite a verified delivered passage") + return_line = delivery.get("return_line") + if type(return_line) is not int or item["decision_line"] <= return_line: + raise ValueError("archive review decision must follow the visible tool return") + indexer_event = event.get("tool") == "collect" + if indexer_event and not item.get("passage_path"): + raise ValueError("Indexer review must name the exact captured resource path") + if not indexer_event and "passage_path" in item: + raise ValueError("passage_path is only valid for a verified Indexer return") + passages = [entry for entry in delivery.get("passages", []) + if isinstance(entry, dict) and entry.get("ref") == item["ref"] + and entry.get("line") == item["passage_line"] + and ("passage_path" not in item + or entry.get("path") == item["passage_path"])] + if item["passage_line"] is None and delivery.get("archive_omitted"): + raise ValueError("source excerpt has no unique line locator; use a line-numbered read") + if len(passages) > 1: + raise ValueError("source excerpt has multiple matches; use a line-numbered read") + passage = passages[0] if passages else None + if passage is None or not isinstance(passage.get("text"), str) or not passage["text"]: + raise ValueError("archive review must cite an exact delivered source passage") + passage_text = passage["text"] + if _text(passage_text, "", limit=len(passage_text)) != passage_text: + raise ValueError("archive review source passage is not safe to publish verbatim") + decision_text = decisions.get(item["decision_line"]) + if not isinstance(decision_text, str) or item["decision_quote"] not in decision_text: + raise ValueError("decision_quote must match the assistant message exactly") + event_id = event.get("id") + if not isinstance(event_id, str) or not event_id: + raise ValueError("archive review event is missing its stable public ID") + evidence = { + "use": item["use"], "basis": "archive_reviewed", "event_id": event_id, + "passage_line": item["passage_line"], "passage_text": passage_text, + "decision_line": item["decision_line"], "decision_quote": item["decision_quote"], + } + if "passage_path" in item: + evidence["passage_path"] = item["passage_path"] + yield item["ref"], item["use"], evidence diff --git a/src/inferrsi/harness/search_attribution.py b/src/inferrsi/harness/search_attribution.py index 367e768b..e1d5d3f8 100644 --- a/src/inferrsi/harness/search_attribution.py +++ b/src/inferrsi/harness/search_attribution.py @@ -2,39 +2,17 @@ from __future__ import annotations -import hashlib import json import os from pathlib import Path from tempfile import NamedTemporaryFile from .campaign_records import _identifier, _round_records +from .capture_delivery import capture_event, capture_return_texts, load_campaign_capture from .codex_search import read_window from .dashboard_data import _inside, _object -from .knowledge_capture import MAX_CONTEXT_CAPTURE_BYTES, verify_capture from .optimization_log import STAGES, read_latest_log - - -def _capture(root, campaign_id, context_id): - _identifier(context_id, "context ID") - path = root / "knowledge" / f"{context_id}.json" - if not _inside(root, path): - raise ValueError("capture escapes the campaign") - with path.open("rb") as stream: - raw = stream.read(MAX_CONTEXT_CAPTURE_BYTES + 1) - doc, state = verify_capture(raw) - if not doc or doc["campaign_id"] != campaign_id or doc["context_id"] != context_id: - raise ValueError(f"invalid or mismatched campaign capture ({state})") - resources = {c["ref"]: [r["path"] for r in c["resources"]] for c in doc["contexts"]} - if any(not isinstance(item.get("ref"), str) or not isinstance(item.get("revision"), str) - for item in doc["items"]): - raise ValueError("capture item is missing its ref or revision") - materials = [{"ref": item["ref"], "revision": item["revision"], - "resources": resources.get(item["ref"], [])} for item in doc["items"]] - return {"action": "search", "tool": "collect", "context_id": context_id, - "source": path.relative_to(root).as_posix(), "request": doc["request"], - "selection": doc.get("selection"), "materials": materials, - "sha256": hashlib.sha256(raw).hexdigest()} +from .search_archive import read_archive_review, reviewed_uses def _use(materials, ref, status, evidence): @@ -49,15 +27,28 @@ def _use(materials, ref, status, evidence): item["use"] = status if len(states) == 1 else "used_and_rejected" -def _log_uses(root, campaign_id, number, captures, materials): +def _atomic_json(target, record): + with NamedTemporaryFile(mode="w", encoding="utf-8", dir=target.parent, delete=False) as stream: + temporary = Path(stream.name) + try: + json.dump(record, stream, ensure_ascii=False, indent=2, allow_nan=False) + stream.write("\n") + stream.close() + os.replace(temporary, target) + finally: + temporary.unlink(missing_ok=True) + + +def _log_uses(root, campaign_id, number, captures, materials, *, archive=False): if not captures: - return + return [] log, versions, state = read_latest_log(root) if state == "not_recorded": - return + return [] if not log or log["campaign_id"] != campaign_id: raise ValueError(f"cannot correlate campaign optimization log ({state})") iteration = next((i for i in log["iterations"] if i["round"] == number), {}) + declarations = [] for index, entry in enumerate(log["knowledge"]): key = entry["context_id"] if key not in iteration.get("context_ids", []) or key not in captures: @@ -65,13 +56,30 @@ def _log_uses(root, campaign_id, number, captures, materials): capture = captures[key] if capture["sha256"] != entry["sha256"].lower(): raise ValueError("optimization log and selected capture revisions differ") - resources = {m["ref"]: m["resources"] for m in capture["materials"]} + material_rows = {m["ref"]: m for m in capture["materials"]} + resources = {ref: material.get("resources", []) + for ref, material in material_rows.items()} source = f"{versions[-1]['path']}#/knowledge/{index}" for status, refs in (("used", entry["used_refs"]), ("rejected", [r["ref"] for r in entry.get("rejected_refs", [])])): for ref in refs: - if ref not in resources: + if ref not in resources and not archive: raise ValueError("optimization log cites a material absent from its capture") + material = material_rows.get(ref) + if archive: + delivery = capture.get("delivery", {}) + visible = any(isinstance(row, dict) and row.get("ref") == ref + for row in delivery.get("visible_resources", [])) + declarations.append({ + "ref": ref, "use": status, "basis": "optimization_log_declaration", + "source": source, "log_sha256": versions[-1].get("sha256"), + "context_id": key, "capture_source": capture["source"], + "capture_sha256": capture["sha256"], "stage": entry["stage"], + "preparation_state": (material.get("preparation_state", "prepared") + if material else "not_in_capture"), + "delivery_state": "visible" if visible else "unknown", + }) + continue evidence = {"source": source, "context_id": key, "stage": entry["stage"]} if status == "used" and not resources[ref]: conflict = {**evidence, "claimed_use": status, "reason": "context_omitted"} @@ -80,6 +88,7 @@ def _log_uses(root, campaign_id, number, captures, materials): conflicts.append(conflict) else: _use(materials, ref, status, evidence) + return declarations def _write(root, record): @@ -111,9 +120,7 @@ def _write(root, record): current = record["materials"].get(ref, {}) if evidence not in current.get("evidence", []): _use(record["materials"], ref, evidence["use"], evidence) - record["observed"]["used"] = sum( - m["use"] in ("used", "used_and_rejected") for m in record["materials"].values() - ) + record["observed"]["used"] = _used_count(record["materials"]) if old == record: return target continue @@ -123,15 +130,24 @@ def _write(root, record): lo, hi = previous["lines"] if max(lo, window["lines"][0]) <= min(hi, window["lines"][1]): raise ValueError("trace window overlaps another round") - with NamedTemporaryFile(mode="w", encoding="utf-8", dir=directory, delete=False) as stream: - temporary = Path(stream.name) - try: - json.dump(record, stream, ensure_ascii=False, indent=2, allow_nan=False) - stream.write("\n") - stream.close() - os.replace(temporary, target) - finally: - temporary.unlink(missing_ok=True) + _atomic_json(target, record) + return target + + +def _write_archive(root, record): + """Atomically write a portable v2 projection beside the local v1 ledger.""" + directory = root / "search" / "archive" + if not _inside(root, directory): + raise ValueError("search archive directory escapes the campaign") + directory.mkdir(parents=True, exist_ok=True) + target = directory / f"round-{record['round']}.json" + if target.exists(): + old, state = _object(target, root) + if state != "present" or old.get("schema_version") != 2: + raise ValueError("existing archive record has invalid content or version") + if old == record: + return target + _atomic_json(target, record) return target @@ -162,8 +178,41 @@ def _round_candidates(root, number): return runs +def _observations(events, *, archive=False): + """Count tool hits separately from what appeared in a model-facing return.""" + materials, hits = {}, set() + counts = {"searches": 0, "hits": 0, "unique_hits": 0, "used": 0} + for event in events: + refs = {row["ref"] for row in event["materials"]} + if event["action"] == "search": + counts["searches"] += 1 + counts["hits"] += len(refs) + hits.update(refs) + for ref in sorted(refs): + materials.setdefault(ref, {"use": "unknown"}) + if archive: + delivery = event["delivery"] + passages = {row["ref"] for row in delivery.get("passages", [])} + visible_refs = set(delivery.get("visible_refs", [])) + resources = {row["ref"] for row in delivery.get("visible_resources", [])} + for ref in refs: + material = materials[ref] + state = material.setdefault("delivery_state", "unknown") + if ref in passages or ref in resources: + material["delivery_state"] = "visible" + elif ref in visible_refs and state == "unknown": + material["delivery_state"] = "ref_visible" + counts["unique_hits"] = len(hits) + return materials, counts + + +def _used_count(materials): + return sum(m["use"] in ("used", "used_and_rejected") for m in materials.values()) + + def import_search(root: Path, *, number: int, trace=None, lines=None, knowledge_root=None, - context_ids=(), used=(), rejected=(), complete=False, stage=None): + context_ids=(), used=(), rejected=(), complete=False, stage=None, + archive_review=None): root = root.resolve() campaign, state = _object(root / "campaign.json", root) if state != "present" or number < 1: @@ -172,34 +221,51 @@ def import_search(root: Path, *, number: int, trace=None, lines=None, knowledge_ trace_inputs = (trace is not None, lines is not None, knowledge_root is not None) if any(trace_inputs) and not all(trace_inputs): raise ValueError("--trace, --lines and --knowledge-root must be supplied together") + review = read_archive_review(archive_review) if archive_review is not None else None if stage is not None and stage not in STAGES: raise ValueError(f"stage must be one of {', '.join(STAGES)}") if complete and not trace and not context_ids: raise ValueError("complete coverage needs a trace window or a capture") - window, events, decisions, uncertain = None, [], set(), [] + capture_ids = list(dict.fromkeys(context_ids)) + if review is not None and not trace and (review["decisions"] or review["capture_returns"]): + raise ValueError("archive decisions and capture_returns need a trace window") + if review is not None and not trace and not capture_ids: + raise ValueError("archive review needs a trace window or a verified capture") + if review is not None and set(review["capture_returns"]) - set(capture_ids): + raise ValueError("archive review capture_returns must name selected context IDs") + capture_return_lines = list(review["capture_returns"].values()) if review else [] + if len(set(capture_return_lines)) != len(capture_return_lines): + raise ValueError("capture return line is ambiguous across selected contexts") + indexer_decisions = {key: [] for key in capture_ids} + command_decisions = [] + if review: + context_by_return = {line: key for key, line in review["capture_returns"].items()} + for item in review["decisions"]: + context_id = context_by_return.get(item["event_line"]) + if context_id is not None: + if "passage_path" not in item: + raise ValueError("Indexer review must name the exact captured resource path") + indexer_decisions[context_id].append(item) + else: + command_decisions.append(item) + if "passage_path" in item: + raise ValueError("passage_path is only valid for a verified Indexer return") + + window, events, decisions, uncertain = None, [], {}, [] if trace: - window, events, decisions, uncertain = read_window(trace, lines, knowledge_root) - captures = {key: _capture(root, campaign_id, key) for key in dict.fromkeys(context_ids)} + window, events, decisions, uncertain, _ = read_window( + trace, lines, knowledge_root, archive=False) + capture_inputs = {key: load_campaign_capture(root, campaign_id, key) for key in capture_ids} + captures = {key: capture_event(key, capture_inputs[key]) for key in capture_ids} events.extend(captures.values()) - materials = {} - hits = set() - searches = hit_count = 0 - for event in events: - refs = {m["ref"] for m in event["materials"]} - if event["action"] == "search": - searches += 1 - hit_count += len(refs) - hits.update(refs) - for ref in sorted(refs): - materials.setdefault(ref, {"use": "unknown"}) + materials, observed = _observations(events) _log_uses(root, campaign_id, number, captures, materials) for status, entries in (("used", used), ("rejected", rejected)): for ref, line in entries: if int(line) not in decisions: raise ValueError("decision line must reference an assistant message in this round") _use(materials, ref, status, {"decision_line": int(line), "basis": "caller_reviewed"}) - observed = {"searches": searches, "hits": hit_count, "unique_hits": len(hits), - "used": sum(m["use"] in ("used", "used_and_rejected") for m in materials.values())} + observed["used"] = _used_count(materials) coverage = "complete" if complete and not uncertain else "partial" if not window and not captures: coverage = "missing" @@ -208,4 +274,62 @@ def import_search(root: Path, *, number: int, trace=None, lines=None, knowledge_ "unresolved_command_lines": uncertain, "events": events, "materials": materials, "observed": observed, "counts": observed if coverage == "complete" else None, "candidates": _round_candidates(root, number)} - return _write(root, record) + if review is None: + return _write(root, record) + + selections = {} + for item in command_decisions: + selections.setdefault(item["event_line"], []).append((item["ref"], item["passage_line"])) + archive_events, archive_decisions, archive_uncertain, unresolved_details = [], {}, [], [] + if trace: + _, archive_events, archive_decisions, archive_uncertain, unresolved_details = read_window( + trace, lines, knowledge_root, archive=True, selections=selections) + return_texts = {} + if review["capture_returns"]: + return_texts = capture_return_texts(trace, lines, review["capture_returns"], capture_ids) + archive_captures = {} + for key in capture_ids: + return_line = review["capture_returns"].get(key) + archive_captures[key] = capture_event( + key, capture_inputs[key], archive=True, + return_chunks=return_texts.get(key), + return_line=return_line, + selected_passages=indexer_decisions[key]) + archive_events.extend(archive_captures.values()) + archive_materials, archive_observed = _observations(archive_events, archive=True) + declarations = _log_uses(root, campaign_id, number, archive_captures, archive_materials, + archive=True) + for status, entries in (("used", used), ("rejected", rejected)): + for ref, line in entries: + line = int(line) + if line not in archive_decisions: + raise ValueError("decision line must reference an assistant message in this round") + if ref not in archive_materials: + raise ValueError("use declaration must name a material observed in this round") + declarations.append({ + "ref": ref, "use": status, "basis": "caller_flag_declaration", + "decision_line": line, + "delivery_state": archive_materials[ref].get("delivery_state", "unknown"), + }) + for ref, status, evidence in reviewed_uses(review, archive_events, archive_decisions): + _use(archive_materials, ref, status, evidence) + archive_observed["used"] = _used_count(archive_materials) + archive_coverage = ("complete" if complete and not archive_uncertain and not unresolved_details + else "partial") + archive_record = { + "schema_version": 2, "campaign_id": campaign_id, "round": number, + "stage": stage, "task": review["task"], "corpus": review["corpus"], + "trace": ({"lines": window["lines"], "archive": True} if window else None), + "context_ids": list(archive_captures), "coverage": archive_coverage, + "unresolved_command_lines": archive_uncertain, + "unresolved_details": unresolved_details, + "events": archive_events, "materials": archive_materials, + "declarations": declarations, "observed": archive_observed, + "counts": archive_observed if archive_coverage == "complete" else None, + "candidates": record["candidates"], + } + if review["result_note"] is not None: + archive_record["result_note"] = review["result_note"] + archive_record["result_note_basis"] = "caller_reviewed" + _write(root, record) + return _write_archive(root, archive_record) diff --git a/tests/test_codex_search_delivery.py b/tests/test_codex_search_delivery.py new file mode 100644 index 00000000..5d3911be --- /dev/null +++ b/tests/test_codex_search_delivery.py @@ -0,0 +1,413 @@ +"""Model-visible file delivery must be traceable to exact tool output.""" + +import json + +from inferrsi.harness.codex_search import read_window + + +def _meta(cwd): + return {"type": "session_meta", + "payload": {"id": "synthetic-session", "cwd": str(cwd)}} + + +def _call(cwd, command, call_id): + source = ( + "const r = await tools.exec_command({\n" + f" cmd: {json.dumps(command)},\n" + f" workdir: {json.dumps(str(cwd))}\n" + "});\ntext(r.output);" + ) + return {"type": "response_item", + "payload": {"type": "custom_tool_call", "name": "exec", + "call_id": call_id, "input": source}} + + +def _command(cwd, command, output, event_id, exit_code=0): + return { + "type": "event_msg", + "payload": { + "type": "item_completed", + "item": { + "type": "CommandExecution", "id": event_id, + "cwd": cwd.as_uri(), "command": ["/bin/zsh", "-lc", command], + "exit_code": exit_code, "aggregated_output": output, + }, + }, + } + + +def _result(call_id, output): + return { + "type": "response_item", + "payload": { + "type": "custom_tool_call_output", "call_id": call_id, + "output": [{"type": "input_text", + "text": f"Script completed\nWall time in 0.1 seconds\nOutput:\n{output}"}], + }, + } + + +def _message(text): + return {"type": "response_item", + "payload": {"type": "message", "role": "assistant", + "phase": "final_answer", + "content": [{"type": "output_text", "text": text}]}} + + +def _rollout(path, *rows): + path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8") + + +def test_archive_keeps_multiple_exact_matches_and_legacy_materials(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "rg -n needle ." + output = "a.md:2:first needle\na.md:8:second needle\nb.md:4:third needle\n" + _rollout( + trace, _meta(corpus), _call(corpus, command, "call-search"), + _command(corpus, command, output, "exec-private-id"), + _result("call-search", output), _message("Use the second match in a.md."), + ) + + window, events, decisions, uncertain, details = read_window( + trace, (2, 5), corpus, archive=True, + ) + event = events[0] + assert window["archive"] is True + assert event["id"] == event["event_id"] == "file-3" + assert event["materials"] == [ + {"ref": "a.md", "match_line": 2}, {"ref": "b.md", "match_line": 4}, + ] + assert event["delivery"]["state"] == "visible" + assert event["delivery"]["return_line"] == 4 + assert event["delivery"]["runtime_truncated"] is False + assert [(item["ref"], item["line"], item["text"]) + for item in event["delivery"]["passages"]] == [ + ("a.md", 2, "first needle"), + ("a.md", 8, "second needle"), + ("b.md", 4, "third needle"), + ] + assert decisions == {5: "Use the second match in a.md."} + assert event["delivery"]["return_line"] < next(iter(decisions)) + assert uncertain == [] and details == [] + + _, legacy_events, legacy_decisions, legacy_uncertain, legacy_details = read_window( + trace, (2, 5), corpus, + ) + assert legacy_events[0]["materials"] == event["materials"] + assert "delivery" not in legacy_events[0] and legacy_events[0]["event_id"] == "exec-private-id" + assert legacy_decisions == decisions + assert legacy_uncertain == [] and legacy_details == [] + + +def test_sed_late_selected_line_survives_bounded_default_excerpt(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "sed -n '1,15p' case.md" + output = "".join(f"content line {line}\n" for line in range(1, 16)) + _rollout(trace, _meta(corpus), _call(corpus, command, "call-read"), + _command(corpus, command, output, "exec-read"), + _result("call-read", output)) + + _, events, _, uncertain, details = read_window( + trace, (2, 4), corpus, archive=True, + selections={3: [("case.md", 15)]}, + ) + delivery = events[0]["delivery"] + assert delivery["state"] == "visible" and delivery["return_line"] == 4 + assert len(delivery["passages"]) == 13 + assert {key: delivery["passages"][-1][key] for key in ("ref", "line", "text")} == { + "ref": "case.md", "line": 15, "text": "content line 15", + } + assert not any(key in delivery["passages"][-1] + for key in ("redacted", "root_normalized", "truncated")) + assert delivery["archive_omitted"] is True + assert delivery["omitted_passages"] == 2 + assert uncertain == [] and details == [] + + +def test_read_tools_infer_only_reliable_line_numbers(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + cases = [ + ("cat case.md", "first\nsecond\n", [1, 2]), + ("nl -ba case.md", " 1\tfirst\n 2\tsecond\n", [1, 2]), + ] + rows = [_meta(corpus)] + for index, (command, output, _) in enumerate(cases): + call_id = f"call-{index}" + rows.extend([ + _call(corpus, command, call_id), + _command(corpus, command, output, f"exec-{index}"), + _result(call_id, output), + ]) + _rollout(trace, *rows) + + _, events, _, uncertain, _ = read_window(trace, (2, len(rows)), corpus, archive=True) + assert [event["delivery"]["state"] for event in events] == ["visible", "visible"] + assert [[passage["line"] for passage in event["delivery"]["passages"]] + for event in events] == [[1, 2], [1, 2]] + assert uncertain == [] + + +def test_join_must_be_inside_window_and_not_rebuilt_from_adjacent_command(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "rg -n needle ." + output = "a.md:2:needle\n" + _rollout(trace, _meta(corpus), _call(corpus, command, "call-search"), + _command(corpus, command, output, "exec-search"), + _result("call-search", output)) + + _, events, _, uncertain, details = read_window(trace, (3, 4), corpus, archive=True) + delivery = events[0]["delivery"] + assert delivery["state"] == "unknown" + assert "return_line" not in delivery + assert delivery["passages"] == [] and uncertain == [3] + assert details[0]["id"] == "file-3" + assert "exec-search" not in json.dumps(details) + + +def test_closed_window_ignores_later_duplicate_and_partial_trace_rows(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "rg -n needle ." + output = "a.md:2:needle\n" + _rollout( + trace, _meta(corpus), _call(corpus, command, "call-first"), + _command(corpus, command, output, "exec-first"), + _result("call-first", output), + _call(corpus, command, "call-later"), + _command(corpus, command, output, "exec-later"), + ) + with trace.open("a", encoding="utf-8") as stream: + stream.write('{"type":"response_item","payload":') + + _, events, _, uncertain, details = read_window(trace, (2, 4), corpus, archive=True) + assert events[0]["delivery"]["state"] == "visible" + assert events[0]["delivery"]["return_line"] == 4 + assert uncertain == [] and details == [] + + +def test_truncated_result_keeps_only_exact_visible_lines_and_refs(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "rg -n needle ." + aggregate = "a.md:1:first\nb.md:2:middle\nc.md:3:last\n" + visible = ( + "Warning: truncated output (original token count exceeded)\n" + "a.md:1:first\n[omitted]\nc.md:3:last\n" + ) + _rollout(trace, _meta(corpus), _call(corpus, command, "call-search"), + _command(corpus, command, aggregate, "exec-search"), + _result("call-search", visible)) + + _, events, _, uncertain, details = read_window(trace, (2, 4), corpus, archive=True) + event = events[0] + delivery = event["delivery"] + assert delivery["state"] == "visible" and delivery["return_line"] == 4 + assert delivery["runtime_truncated"] is True + assert delivery["visible_refs"] == ["a.md", "c.md"] + assert [(passage["ref"], passage["line"]) for passage in delivery["passages"]] == [ + ("a.md", 1), ("c.md", 3), + ] + assert [material["ref"] for material in event["materials"]] == ["a.md", "b.md", "c.md"] + assert uncertain == [3] + assert details[0]["reason"] == "command_output_incomplete" + + +def test_transformed_and_multi_file_outputs_cannot_backfill_passages(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + transformed_command = "rg -n needle ." + transformed_aggregate = "a.md:2:private content\n" + transformed_response = "a.md:2:[redacted]\n" + multi_command = "cat a.md b.md" + multi_output = "first file text\nsecond file text\n" + _rollout( + trace, _meta(corpus), + _call(corpus, transformed_command, "call-transformed"), + _command(corpus, transformed_command, transformed_aggregate, "exec-transformed"), + _result("call-transformed", transformed_response), + _call(corpus, multi_command, "call-multi"), + _command(corpus, multi_command, multi_output, "exec-multi"), + _result("call-multi", multi_output), + ) + + _, events, _, uncertain, details = read_window(trace, (2, 7), corpus, archive=True) + assert [event["delivery"]["state"] for event in events] == ["unknown", "unknown"] + assert all(event["delivery"]["passages"] == [] for event in events) + assert all("return_line" not in event["delivery"] for event in events) + assert uncertain == [3, 6] + assert [detail["reason"] for detail in details] == [ + "output_bytes_do_not_match", "read_sources_not_separable", + ] + assert "private content" not in json.dumps(details) + + +def test_zero_hit_rg_is_visible_when_empty_outer_result_is_correlated(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "rg -n absent ." + _rollout(trace, _meta(corpus), _call(corpus, command, "call-empty"), + _command(corpus, command, "", "exec-empty", exit_code=1), + _result("call-empty", "")) + + _, events, _, uncertain, details = read_window(trace, (2, 4), corpus, archive=True) + delivery = events[0]["delivery"] + assert events[0]["materials"] == [] + assert delivery["state"] == "visible" and delivery["return_line"] == 4 + assert delivery["runtime_truncated"] is False + assert delivery["passages"] == [] and delivery["visible_refs"] == [] + assert uncertain == [] and details == [] + + +def test_filename_only_search_uses_visible_refs_without_invented_passages(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "rg -l needle ." + output = "a.md\n" + _rollout(trace, _meta(corpus), _call(corpus, command, "call-files"), + _command(corpus, command, output, "exec-files"), + _result("call-files", output)) + + _, events, _, uncertain, details = read_window(trace, (2, 4), corpus, archive=True) + delivery = events[0]["delivery"] + assert delivery["state"] == "visible" + assert delivery["visible_refs"] == ["a.md"] + assert delivery["passages"] == [] + assert uncertain == [] and details == [] + + +def test_read_truncation_marker_blocks_inferred_late_line_numbers(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "cat case.md" + output = "first line\ntokens truncated …\nlate text\n" + _rollout(trace, _meta(corpus), _call(corpus, command, "call-read"), + _command(corpus, command, output, "exec-read"), + _result("call-read", output)) + + _, events, _, uncertain, details = read_window(trace, (2, 4), corpus, archive=True) + delivery = events[0]["delivery"] + assert delivery["state"] == "visible" and delivery["runtime_truncated"] is True + assert [(item["line"], item["text"]) for item in delivery["passages"]] == [ + (1, "first line"), + ] + assert uncertain == [3] + assert details[0]["reason"] == "command_output_incomplete" + + +def test_passage_text_normalizes_corpus_root_and_redacts_secret_values(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = f"rg -n needle {corpus}" + output = f"{corpus}/a.md:4:source={corpus}/private.md token=synthetic-secret\n" + _rollout(trace, _meta(corpus), _call(corpus, command, "call-search"), + _command(corpus, command, output, "exec-search"), + _result("call-search", output)) + + _, events, _, _, _ = read_window(trace, (2, 4), corpus, archive=True) + passage = events[0]["delivery"]["passages"][0] + assert passage["ref"] == "a.md" and passage["line"] == 4 + assert passage["root_normalized"] is True and passage["redacted"] is True + assert str(corpus) not in passage["text"] + assert "synthetic-secret" not in passage["text"] + + +def test_pipeline_unresolved_detail_keeps_only_safe_operation_skeleton(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "rg -n needle a.md | head -1" + output = "a.md:2:needle\n" + _rollout(trace, _meta(corpus), _command(corpus, command, output, "exec-pipe")) + + _, events, _, uncertain, details = read_window(trace, (2, 2), corpus, archive=True) + assert events == [] and uncertain == [2] + assert details[0]["id"] == "file-2" + assert details[0]["operation"] == { + "tools": ["rg"], "action": "unknown", "compound": True, + } + assert "a.md" not in json.dumps(details) + assert "needle" not in json.dumps(details) + + +def test_scoped_file_list_filter_keeps_only_a_few_normalized_path_samples(tmp_path): + corpus = tmp_path / "knowledge" + scope = corpus / "skill_library" + scope.mkdir(parents=True) + trace = tmp_path / "rollout.jsonl" + for ignore_case, flag in ((False, ""), (True, "-i ")): + command = f"rg --files {scope} | rg {flag}needle" + call_id = f"call-list-{ignore_case}" + output = "".join(f"{scope}/case-{index}.md\n" for index in range(1, 6)) + _rollout( + trace, _meta(corpus), _call(corpus, command, call_id), + _command(corpus, command, output, f"exec-list-{ignore_case}"), + _result(call_id, output), + ) + + _, events, _, uncertain, details = read_window( + trace, (2, 4), corpus, archive=True, + ) + detail = details[0] + assert events == [] and uncertain == [3] + assert detail["id"] == "file-3" and detail["return_line"] == 4 + assert detail["operation"] == { + "tools": ["rg", "rg"], "action": "file_list_filter", + "scope": ["skill_library"], "compound": True, + } + assert detail["runtime_truncated"] is False + assert detail["archive_omitted"] is True + assert len(detail["excerpt"]) == 4 + assert detail["excerpt"][0]["ref"] == "skill_library/case-1.md" + assert detail["excerpt"][0]["line"] is None + assert detail["excerpt"][0]["text"] == ( + "/skill_library/case-1.md" + ) + assert str(corpus) not in json.dumps(detail) + assert "exec-list" not in json.dumps(detail) + assert "materials" not in detail + + +def test_duplicate_identical_command_events_make_join_ambiguous(tmp_path): + corpus = tmp_path / "knowledge" + corpus.mkdir() + trace = tmp_path / "rollout.jsonl" + command = "rg -n needle ." + output = "a.md:2:needle\n" + _rollout(trace, _meta(corpus), _call(corpus, command, "call-search"), + _command(corpus, command, output, "exec-one"), + _command(corpus, command, output, "exec-two"), + _result("call-search", output)) + + _, events, _, uncertain, details = read_window(trace, (2, 5), corpus, archive=True) + assert [event["delivery"]["state"] for event in events] == ["unknown", "unknown"] + assert all(event["delivery"]["passages"] == [] for event in events) + assert uncertain == [3, 4] + assert all(detail["reason"] == "command_not_unique" for detail in details) + assert "exec-one" not in json.dumps(events) and "exec-two" not in json.dumps(events) + +def test_truncated_repeated_paragraph_has_no_unambiguous_source_line(tmp_path): + corpus = tmp_path / "knowledge" + trace = tmp_path / "rollout.jsonl" + command = "cat case.md" + body = "first\nrepeat\nagain\nrepeat\nagain\nlast\n" + shown = "Warning: truncated output\nrepeat\nagain\nlast\n" + _rollout(trace, _meta(corpus), _call(corpus, command, "read"), + _command(corpus, command, body, "read-file"), _result("read", shown)) + + _, events, _, _, _ = read_window(trace, (2, 4), corpus, archive=True) + assert [(p["line"], p["text"]) for p in events[0]["delivery"]["passages"]] == [(6, "last")] diff --git a/tests/test_evidence_inventory.py b/tests/test_evidence_inventory.py index 4efa342b..27be6095 100644 --- a/tests/test_evidence_inventory.py +++ b/tests/test_evidence_inventory.py @@ -145,6 +145,6 @@ def test_checked_in_inventory(): repo = Path(__file__).resolve().parents[1] result = validate_inventory(repo) assert result["errors"] == [] - assert len(result["bundles"]) == 22 + assert len(result["bundles"]) == 23 assert result["checked_hashes"] == 430 assert result["unavailable"] # Historical raw artifacts are not in this checkout. diff --git a/tests/test_search_archive.py b/tests/test_search_archive.py new file mode 100644 index 00000000..9b49060e --- /dev/null +++ b/tests/test_search_archive.py @@ -0,0 +1,308 @@ +"""Portable search records require exact delivered passages and reviewed decisions.""" + +import hashlib +import json + +import pytest + +from inferrsi.harness.campaign_records import create_campaign +from inferrsi.harness.knowledge_capture import assemble_context_text, capture_context +from inferrsi.harness.search_attribution import import_search + +REVISION = "f6d200" + "a" * 34 +CORPUS = {"repository": "InferMatrixLab/InferRSI", "revision": REVISION, + "root": "src/inferrsi/data/skill_library"} + + +def _review(path, *, decisions, **extra): + document = {"task": "Review the search and explain the decision.", + "corpus": CORPUS, "decisions": decisions, **extra} + path.write_text(json.dumps(document), encoding="utf-8") + return path + + +def _archive_event(*, delivery_state="visible", passage_ref="cases/a.md", line=1, + text="Use the stable RoPE case.", archive_omitted=False): + return { + "action": "read", "tool": "cat", "id": "tool-event-7", "event_id": "tool-event-7", + "trace_line": 2, + "materials": [{"ref": "cases/a.md"}], + "delivery": {"state": delivery_state, "return_line": 3, + "archive_omitted": archive_omitted, + "passages": [ + {"ref": passage_ref, "line": line, "text": text}, + ]}, + } + + +def _reader(monkeypatch, event): + def read_window(trace, lines, corpus, *, archive=False, selections=None): + window = {"session": "private-session", "lines": [2, 4]} + if archive: + return {**window, "archive": True}, [event], { + 4: "I will use the stable RoPE case for validation."}, [], [] + return window, [], {4: "I will use the stable RoPE case for validation."}, [], [] + + monkeypatch.setattr("inferrsi.harness.search_attribution.read_window", read_window) + + +def _decision(**changes): + return {"use": "used", "ref": "cases/a.md", "event_line": 2, + "passage_line": 1, "decision_line": 4, + "decision_quote": "use the stable RoPE case", **changes} + + +def _real_read_trace(path, corpus): + output = "Use the stable RoPE case." + rows = [ + {"type": "session_meta", "payload": {"id": "private-session"}}, + {"type": "response_item", "payload": { + "type": "function_call", "name": "exec_command", "call_id": "call-1", + "arguments": {"cmd": "cat case.md", "workdir": str(corpus)}, + }}, + {"type": "event_msg", "payload": {"type": "item_completed", "item": { + "type": "CommandExecution", "id": "tool-event-7", "cwd": corpus.as_uri(), + "command": ["/bin/sh", "-lc", "cat case.md"], "exit_code": 0, + "aggregated_output": output, + }}}, + {"type": "response_item", "payload": { + "type": "function_call_output", "call_id": "call-1", "output": f"Output:\n{output}", + }}, + {"type": "response_item", "payload": { + "type": "message", "role": "assistant", "phase": "final_answer", + "content": [{"type": "output_text", + "text": "I will use the stable RoPE case for validation."}], + }}, + ] + path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8") + + +def test_archive_rejects_unavailable_or_mismatched_passage_before_writing(tmp_path, monkeypatch): + campaign = tmp_path / "campaign" + create_campaign(campaign, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "corpus" + trace.write_text("trace", encoding="utf-8") + inputs = {"number": 1, "trace": trace, "lines": [2, 4], + "knowledge_root": corpus, "complete": True} + review = _review(tmp_path / "review.json", decisions=[_decision()]) + + _reader(monkeypatch, _archive_event(delivery_state="unknown")) + with pytest.raises(ValueError, match="verified delivered passage"): + import_search(campaign, archive_review=review, **inputs) + assert not (campaign / "search/round-1.json").exists() + + _reader(monkeypatch, _archive_event()) + # The selected event is visible, but the reviewed source line is absent. + review.write_text(json.dumps({"task": "Review the search and explain the decision.", + "corpus": CORPUS, "decisions": [ + _decision(passage_line=3), + ]}), encoding="utf-8") + with pytest.raises(ValueError, match="exact delivered source passage"): + import_search(campaign, archive_review=review, **inputs) + assert not (campaign / "search/round-1.json").exists() + + review = _review(tmp_path / "review.json", decisions=[ + _decision(decision_quote="a quote not in the assistant response")]) + with pytest.raises(ValueError, match="decision_quote"): + import_search(campaign, archive_review=review, **inputs) + assert not (campaign / "search/round-1.json").exists() + + +def test_archive_requires_decision_after_visible_return_and_unique_locator(tmp_path, monkeypatch): + campaign = tmp_path / "campaign" + create_campaign(campaign, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "corpus" + trace.write_text("trace", encoding="utf-8") + inputs = {"number": 1, "trace": trace, "lines": [2, 4], + "knowledge_root": corpus, "complete": True} + + _reader(monkeypatch, _archive_event()) + review = _review(tmp_path / "review.json", decisions=[ + _decision(decision_line=3)]) + with pytest.raises(ValueError, match="follow the visible tool return"): + import_search(campaign, archive_review=review, **inputs) + + _reader(monkeypatch, _archive_event(line=None, archive_omitted=True)) + review = _review(tmp_path / "review.json", decisions=[ + _decision(passage_line=None)]) + with pytest.raises(ValueError, match="line-numbered read"): + import_search(campaign, archive_review=review, **inputs) + + +def test_legacy_flag_remains_a_noncounting_declaration_in_archive(tmp_path): + campaign = tmp_path / "campaign" + create_campaign(campaign, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "corpus" + _real_read_trace(trace, corpus) + review = _review(tmp_path / "review.json", decisions=[]) + path = import_search(campaign, number=1, trace=trace, lines=[2, 5], + knowledge_root=corpus, used=[("case.md", 5)], + archive_review=review) + record = json.loads(path.read_text(encoding="utf-8")) + assert record["materials"]["case.md"]["use"] == "unknown" + assert record["observed"]["used"] == 0 + assert record["declarations"] == [{ + "ref": "case.md", "use": "used", "basis": "caller_flag_declaration", + "decision_line": 5, "delivery_state": "visible", + }] + local = json.loads((campaign / "search/round-1.json").read_text(encoding="utf-8")) + assert local["materials"]["case.md"]["use"] == "used" + + +def test_archive_embeds_verified_quote_and_reimport_is_stable_and_correctable(tmp_path): + campaign = tmp_path / "campaign" + create_campaign(campaign, "example") + trace, corpus = tmp_path / "rollout.jsonl", tmp_path / "corpus" + _real_read_trace(trace, corpus) + review = _review(tmp_path / "review.json", decisions=[_decision( + ref="case.md", event_line=3, decision_line=5, + decision_quote="I will use the stable RoPE case")], + result_note="Decision only; no device run.") + inputs = {"number": 1, "trace": trace, "lines": [2, 5], + "knowledge_root": corpus, "complete": True} + + path = import_search(campaign, archive_review=review, **inputs) + assert path == campaign / "search/archive/round-1.json" + original = path.read_bytes() + assert import_search(campaign, archive_review=review, **inputs) == path + assert path.read_bytes() == original + + document = json.loads(original) + assert document["schema_version"] == 2 + assert document["task"] == "Review the search and explain the decision." + assert document["corpus"] == CORPUS + assert document["result_note"] == "Decision only; no device run." + assert document["result_note_basis"] == "caller_reviewed" + assert document["trace"] == {"lines": [2, 5], "archive": True} + assert "session" not in document["trace"] + evidence, = document["materials"]["case.md"]["evidence"] + assert evidence == { + "use": "used", "basis": "archive_reviewed", + "event_id": document["events"][0]["id"], + "passage_line": 1, "passage_text": "Use the stable RoPE case.", + "decision_line": 5, "decision_quote": "I will use the stable RoPE case", + } + assert document["observed"]["used"] == 1 + assert document["candidates"] == [] + assert str(tmp_path) not in original.decode() + + review = _review(tmp_path / "review.json", decisions=[_decision( + use="rejected", ref="case.md", event_line=3, decision_line=5, + decision_quote="I will use the stable RoPE case")]) + import_search(campaign, archive_review=review, **inputs) + corrected = json.loads(path.read_bytes()) + assert corrected["materials"]["case.md"]["use"] == "rejected" + assert corrected["materials"]["case.md"]["evidence"][0]["use"] == "rejected" + + +class _Reader: + def __init__(self, text): + self.text = text + + def collect(self, query="", **options): + return { + "items": [{"ref": "package:manual", "revision": "sha256:" + "a" * 64, + "title": "manual", "kind": "knowledge"}], + "omissions": [], "context_omissions": [], + "selection": {"method": options.get("relevance_method"), + "cue": options.get("relevance") or query, + "scope": "generated-packaged-skills", "ranking_applied": True}, + "context_bytes": len(self.text.encode()), "context_budget": {"max_bytes": 4096}, + "contexts": [{"ref": "package:manual", "revision": "sha256:" + "a" * 64, + "content_bytes": len(self.text.encode()), "resources": [{ + "path": "skills/manual/SKILL.md", "role": "core", + "sha256": hashlib.sha256(self.text.encode()).hexdigest(), + "content": self.text, + }], "omissions": [], "sources": []}], + } + + +def _capture_trace(path, output, assistant_text="No adoption decision."): + rows = [{"type": "session_meta", "payload": {"id": "private-session"}}, + {"type": "response_item", "payload": {"type": "custom_tool_call_output", + "output": output}}, + {"type": "response_item", "payload": {"type": "message", "role": "assistant", + "phase": "final_answer", "content": [{"type": "output_text", + "text": assistant_text}]}}] + path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8") + + +def _make_capture(tmp_path): + campaign = tmp_path / "campaign" + create_campaign(campaign, "example") + capture, _ = capture_context(campaign, context_id="r1", reader=_Reader("Manual line.\n"), + relevance="manual", context_bytes=4096) + document = json.loads(capture.read_text(encoding="utf-8")) + output = assemble_context_text(document["contexts"]) + return campaign, output + + +def test_indexer_capture_is_prepared_and_unknown_without_return(tmp_path): + campaign, output = _make_capture(tmp_path) + trace = tmp_path / "rollout.jsonl" + _capture_trace(trace, output) + review = _review(tmp_path / "review.json", decisions=[]) + + path = import_search(campaign, number=1, trace=trace, lines=[2, 3], + knowledge_root=tmp_path / "corpus", context_ids=["r1"], + archive_review=review) + record = json.loads(path.read_text(encoding="utf-8")) + event, = record["events"] + material, = event["materials"] + assert event["capture"] == {"state": "verified", "source": "knowledge/r1.json", + "sha256": event["sha256"]} + assert event["request"]["relevance"] == "manual" + assert event["selection"] == {"method": "weighted-lexical-v1", "cue": "manual", + "scope": "generated-packaged-skills", + "ranking_applied": True} + assert material["preparation_state"] == "prepared" + assert event["delivery"]["state"] == "unknown" + assert material["delivery_state"] == "unknown" + assert record["materials"]["package:manual"]["use"] == "unknown" + assert str(tmp_path) not in path.read_text(encoding="utf-8") + + +def test_indexer_archive_review_matches_exact_visible_resource_and_return(tmp_path): + campaign, output = _make_capture(tmp_path) + trace = tmp_path / "rollout.jsonl" + _capture_trace(trace, output, "I will use Manual line for validation.") + use = { + "use": "used", "ref": "package:manual", "event_line": 2, "passage_line": 1, + "passage_path": "skills/manual/SKILL.md", "decision_line": 3, + "decision_quote": "I will use Manual line", + } + review = _review(tmp_path / "review.json", decisions=[use, use], capture_returns={"r1": 2}) + + path = import_search(campaign, number=1, trace=trace, lines=[2, 3], + knowledge_root=tmp_path / "corpus", context_ids=["r1"], + archive_review=review) + record = json.loads(path.read_text(encoding="utf-8")) + event, = record["events"] + material, = event["materials"] + assert event["delivery"]["state"] == "visible" + assert material["delivery_state"] == "visible" + assert event["delivery"]["visible_resources"] == [ + {"ref": "package:manual", "path": "skills/manual/SKILL.md"}] + assert event["delivery"]["passages"] == [{ + "ref": "package:manual", "path": "skills/manual/SKILL.md", "line": 1, + "text": "Manual line.", + }] + assert record["materials"]["package:manual"]["use"] == "used" + assert record["materials"]["package:manual"]["evidence"][0]["basis"] == "archive_reviewed" + assert str(tmp_path) not in path.read_text(encoding="utf-8") + + +def test_capture_only_archive_needs_no_trace_and_is_idempotent(tmp_path): + campaign, _ = _make_capture(tmp_path) + review = _review(tmp_path / "review.json", decisions=[]) + path = import_search(campaign, number=1, context_ids=["r1"], complete=True, + archive_review=review) + original = path.read_bytes() + import_search(campaign, number=1, context_ids=["r1"], complete=True, + archive_review=review) + assert path.read_bytes() == original + record = json.loads(original) + assert record["trace"] is None + assert record["coverage"] == "complete" + assert record["counts"] is not None + assert record["events"][0]["delivery"]["state"] == "unknown" diff --git a/tests/test_search_capture_delivery.py b/tests/test_search_capture_delivery.py new file mode 100644 index 00000000..513b6ccc --- /dev/null +++ b/tests/test_search_capture_delivery.py @@ -0,0 +1,37 @@ +import json + +import pytest + +from inferrsi.harness.capture_delivery import capture_return_texts, visible_capture_resources +from inferrsi.harness.knowledge_capture import assemble_context_text + + +def test_delivery_requires_a_versioned_resource_block_in_a_model_tool_return(tmp_path): + context = { + "ref": "package:case", "revision": "r1", "omissions": [], + "resources": [ + {"path": "case.md", "content": "Check residency before initializing."}, + {"path": "other.md", "content": "Second resource."}, + ], + } + doc = {"contexts": [context]} + block = assemble_context_text([{**context, "resources": context["resources"][:1]}]) + trace = tmp_path / "trace.jsonl" + + def output(text): + return {"type": "response_item", "payload": {"type": "custom_tool_call_output", + "output": [{"type": "input_text", "text": text}]}} + + trace.write_text("\n".join(json.dumps(row) for row in [ + {"type": "session_meta"}, output("knowledge/r1.json"), + output(block.replace("@ r1", "@ r2")), output(block.rstrip("\n")), + {"type": "response_item", "payload": {"type": "message", "role": "assistant"}}, + ])) + for number in (2, 3, 4): + texts = capture_return_texts(trace, (2, 5), {"r1": number}, ["r1"]) + expected = [{"ref": "package:case", "path": "case.md"}] if number == 4 else [] + assert visible_capture_resources(doc, texts["r1"]) == expected + with pytest.raises(ValueError, match="model-facing"): + capture_return_texts(trace, (2, 5), {"r1": 5}, ["r1"]) + with pytest.raises(ValueError, match="outside the round"): + capture_return_texts(trace, (2, 3), {"r1": 4}, ["r1"]) diff --git a/tools/evidence_inventory.json b/tools/evidence_inventory.json index 6fc71a38..d278badc 100644 --- a/tools/evidence_inventory.json +++ b/tools/evidence_inventory.json @@ -150,6 +150,10 @@ "kind": "review", "manifests": [], "external": ["Review summary only; no local freeze manifest or raw output bundle is recorded."] }, + "search-attribution-qwen-s2-b1-20261110": { + "kind": "review", "manifests": [], + "external": ["The raw Codex rollout and temporary local campaign ledger are not committed; bounded reviewed excerpts and relative source references are preserved in search/round-1.json."] + }, "skill-eval-v1-20261021": { "kind": "campaign", "manifests": [{"path": "plan/integrity.json", "schema": "bundle-sha256"}], From 06999c4842d218a535ce09ac8d9c329833b92d98 Mon Sep 17 00:00:00 2001 From: Zhou ziheng Date: Tue, 10 Nov 2026 21:24:02 +0800 Subject: [PATCH 6/6] Clarify the source and destination for archived search records Agent-assisted-by: Codex (implementer) Signed-off-by: Zhou ziheng --- evidence/README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/evidence/README.md b/evidence/README.md index 662775a2..edbf2fc4 100644 --- a/evidence/README.md +++ b/evidence/README.md @@ -22,8 +22,10 @@ evidence/ Populate per acceptance issue. This scaffold keeps only `.gitkeep` placeholders—no campaign artifacts or pinned SHAs in the template. -When the experiment studies knowledge retrieval or adoption, archive its -reviewed `search/round-.json` and the campaign-relative artifacts needed -to audit its claims. The [round import guide](../docs/dashboard.md#archive-evidence-for-readers-outside-the-session) -describes how to retain visible passages and decision evidence while keeping -the original session private. +When archiving a round that studied knowledge retrieval or adoption, copy +`search/archive/round-.json` from the local campaign directory to +`evidence//search/round-.json` in the repository. This is +the portable v2 record produced by `campaign import-search --archive-review`. +The campaign's `search/round-.json` is the local v1 window record. +Preserve the campaign-relative paths of any artifacts needed to audit the +round. See the [round import guide](../docs/dashboard.md#archive-evidence-for-readers-outside-the-session).