Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 178 additions & 0 deletions docs/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,184 @@ campaign-relative files; the recorder hashes them and does not copy them. The
caller assigns IDs and must serialize campaign record updates. Neither command
starts an agent or validates the experiment.

### Import one round's searches

A search import writes `search/round-<n>.json` inside an existing campaign.
The record can exist before a candidate or run. Read it directly with a JSON
viewer. The agent or host selects the trace for its own session and keeps the
original private. When the Desktop host exposes `CODEX_THREAD_ID`, the agent
can locate its own rollout by matching that exact thread ID in the filenames
under the local Codex sessions directory. A host can also supply the trace
path directly. The import command only opens the supplied file.

For example, set `campaign` to the campaign directory, `trace` to the local
Codex Desktop rollout JSONL and `corpus` to the absolute skill-library path
**as it appeared in that session**. The importer normalizes path components
such as `.` and `..` lexically. Use the root spelling recorded in the trace,
including any symlink names, so replay also works after the checkout is gone.
In the agent's first shell call of the round, save the next line number.
In separate shell calls, use the knowledge
directory normally. Once the completed tool events are visible in the rollout,
run the import from the agent's closing shell call:

```sh
# Start of round 1. Save the printed value as FIRST for the closing call.
awk 'END { print NR + 1 }' "$trace"

# Separate agent shell calls with the skill library as the tool working directory.
rg -Hn 'Operator tests passed' methods/inference-optimization-casebook/cases
cat methods/inference-optimization-casebook/cases/qwen-b300-rejected-directions.md

# End of round 1. FIRST is the value retained at the start of this round.
LAST=$(wc -l < "$trace")
inferrsi campaign import-search --directory "$campaign" --round 1 \
--trace "$trace" --lines "$FIRST" "$LAST" --knowledge-root "$corpus" --complete
```

For round 2 take a new first/last pair, even in the same session. Windows are
one-based and inclusive. If the host buffers events, import after they have
flushed. After an interruption the host can do the same handoff. Use
`--context-id <id>` once per `campaign capture-context` performed in this round.
These inputs can be combined with the rollout or used alone for an Indexer-only
round. Each capture is verified and its returned refs, revisions, request and
resource paths are recorded. Opening a resource and receiving a prepared context
are distinct observations. A bare `collect` leaves no such capture.

The importer reads Desktop `CommandExecution` events. It supports direct text
or filename-match searches with `rg`, literal `cat`, `nl -ba` and `sed -n N,Mp`
reads. Use `rg -Hn` for unambiguous filenames and matching lines. File results
are corpus-relative. A shell event establishes the command's recorded output;
the outer agent tool may have displayed a shorter excerpt. The default local
record keeps locators and omits source text, output and assistant replies. Multiline
retrieval scripts, unsupported pipelines and unparseable output leave
unresolved event line numbers. Other tools and scripted/remote
retrievals require separate evidence. Review query text and paths before sharing.

- `observed` counts recognized searches, distinct materials per search (hits),
distinct hit refs across searches, and distinct refs with supported use.
A later file read is a separate event. File paths and versioned Indexer
entries keep their original identities, with no inferred cross-tool alias.
- Supply `--complete` only when the selected inputs cover every in-scope
retrieval for the round. Otherwise only observed counts are known and
`counts` is null. A complete round without a search has zero searches.
With no inputs the command records missing coverage. A failed query or
unresolved supported command also makes totals unknown. For example,
`rg -C 2` without line numbers cannot reliably separate context text from
matching filenames. A failed multi-file `cat` may have read only some
operands. These commands retain an unresolved trace locator.
- Use and rejection apply to materials observed in the imported round.
A decision based only on a previous round's material is outside this
record's scope. The existing optimization log can supply evidence when
its iteration references the exact verified capture. If that log claims
use of a selected item whose text was omitted, the record retains the
claim under `conflicts`. Counting it as used requires separate evidence.
For a decision recorded
in an assistant message, a reviewer can pass `--used REF LINE` or
`--rejected REF LINE` with the actual material ref and the JSONL line of
the relevant message. This explicitly attributes use; the importer validates
the locator and never infers a decision from a read. Unattributed use stays
unknown. Distinct use and rejection decisions for the same ref are both retained
and it counts once as used. Candidate links carry the recorded evaluation verdict
when present. `evaluation_state` reports the source file's status, including
`missing` for an absent file and `corrupt` for invalid JSON.
- Serialize imports along with other campaign updates. Repeating an import
rebuilds the snapshot without adding events. To attach a later decision or
run, retain the same trace, start line, capture IDs and scope, and extend the
end line if necessary before the next round starts. Separate rounds cannot
claim overlapping windows or the same retrieval.

#### Archive evidence for readers outside the session

Use `--archive-review review.json` on the same import when a campaign calls
for independently readable evidence. Use the historical knowledge directory
as `--knowledge-root`. In the review file, identify the repository, revision
and repository-relative directory from the experiment's frozen inputs. These
are caller-declared source coordinates. The importer obtains the passages
from the recorded tool return, so it can also handle a corpus that the host
copied into a temporary directory.

For example, this reviewer input links a visible match to a passage of the
agent's actual decision. The line numbers for the event and decision locate
records in the private input. The `passage_line` field locates a line in the material.
Use a verbatim, contiguous decision excerpt. The importer checks the cited
event, passage and message before recording the reviewed relation.

```json
{
"task": "Decide what to check before combining three Qwen Ascend optimizations.",
"corpus": {
"repository": "InferMatrixLab/InferRSI",
"revision": "f6d200775d554d08fb73673833611dd6ac02d14b",
"root": "src/inferrsi/data/skill_library"
},
"decisions": [{
"use": "used",
"ref": "methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md",
"event_line": 73,
"passage_line": 18,
"decision_line": 108,
"decision_quote": "报告称 **RoPE + VAE fastpath + 融合 attention 在初始化时 OOM**,并建议先盘点初始化驻留。因此这是这次组合的首要风险,应先做资源检查,再决定是否进入组合试验。"
}]
}
```

```sh
inferrsi campaign import-search --directory "$campaign" --round 1 \
--trace "$trace" --lines "$FIRST" "$LAST" --knowledge-root "$corpus" \
--archive-review review.json --complete
```

The command keeps `search/round-1.json` as the local window record and
returns the portable file `search/archive/round-1.json`. The latter embeds
the evidence selected by this review. Supply all intended decisions on each
archive import. Existing `--used`, `--rejected` and optimization-log claims
retain their local meaning. In the portable record they appear under
`declarations`. Reviewed passage-to-decision links appear under
`materials[ref].evidence` and contribute to its adopted-material count.

In the portable file, inspect the model-visible tool return and its
passages before the adoption entry. A search can match three lines of one
file and still count as one material hit. A reviewed decision can use one
of those lines. Hit counts include references the command matched or the Indexer
selected. Use each event's `delivery` to see which of those references can
be placed in the agent's return. A material marked `ref_visible` has a
confirmed reference but no archived content excerpt for that material. A zero `observed.used` means there were no
reviewed positive uses. Each unattributed material retains `use: unknown`.
Runtime truncation describes what the agent received. Archival omissions
describe what the reader can inspect. An ambiguous host
return leaves delivery unknown. A verified Indexer capture establishes what
the Indexer prepared. Its campaign-relative source is the evidence for that
preparation. When a tool actually returned its rendered context, the review
file can also supply `"capture_returns": {"r1-diagnosis": 52}`, where 52 is
the line of the model-facing tool response in the round. Pass the corresponding
`--context-id r1-diagnosis`. Only complete resource blocks with matching
owner, revision, path and body count as visibly delivered. A return that just
names the capture file or prints its digest leaves delivery unknown.
To review the use of a delivered Indexer passage, use its return line as
`event_line`, the entry ID as `ref`, and its capture resource path as
`passage_path`. The one-based `passage_line` selects a line within that
resource. Cite the later assistant decision in the same way as for a file.

To include a round in repository evidence, copy the portable file to
`evidence/<campaign-id>/search/round-1.json`. Retain the campaign-relative
paths to cited `knowledge/`, candidate and run artifacts, or document which
are unavailable in the published bundle. Review the generated text,
commands and referenced artifacts under the [privacy rules](privacy-security.md).
The raw Codex rollout stays local. A missing candidate is expected for a
decision-only exercise. See the [Qwen round archive](../evidence/search-attribution-qwen-s2-b1-20261110/README.md)
for the generated evidence and its interpretation. To follow one material across
events and then see its reviewed uses, set `record` to the portable JSON:

```sh
ref=methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md
jq --arg ref "$ref" '{
returns: [.events[] | {event: .id, action, query,
passages: [.delivery.passages[]? | select(.ref == $ref)]}
| select(.passages | length > 0)],
decisions: .materials[$ref].evidence
}' "$record"
```

### Record the optimization story (campaign trace)

The **Optimization story** section is the campaign-level trace a reviewer reads
Expand Down
1 change: 1 addition & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Effect classes used below:
| Execute/inspect experiments | CLI `run`; `status`; `report` | Experiment-executing / read-only / report-generating | `run` executes caller-supplied commands and records evidence; `status` reads recorded progress; `report` generates HTML from recorded evidence |
| Visualize a recorded run or campaign | CLI `dashboard`; [dashboard](dashboard.md) | Report-generating / local read-only serving | Writes an offline HTML snapshot or serves an optional live view on loopback; does not run an agent or validate a claim |
| Capture a knowledge context | CLI `campaign capture-context`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Runs `collect` with a context budget and keeps the full envelope and resource hashes in a new `knowledge/<id>.json`; proves what was prepared, not what a model read |
| Import one round of knowledge searches | CLI `campaign import-search`; [round import](dashboard.md#import-one-rounds-searches) | Local evidence-writing | Reads a Codex rollout window or context captures. With `--archive-review`, retains excerpts of attributable tool returns and reviewed decisions for readers of the evidence bundle. |
| Record the optimization log | CLI `campaign log`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Validates and stores a new numbered campaign log version; the dashboard cross-checks it against captures and runs |
| Assemble an optimization trace | CLI `campaign assemble-trace`; [offline dashboard](dashboard.md#capture-the-optimization-trace) | Local evidence-writing | Copies an explicitly reviewed context into a new bounded campaign-root snapshot; caller owns redaction and narrative accuracy |
| Associate runs with a local campaign | CLI `campaign init` / `campaign record` / `campaign decide`; [offline dashboard](dashboard.md) | Local evidence-writing | Creates caller-named campaign, round and candidate records; references existing artifacts and never starts an agent |
Expand Down
8 changes: 8 additions & 0 deletions evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,11 @@ evidence/
```

Populate per acceptance issue. This scaffold keeps only `.gitkeep` placeholders—no campaign artifacts or pinned SHAs in the template.

When archiving a round that studied knowledge retrieval or adoption, copy
`search/archive/round-<n>.json` from the local campaign directory to
`evidence/<campaign-id>/search/round-<n>.json` in the repository. This is
the portable v2 record produced by `campaign import-search --archive-review`.
The campaign's `search/round-<n>.json` is the local v1 window record.
Preserve the campaign-relative paths of any artifacts needed to audit the
round. See the [round import guide](../docs/dashboard.md#archive-evidence-for-readers-outside-the-session).
29 changes: 29 additions & 0 deletions evidence/search-attribution-qwen-s2-b1-20261110/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Qwen S2-B1 search and decision record

In the [Qwen material-discovery exercise](https://github.com/InferMatrixLab/InferRSI/issues/380),
the agent considered combining RoPE, the VAE fastpath and fusion attention on
Ascend. It recommended checking initialization memory before attempting the
combination. The round produced a recommendation and had no candidate or new
device measurement.

Open the [generated round record](search/round-1.json) to trace that decision.

1. In `events`, `file-39` records the initial read of the Qwen case. The later
search `file-73` records the query, scope and tool response. Under
`delivery.passages` are three returned lines from the case. Line 18 describes
the reported initialization OOM. Line 19 describes a resource reservation
failure on a different host with a different combination.
2. In `materials`, find `qwen-ascend-rejected-directions.md`. Its `evidence`
links line 18 to the assistant's actual recommendation. Line 16 also has a
reviewed decision. Line 19 has no recorded adoption or rejection.
3. In `unresolved_details`, the broad search and the filename-filtering pipeline
explain the partial coverage. `counts` is null. The individual verified
passages and decisions remain available in the same file.

The [round outcome](results/summary.json) summarizes the decision and its limits.
The corpus was frozen at [InferMatrixLab/InferRSI revision f6d2007](https://github.com/InferMatrixLab/InferRSI/tree/f6d200775d554d08fb73673833611dd6ac02d14b/src/inferrsi/data/skill_library).
The reviewed passages come from three public sources.

- Cohort, fusion attention and initialization residency. [Qwen Ascend case, lines 6, 16 and 18](https://github.com/InferMatrixLab/InferRSI/blob/f6d200775d554d08fb73673833611dd6ac02d14b/src/inferrsi/data/skill_library/methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.md#L6)
- Applicability conditions and evidence limits. [Typed case, lines 15 and 17](https://github.com/InferMatrixLab/InferRSI/blob/f6d200775d554d08fb73673833611dd6ac02d14b/src/inferrsi/data/skill_library/methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json#L15)
- Conditions for a retry and guidance on adapting a precedent. [Casebook procedure, lines 96 and 103](https://github.com/InferMatrixLab/InferRSI/blob/f6d200775d554d08fb73673833611dd6ac02d14b/src/inferrsi/data/skill_library/methods/inference-optimization-casebook/SKILL.md#L96)
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"title": "Qwen S2-B1 search archive",
"campaign_id": "search-attribution-qwen-s2-b1-20261110",
"record_path": "search/round-1.json",
"decision": "The historical answer treated an author-reported triple-stack initialization OOM as a reason to check initialization residency before deciding whether to combine the optimizations.",
"outcome": {
"candidate_count": 0,
"new_device_measurement": "none"
},
"coverage": {
"state": "partial",
"counts": null,
"reason": "The broad-search return was truncated. The filename filter has verified path excerpts and an unknown hit count."
}
}
Loading
Loading