Repository navigation
feat: share RSI optimization management across project and benchmark tasks - #338
Bounty-hunter wants to merge 1 commit into
Conversation
Dong1017
added
the
area:harness
ChatGPT Codex Connector
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Scoped draft review of the new CPU benchmark measurement path against #252 and the campaign no-output-replay rule in #30. P2 benchmark-validity concern is attached inline: the precheck and timed samples reuse the same input object/content, so a memoizing candidate can appear faster without performing the operation on each call. Please address or explicitly constrain this before treating timing as optimization evidence. Static source review only; no tests, model runs, hardware measurements, or acceptance decision made here. The initial task subset remains subject to the recorded owner decision.
| if profiler: | ||
| profiler.enable() | ||
| started = time.perf_counter_ns() | ||
| output = compute(x, eps=oracle["eps"]) |
There was a problem hiding this comment.
P2 — measured calls can become cached-output replay. The precheck at line 94 already invokes compute(x), and this loop passes the same x object and values to every warmup/sample. A candidate with a process-local id(x)/content → output memo can pass every value/input-preservation assertion while most timed calls only return the stored output; the reported suite_latency_ms then measures replay, contrary to the campaign's no-final-output-caching boundary (#30). Suggest preparing a frozen, paired varying input sequence outside the timing boundary (same sequence for baseline and candidate) and validating each invocation; add a stateful memoizing-candidate negative regression. Public development workloads still cannot establish hostile-code resistance or held-out generalization.
There was a problem hiding this comment.
Addressed in ce84506e7106c4fc48fafdfab55d2ea2f9dffd50 with numpy-axis1-v2. Inputs/oracles are prepared before candidate invocation and retained for the complete action; each timed input has distinct bytes from precheck/warmup/other samples. Both arms and all consumers share the same deterministic per-call sequence. Each sample is bound to an input hash. The invariant zero case remains correctness-only, so the new score covers five varying workloads; old task/grader identities and evidence are preserved.
Regression controls confirm exact id/content memo implementations have no cache hits in the sequence, while shape/layout output replay is rejected without a metric. The fix does not claim hostile-code resistance. Forty focused tests and related CLI/SkillEval tests passed; actual Workbench and offline verifier execution retained 300 samples with matching planned and actual inputs. Statistical resolution remains unqualified.
Fix and validation · RSI consumer boundaries. Please re-review; this response does not mark the thread resolved or claim maintainer acceptance.
There was a problem hiding this comment.
The specific cached-output replay finding is addressed in ac03ae23; this is a scoped source confirmation, not approval of the whole draft.
- Inputs and independent oracles are prepared before candidate calls. All input objects stay alive, and later inputs must have distinct bytes; the zero case is correctness-only and five varying cases contribute timing.
- The new regressions require both identity-keyed and content-keyed memoization to miss, and reject stale shape/layout output replay without a usable performance metric. Per-sample input hashes and the frozen task/grader bindings are retained.
- @Bounty-hunter remains the implementer. Full-PR review, v2 timing/noise qualification and the recorded parent scope decision remain open; this confirmation establishes neither reliable performance ordering nor optimizer/skill benefit.
Evidence and limits
Assessed head: ac03ae23ce0bb5864a55c2c97d5f3d8d3b550d39; API base and actual main: 663f3b327998526a13aa31e72ad729025c18983c.
Read the current task contract, NumPy evaluator, worker, consumer/verifier bindings, both task manifests and the regression definitions, including input preparation and memo/replay controls. This is not a completed review of all 42 changed files.
The Python 3.11 hosted job reports Ruff success and 3,584 passed / 13 skipped / 284 subtests. It checked out merge 396688581967ea85d8edf2f7f169925546c5e207 (current head into the main above); the API comparison returned zero file differences from the standalone head. Draft qualification was skipped. I did not rerun tests or independently inspect the author's local 300-sample archive. No model/device execution or scope acceptance occurred.
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Provider/settings: Uncovered / unsupported paths (108)
|
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Incremental review at fd0f396c of the formal artifact integration since ac03ae23: one P2 correctness blocker, attached inline. The new built-in-only adapter can resolve a same-named working-directory task instead of the packaged task, and its freeze/revalidation path preserves that substitution.
The earlier cached-output replay finding remains addressed; I am not reopening or duplicating it. I inspected the new adapter, reused task/sandbox contracts, controller/run-trial/local-capture/report changes, regression definitions and CI wiring. Current Ready CI succeeds: the dedicated Linux controls report 49 passed with strict sandbox availability enabled; the 3.11 full check reports 3,631 passed, 18 skipped and 284 subtests. Actual checkout 98156741 has zero file differences from this head. Those tests do not cover cwd task shadowing.
Static source review plus existing hosted logs only; no local candidate/test execution, model or hardware work. Bounty-hunter retains the correction. Correctness-only limits, performance/skill-benefit gaps and #252's outstanding acceptance decision remain separate; this review does not accept the broader goal.
| def task_descriptor(name: str) -> dict: | ||
| if name not in builtin_tasks(): | ||
| raise ValueError("Only built-in shared CPU benchmark tasks are supported") | ||
| package = load_task(name) |
There was a problem hiding this comment.
P2 — resolve formal tasks from the packaged resource, not the working directory. Checking that name occurs in builtin_tasks() does not ensure that load_task(name) loads that built-in package: contract.load_task first prefers any existing Path(name) directory. validate_task(), starter() and verify() repeat the same lookup, so they all agree on a shadow rather than detect it.
For a concrete source-trace case, run from the InferRSI checkout with a local softmax-cpu/ copy of the packaged task and change only its oracle.atol to 1. Generate the descriptor through task_descriptor('softmax-cpu'). The manifest validator permits that positive finite tolerance. prepare() copies this same package, freezes its checkout-relative paths, and load_plan() compares against the same shadow, so the paired path also accepts it. A compute returning np.zeros_like(x) then satisfies all four output checks for every input: true softmax values lie in [0,1], so np.allclose(0, expected, atol=1) accepts them. The canonical packaged files need not change. This is static tracing/arithmetic, not a claim that I executed the candidate.
This makes a built-in-only, independently frozen correctness task depend on the caller's cwd and allows an incorrect artifact to pass under an accidentally or deliberately substituted rubric. It is separate from the documented public-suite/protocol-spoofing limits. Please use an explicit packaged-resource/built-in-only loader throughout the formal adapter and its prepare() dependency collection, while retaining path loading for the trusted Workbench API. Add a regression with a same-named cwd task directory that proves the canonical oracle/hash is used or the shadow is rejected.
There was a problem hiding this comment.
The specific cwd-shadowing finding is addressed at 6308268fc6d1fbf90e9ab97b5da8af79152f39ea (fix introduced in 00c41b24). This clears this finding only.
load_builtin_task()resolves the packaged resource and checks its manifest ID. The formal descriptor, starter and parent verifier now usetask_package(); prepare copies and freezes dependencies through that same canonical loader. Revalidation compares the retained package hash and starter against the canonical descriptor. Trusted Workbench path loading remains available.- The new regressions cover a same-named
softmax-cpu/cwd directory withatol=1, canonical descriptor/starter/oracle selection, prepare/dependency freezing and later revalidation. The real sandbox control rejects the zero-output implementation despite that shadow. The formal CPU path also rejects the new NPU adapter before execution. - I read the exact-head source and dedicated Linux control log: 54 passed, zero skips, with
INFERRSI_REQUIRE_BENCHMARK_SANDBOX=1and NumPy 2.2.6. The 3.11 full check reports 3,649 passed / 20 skipped / 284 subtests. CI checked out merge6f1506bc; its tree has zero file differences from this head. I did not execute code locally.
The separate NPU action/total-timeout finding remains open and is not superseded by this reply. This is not whole-PR approval, device qualification, performance/skill-benefit evidence, or scope acceptance; Bounty-hunter retains the implementation.
Bounty-hunter
changed the title
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Scoped incremental static review of the new rmsnorm-npu pilot in PR #338 (head 6308268f), against the execution limits recorded in Issue #337. One P2 execution-budget contract gap, attached inline: the 180-second action timeout is applied to standalone check/profile/negative controls but not to the A/A or reference Workbench trials. This can leave an individual trial running until the 8-minute comparison deadline; the script also does not itself enforce the separately stated 20-minute pilot envelope. Please bind the per-action and total deadlines or identify and verify the outer enforcement, with a timeout-control regression. This is a source/contract finding, not a hardware result. I inspected the new NPU adapter/driver, RunSpec, the Workbench runner/external/local executor chain and relevant tests. The earlier canonical-task shadowing review remains separate and is not repeated. No NPU/model/test execution or acceptance decision was performed.
| comparisons = {} | ||
| # Same implementation in both arms is the explicit no-op/A-A control. | ||
| for name, path in (("aa", baseline), ("reference", candidate)): | ||
| spec = task.runspec(path, repo=args.repo, output_dir=output / "results" / name, |
There was a problem hiding this comment.
P2 — the frozen per-action timeout is not applied to paired NPU trials. Issue #337's predeclared pilot says --timeout-seconds 180, two Workbench comparisons capped at 8 minutes each, and a 20-minute total runtime. Here args.timeout_seconds is passed only to run_action() for check/profile/negative controls (lines 96–116); task.runspec() does not carry it. run_experiment() passes its whole-comparison deadline to execute_trial() → execute_command(), which calls process.wait(timeout=deadline-now). Thus a first stalled A/A or reference trial may occupy almost the entire 480-second comparison instead of stopping at 180 seconds; two comparisons plus all standalone controls can also exceed 20 minutes unless an independently enforced outer deadline exists. That changes the frozen resource/lifecycle budget, particularly on a shared NPU allocation. Please cap each Workbench trial at min(comparison_remaining, action_timeout, pilot_remaining) (or implement equivalent tested enforcement), and include a deliberately nonterminating synthetic command in a CPU-only timeout regression. If a scheduler supplies the global bound, record its enforceable receipt and distinguish that from this script's guarantees.
Bounty-hunter
changed the title
|
Review at head
|
Bounty-hunter
changed the title
|
Incremental review This delta: the 12 files under Still open:
Static doc review of the delta only; I did not run the NPU tooling. |
|
Incremental review This delta looks consistent:
Still open:
Static review of the delta only; I did not run the tests or the NPU tooling. |
|
Incremental review This delta:
Still open:
Static review of the delta only; I did not run the tests or the NPU tooling. |
Bounty-hunter
changed the title
|
Incremental review This delta: adds the Findings / questions:
Static review of the delta only; I did not run the tests, the NPU tooling or the MoE evaluator. |
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Scoped follow-up at 2eed487a to Amy's CI question: the evidence-inventory count is now repaired, but both current-head full checks still fail in detached-worker cleanup. The new diagnostics identify a process-discovery failure before any worker is signalled; the P2 below needs correction. The previously reported malformed-field classification and RMSNorm timeout findings remain separate and are not cleared by this review.
CI evidence and scope
- Inspected head:
2eed487a1edcac30a3a32b5a504bc7b013e9616b; incremental commit only changes inventory totals and assertion diagnostics. - Python 3.10 log and Python 3.11 log: each reports 1 failed, 3,682 passed, 19 skipped. The sole failure is
test_deadline_cleans_detached_workers_but_preserves_unrelated_process, withverified: false,signalled_pids: [], and permission denied reading a same-user/proc/.../environ. - Read the affected cleanup code, test and hosted logs. No local execution, hardware/model run, or new spending.
| fields = (entry / "stat").read_text().rsplit(")", 1)[1].split() | ||
| if fields[0] == "Z": | ||
| continue | ||
| if marker in (entry / "environ").read_bytes().split(b"\0"): |
There was a problem hiding this comment.
P2: An unreadable same-user process prevents cleanup of the invocation's workers. _members() scans every same-UID process, and a permission error reading any environ propagates before it returns its accumulated matches. cleanup() therefore reaches its outer exception handler without signalling even the marked workers it could inspect. Both current-head Linux jobs reproduce this after the launcher times out: verified=false, signalled_pids=[], with /proc/.../environ permission denied. This can leave detached workers running and makes the standard Linux qualification fail. Scope discovery to an inspectable invocation-owned process set, and ensure a partial scan still attempts cleanup of positively identified workers. Preserve an inconclusive result when absence cannot be established; do not merely ignore unreadable entries and report verified success. Add coverage for an unreadable unrelated same-user process.
There was a problem hiding this comment.
Scoped follow-up at 8f3aebd2e1359d11d8bcdd034f5dacedf9a1f12d: the original hosted-CI failure is cleared, but this P2 is only partially fixed.
Both Python 3.10 and 3.11 now report 3,688 passed, 19 skipped. Skipping strictly older processes is sound for the fresh marker. However, an unreadable same-tick or newer process still prevents cleanup of workers already positively identified earlier in the scan.
I reproduced this with a synthetic procfs and mocked os.kill (no real signals): driver start tick 100, readable exact-marker worker PID 990001 at tick 101, then an unrelated same-UID process whose environ raises PermissionError. With the unrelated process at tick 100 or 101, cleanup() returns verified=false, signalled_pids=[], and the mock receives zero calls. At tick 99, it skips that unrelated environment and attempts to signal the known worker.
_members() still throws away its accumulated matches when the later read raises; cleanup() catches the exception outside the signalling loop. The new unreadable-process test has no owned worker, so it does not cover this consequence.
Please preserve discovered matches alongside scan errors, attempt cleanup of the PID/start-time-verified owned workers, and retain an inconclusive result for unreadable entries. Add the mixed owned-worker/unreadable-process case. This is the remaining part of the original finding, separate from Amy's malformed-field and whole-pilot timeout findings.
|
Incremental review This delta bumps the evidence inventory totals in Findings:
Static review of the delta plus the CI logs; I did not run the tests or the MoE evaluator. |
|
Incremental review This delta addresses item 1 of that review in a reasonable way:
Remaining items:
Static review of the delta plus surrounding source only; I did not run the tests or any NPU tooling. |
Structural question: two task libraries, two entry styles, two identity freezesReviewed at
The MoE addition is documented as a deliberate separation, not an oversight: 1. Two parallel content libraries with the same file roles
Both hold the same kind of thing: a frozen task manifest, a baseline, a correctness oracle, a license and provenance. The stated distinguishing axis is single-operation versus distributed layer. That reads like a property a task declares, not a reason for a parallel tree — and Two costs are visible in this diff rather than hypothetical. The practical consequence: an agent has to know which library a task lives in before it can choose the invocation. For a task library that is the thing you most want to avoid. 2. Each entry point verifies something different, and the differences are not visible from the imports
Its imports are These two are not interchangeable: What is worth checking is that reusing a shared execution helper conveys no upper-layer guarantee. Per entry point, as read at this head:
That table is the useful artifact, and it is also the point: filling it per module produces the wrong answer, which is how I got the original claim wrong. It has to be filled per entry point. Nothing here says any of these entry points needs source-snapshot adjudication — 3. Three low-cost suggestionsNone of these requires restructuring what is already here:
4. What this comment does not claim
|
8f3aebd to
3e4aa1f
Compare
|
Re-review at head What changed since
Blocking / open
Questions on the new scenarios (static reading only)
Static review of the diff only; I did not run any NPU tooling or the new tests. |
3e4aa1f to
4afcbfb
Compare
Bounty-hunter
changed the title
|
Re-review at head What changed since
Resolved or moot
Blocking / open
Static review of the interdiff and the cited sources only. I didn't run the tests or any NPU tooling. |
4afcbfb to
f2c3e7a
Compare
|
Re-review at head What changed since
This split looks cleaner to me. Each package now owns its entry and its assessment, and the CLI only does discovery and launch. Blocking / open
Dong1017's earlier CHANGES_REQUESTED still needs their confirmation in the existing thread. |
f2c3e7a to
803d04b
Compare
|
Re-review at head What changed since
Still open
Dong1017's earlier CHANGES_REQUESTED still needs their confirmation in the existing thread. |
803d04b to
c49e4b6
Compare
|
Re-review at head What changed since
The registry and validation look good to me. Still open
|
c49e4b6 to
4a11090
Compare
|
Re-review at head What changed since
Moving the harness dependency out of the task, so the Benchmark core no longer imports Harness, looks right to me. Still open
|
b700ae0 to
21b0a4f
Compare
|
Re-review at head What changed since
The plan/state digests, per-round retained hashes and the pending-attempt guard are a solid design. Still open from earlier reviews
New in this push |
Bounty-hunter
changed the title
21b0a4f to
8baed1a
Compare
|
Re-review at head What changed since
The single-dispatch reservation, idempotent bind, the definite Still open from my previous review (unchanged code)
New in this push Static review of the diff and surrounding code; I didn't run the tests or any host/NPU tooling. Reviewed commit range: |
Bounty-hunter
changed the title
8baed1a to
0b7883e
Compare
Bounty-hunter
changed the title
|
Re-review at head What changed since
New blocking issue
Smaller / questions on the new code Still open from my previous review (code unchanged in this push)
Static review of the tree diff plus the surrounding code. I ran only the cwd-write reproduction above (Linux, Python 3.13, against the PR's own fixture). I did not run the full suite or any NPU task. Reviewed: |
0b7883e to
88bef30
Compare
|
Re-review at head This delta (docs only)
Still open (code unchanged since my previous review)
This is a static review of the tree diff, checked against the surrounding code. I didn't run tests this time, since the code is unchanged. Reviewed: |
Package MoE, Qwen-Image denoising and real-model Qwen3-TTS tasks with task-owned measurement and assessment. Share Harness budget, candidates, feedback, selection and final retesting between packaged tasks and ordinary multi-file projects with explicit caller-provided acceptance conditions. Support optional fresh native-host conversations for benchmark tasks while ordinary projects default to the current Agent. Retain handoff identity, observed usage and deliverables separately from task acceptance. Expose shared records and final outcomes in the dashboard, and document component boundaries, explicit conditions and both session paths. Agent-assisted-by: Codex (implementation, source research, documentation, review and validation) Signed-off-by: dengyunyang <584797741@qq.com>
88bef30 to
3cbb74f
Compare
|
Re-review at head What changed since
My earlier items 8 (final not checking the linked job) and 10 (job-link atomicity) no longer apply now that New in this push
Still open (code unchanged in this push)
This is a static review of the tree diff, checked against the surrounding code at this head. I didn't run the tests or any NPU/host tooling this time. Reviewed: |
本 PR 做什么
新增一套由当前 Agent 逐轮驱动的优化过程管理，让普通项目任务和内置题库任务共用候选保存、预算、反馈、选优和最终复测；复用原有 Harness 执行器、知识留存与 Dashboard。
当前提交：
3cbb74f5f87ba4bbfe54a972a9e9e9b86e5b0ab3。PR 保持 Draft、一个 commit。采用 #362 已接受的模块职责方向；本次没有新增 Apps 包，也没有完成整个仓库的架构迁移。“共用”仅指本 PR 接入的普通项目和题库两条新入口。原有
run → runner.run_experiment()没有改为调用evaluate_round()；旧手动 campaign、SkillEval 等路径也没有自动迁入新控制器。 这次是新增上层管理并复用已有设施，不能把新流程的测试当作整个旧 Harness 已完成统一或回归覆盖。各模块的功能与边界
assess()只读结果，不再运行模型arguments.py、cli.py；新增benchmark_operations.py，连接模块并注入回调session_command()、_session_execute()、_session_assess()均为本 PR 新增，并非原有函数stdout.log、stderr.log、执行记录、文件身份“固定条件”是保存明确给出的脚本、输入和参数，并在后续操作中检测变化。它不表示操作系统只读锁、可信执行沙箱，或已经隔离所有外部模型、数据和环境。
例如“优化 vLLM 吞吐，输出质量不能下降”，仍需 Agent/用户明确模型、精度、请求集、并发、输入输出长度、计时口径和质量标准，并落实到脚本。普通项目适配器依据这些脚本的实际结果判定，不自动证明质量检查充分。服务吞吐要求也不能用单算子或一个 forward 的加速代替。
实际接入点：哪些新增，哪些复用
flowchart TD O["原有命令：inferrsi run"] -->|"解析原有 RunSpec"| R["原有 runner.run_experiment"] N["新增命令：campaign round/final<br/>benchmarks session round/final"] -->|"解析任务、候选和假设"| C["新增 session_command"] C -->|"注入 execute / assess 回调"| H["新增 Harness evaluate_round"] H -->|"提交命令、工作目录和期限"| A["新增执行适配函数"] R -->|"原有基线/候选命令"| X["复用 execute_trial / execute_command"] A -->|"项目检查命令 / 题包评测命令"| X X -->|"保存并返回"| F["退出状态、耗时、stdout/stderr<br/>脚本输出文件路径"]session_command()run不自动转入新控制器harness.scenarios.external.execute_trial()和 Infra 执行器；前者补充环境参数透传runner.run_experiment()knowledge_capture.capture_context()，外层增加预算/知识开关检查改动前：原有一次实验及手动关联
以下只比较本 PR 面向的项目/工程任务路径，不声称仓库原来没有任何评测或优化能力。箭头说明每步操作，返回箭头或紧随其后的动作说明产物。
sequenceDiagram actor U as 用户 participant A as 当前 Agent participant K as Knowledge participant H as 原有 Harness U->>A: 提出目标和约束 A->>A: 读取入口并准备基线、候选、负载、质量命令和 RunSpec 参数 loop 调用方组织各次尝试 opt 需要资料 A->>K: 提交当前问题的检索请求 K-->>A: 返回内容、来源和版本 end A->>A: 分析与按需 profiling，产生修改后的候选代码 A->>H: inferrsi run：提交本次对照实验参数 H->>H: run_experiment：保存 run.lock.json、环境和源码身份 H->>H: 经原有执行器运行基线/候选及质量命令，保存测量、日志和退出状态 H->>H: 原有 evaluate 与 write_report：生成 evaluation.json、状态及报告 H-->>A: 返回本次 run 目录与结果 opt 调用方显式关联历史 A->>H: campaign init/record/decide/log：提交轮次关联和决策 H-->>A: campaign.json、rounds.jsonl、候选引用及显式日志 end A->>A: 根据反馈选择下一次尝试，自己衔接跨轮预算和最终版本 end A->>H: 对选定版本另调用 run 做最终检查 H-->>A: 新的一次实验目录及其判定 A-->>U: 汇总代码、各次结果和未验证范围这条旧路径保留。本 PR 的新流程是下述普通项目与题库两条入口，底层执行器与旧路径复用。
改动后：普通项目完整流程
下图将原来的初始化、知识查询、每轮执行与判定合并，并一直画到独立 final 和结果交付。普通项目在当前 Agent 会话执行；题库默认使用全新会话的差异流程见下一节。
先准备条件，再调用 start；文件复制和状态初始化发生在 start 内部。 基线性能数值在后续评测时才产生。图中的 loop 表示 Agent 连续发起命令，Python 不会自动启动下一轮；final 通过相同执行链重新测量已保存的最佳版本。
session_command()及执行/判定适配函数为本 PR 新增。Harness 通过注入的回调交回适配层，再调用 Benchmark；核心模块不直接互相导入。这里详细展开普通项目的project.steps/inspect_step/assess，题库用自身的命令与判定协议替换这些步骤，复用同一个 Harness 控制器。图中按组件组织：Harness 只画一列，同时表示优化过程管理、候选/状态记录和受控执行。底层 Infra 执行器作为 Harness 使用的实现设施展示在内部动作中；新增与复用的区别放在图下说明。
sequenceDiagram actor U as 用户 participant A as 当前 Agent participant C as CLI 入口与适配层 participant H as Harness participant B as Benchmark participant K as Knowledge U->>A: 提出项目优化目标和约束 A->>A: 先读 RSI 入口，准备任务说明、输入、质量/性能脚本、指标和门槛 Note over A,H: 第一阶段：start 内保存任务与基线，初始化本次运行；此时不测量性能 A->>C: campaign start：提交文件、条件、源码范围、预算和停止规则 C->>C: session_command：检查实验新目录、项目范围及参数 C->>B: project.prepare：校验条件，复制任务说明和评测目录 B-->>C: frozen/TASK.md、frozen/protocol/、配置和文件哈希 C->>H: candidates.retain：复制声明范围内的原始代码 H-->>C: frozen/baseline/ 与源码身份 C->>H: initialize：建立计划、期限、空轮次和初始最佳候选 H-->>C: plan.json、campaign.json、state.json C-->>A: 返回项目编辑路径、实验状态和下一步说明 Note over A,K: 第二阶段：Agent 逐次发起若干 round，结束搜索后再发起一次 final loop 每次命令独立进入相同的执行与判定链路 alt 继续搜索：准备一轮候选 A->>A: 分析反馈，按需 profiling，明确本轮问题 opt 需要资料且知识开关允许 A->>C: campaign knowledge：提交查询、检索方式和上下文预算 C->>H: capture_context：传入 KnowledgeReader H->>H: 检查运行状态、预算和知识开关 H->>K: knowledge_capture 经 reader.collect 检索 K-->>H: 返回资料正文、版本、来源和遗漏信息 H->>H: 保存 knowledge/context-….json，更新 state.contexts 引用 H-->>C: 返回留存引用与正文 C-->>A: 返回可阅读的资料，供本轮分析使用 end A->>A: 提出假设，修改声明范围内代码，形成候选 A->>C: campaign round：提交候选路径、假设和可用用量 else 搜索结束：独立复测 A->>C: campaign final：提交停止原因和可用用量 Note over A,H: final 使用留存的最佳版本；工作区未提交修改不参与 end C->>H: evaluate_round：传入 round/final 标记，注入 execute 与 assess 回调 H->>H: 核验文件、状态和预算，选择提交候选或留存最佳版本 H->>H: 保存到 rounds/… 或 final/…；state.pending 记录本次待完成尝试 H->>C: 调用 execute 回调，即 _session_execute → _project_execute C->>B: project.steps：根据固定配置生成本次评测步骤 B-->>C: 基线/候选的质量、预热、交替测量命令 loop 按固定步骤采集；拒绝或异常时停止本次采集 C->>H: execute_trial：提交命令、源码目录、环境和期限 H->>H: 调用底层 execute_command 启动脚本，脚本检查质量/计时并写 result.json H-->>C: 执行状态、退出码、耗时、stdout/stderr 与结果路径 C->>C: 核对本步骤前后的源码身份，记录 source_checks C->>B: inspect_step：读取本步结果，检查格式、质量字段和性能值 B-->>C: 返回继续采集或拒绝/异常及原因 end C-->>H: 返回 execution；各步结果、日志和汇总执行记录已落盘 H->>H: _settle：再次核验固定条件和候选身份 H->>C: 调用 assess 回调，即 _session_assess C->>B: project.assess：提交本轮目录、固定配置和执行事实 B->>B: 核对完整性与质量，重算中位数/提升比例，比较门槛 B-->>C: assessment：判定、指标、质量标记、证据引用和原因 C-->>H: 返回 assessment；此步骤不重新测量 H->>H: 写 observation.json；round 更新最佳候选，final 记录独立复测结论 H->>H: 更新 state.json、summary.json；执行不明/缺证据时标记 blocked H-->>C: 返回本次结果、最佳快照、剩余预算或 final 状态 C-->>A: Agent 读取反馈，决定下一轮、处理阻塞或交付 end Note over A,H: 第三阶段：读取实际完成的记录，展示并交付 A->>C: dashboard：读取实验目录 C->>H: Dashboard 读取本次运行的计划、轮次和 final H-->>C: 页面/离线 HTML，含优化曲线、已观测用量与 final 结果 C-->>A: 返回展示产物；缺失数据保持未知 A-->>U: 交付最佳代码位置、最终结果、曲线、消耗及未验证范围这张流程图中，本 PR 改了什么
optimization.py、candidates.py，管理初始化、轮次、预算、候选快照、选优、恢复及 finalexecute_trial()和 Infra 的execute_command()；前者补充环境参数透传。图中的启动进程、超时和日志能力来自这条原有执行链knowledge_capture.capture_context()；扩展原有 Dashboard 读取新运行记录并展示曲线/finalsession_command()及执行/判定适配函数，连接新流程与 Benchmark 的任务准备、步骤和判定接口以上是同一流程内部的新旧实现关系。原有
run → runner.run_experiment()仍未转入新增优化控制器，不能由组件合并展示推导出所有旧路径已经统一。初始化步骤及产物
project.prepare()frozen/TASK.md、frozen/protocol/；返回文件哈希及指标/门槛配置candidates.retain()--include声明的源码/配置，核对复制前后身份frozen/baseline/；后续作为原始实现参与对照optimization.initialize()plan.json固定计划，campaign.json标识运行，state.json保存动态状态；这里不测量性能work/candidate.py这些目录由代码规定，实验根路径由调用者提供。普通项目和题库共用三个控制 JSON，但
frozen/的内部布局不同。plan.json当前同时保存任务条件与运行配置；这不表示任务规则和动态状态属于同一职责。知识查询的边界
查询由 Agent 根据当前问题发起，可以多轮再次查询；没有“第一轮自动加载全部 skill”的规则，也不强制每轮检索或 profiling。知识关闭时该受管入口拒绝查询，不能据此证明 Agent 没有从其他来源获得知识。
执行与判定的边界
evaluate_round()rounds/round-NN/attempt-…/或final/attempt-…/下的候选副本；state.pending_session_execute()/_project_execute()execute_trial()/execute_command()stdout.log、stderr.log、execution.json；脚本生成结果 JSONinspect_step()_settle()assess；本次该参数就是_session_assess_session_assess()assess()assess()assessment字典assessment→ 本轮observation.json、全局state.json、summary.json“正确性、性能、有效性与拒绝原因”在普通项目中具体是：
INCONCLUSIVE。FAIL；基线不通过则无法建立有效对照。这里不重新推理质量含义。missing_or_invalid_result、missing_boolean_quality_evidence、invalid_metric、performance_gate_not_met。Qwen 题包的
assess()则核对全部 case、固定采样参数、源码身份、逐步误差和调用次数，并从原始时间样本重算综合加速比。实际张量比较发生在此前运行的题目脚本里。该题当前PASS表示完整有效且正确的评测；加速比低于 1 的有效结果也可返回 PASS，由 Harness 决定它能否替换当前最佳，不能把 PASS 一律解释为已加速。结果存在哪里
frozen/TASK.md、frozen/protocol/、frozen/baseline/frozen/下保存完整题包，包含baseline.py、评测/参考/输入脚本--repo指定的项目目录work/candidate.pycandidate/多文件快照candidate.pyexecution/<step>/result.jsonmeasurement/result.json、源码快照与身份文件execution/execution.json及逐步执行记录/日志execution/execution.json及日志observation.json；运行根目录更新state.json和summary.jsondashboard读取上述记录，生成性能/用量曲线及独立 final 展示up_to是轮次上限，结束必须说明原因；exact要求完成固定轮数。final 不采用工作区里未提交的修改。操作成功退出不等于任务 PASS；应读取 assessment。recover只读取已留存的待完成执行再结算，不重新启动测量。题库：先派发到新会话，再完整执行 RSI
sequenceDiagram participant A as 发起 Agent participant B as Benchmark participant H as Harness participant N as 当前宿主 participant E as 同一种 Agent 的全新会话 A->>B: list/show 解析题目，明确预算、采样和知识开关 B-->>A: 任务定义与使用要求；尚未创建优化实验 A->>H: agents prepare/dispatch：保存任务指令并登记一次创建机会 H-->>A: prompt.md、request.json、job 状态和创建许可 A->>N: 创建新会话并发送任务，不复制父对话 N->>E: 交付任务与配置 N-->>A: 原生会话 ID 和回执 A->>H: agents bind：保存原生会话关联 E->>E: 读取 RSI 入口与题目，自行执行 benchmarks session start Note over E,H: 从初始化到检索、候选、round、final 和报告，完整复用上图；任务规则来自题包 A->>N: 观察同一新会话，获取返回结果和实际可用用量 N-->>A: summary、候选、final 证据及宿主回执 A->>H: agents record：保存回执和返回的交付物 A->>A: 核查交付结果并汇报，不重复执行强制 final通用 Agent job 与优化实验是两个独立记录：前者保存派发及宿主状态，后者由执行会话创建并管理。任务指令写明运行时/题目版本、入口、环境和预算；通用
agents prepare保存 prompt，不自动校验题包版本，执行 Agent 必须在 start 前核对。宿主完成不代表任务通过，缺少 final 时仍须报告不完整。调用位置与新增/复用对照
以下链接固定到当前 PR 提交，可从图里的步骤直接找到实现：
本次交付的具体能力
campaign ...与题库benchmarks session ...进入同一个 Harness 控制器Qwen 使用随机权重和内置输入；TTS 保留真实模型与音频输出。当前完整题库迭代适配为 Qwen，MoE/TTS 已有单候选入口，尚未自动获得完整
benchmarks session适配。新增 Python 文件：作用与调用场景
这里只列出本 PR 新增的 10 个框架与公共接入 Python 文件，说明其作用和调用场景。文件链接固定到当前提交。
这些文件不是在每次优化时全部执行。只有 Agent/调用方发起对应 CLI 操作或 Python API，才会进入对应代码；安装插件、打开仓库或仅发送一句自然语言请求不会直接触发全部脚本。
cli.py、arguments.py、dashboard.py等属于修改的已有文件，不计入本表；运行时生成的candidate.py也不计入源码新增文件。表中的
campaign …指新增的受管 start/knowledge/round/status/final/recover 流程。原有run仍是单次实验；当前只有 Qwen 具备完整题库 session 协议，MoE/TTS 的入口按各自单次评测协议使用。框架与公共接入：10 个
验证结果
本轮修改删除题库专用 handoff，使用通用 Agent job 先派发、子会话从 start 执行到 final。OS 文件锁下沉 Infra，使宿主会话记录与优化控制器解耦。修复 Qwen 将未知 NPU/运行时错误误判为候选错误的问题：仅明确的输出契约检查判 FAIL，其余执行异常为 INCONCLUSIVE。
下列是较早提交的软件验证，保留其原始验证范围：
验证对应提交
0b7883e89f180c58a5055b1fe0ee02d05f0608e7：8baed1a5定向复现 21 项已知失败，涉及 CRLF 假设、FIFO/symlink 和旧 skill-eval 预期。未把本地全量标为通过。这些合成指标验证软件流程及拒绝路径，不是新的 NPU 收益或 RSI 效果证据。
本轮真实宿主 / NPU 验证（2026-11-11）
当前提交
3cbb74f5f87ba4bbfe54a972a9e9e9b86e5b0ab3的 Python 3.10/3.11 full-check、installed-package、pr-references 和 declarations 均通过。独立技术复查未发现本次修正的剩余阻塞，不能代替正式 GitHub 审批。两组均通过宿主能力创建全新 GPT-6.1 Sol/high 会话，串行、无父对话与跨组方案输入。固定同一题包/runtime、8 轮 + 独立 final、pairs=2/samples=3/warmups=1；每组总预算 240 分钟，单次评测 1800 秒，其中自动单卡预约等待 10 分钟、运行上限 18 分钟。
infrastructure_failure / INCONCLUSIVE，campaign 为blocked。各留存 1/8 次尝试，成功候选评测均为 0，独立 final 均未执行。这验证了真实会话交接及环境失败路径，不是完成的 RSI 效果对照。没有候选加速比或有效优化曲线，不把初始 baseline 选优分数 1.0 当成测量结果。原始日志和实际可用 token 计数保留在私有实验目录，不提交仓库；完整 8 轮对照待资源可用后重新验收。
当前限制与后续验收
8baed1a5的真实宿主会话验证覆盖入口/CLI/产物交接；21b0a4f0的 Qwen NPU 控制包含一次预约超时的 blocked 会话及单独的有界候选/final 回放。这些不改称本次提交的新设备验证。详细文档
Refs #252
Refs #336
Refs #337