Repository navigation
Description
Found while working #195.
Problem
The devbox verify run for eve#201 (head f9a9efe, 2026-10-06 05:15 UTC) failed chat-tool-search: the first tool step is "call_tool", not tool_search (steps: call_tool, tool_search, call_tool). Haiku first called call_tool with a guessed name (relay-tides), got unknown tool "relay-tides", then ran tool_search, loaded tides_lookup and answered correctly with the tide code. The same run passed 32 other journeys. eve#201 changes no chat, tool or relay-tools code.
Root cause
The journey requires the model's first tool step to be tool_search. Haiku sometimes guesses a tool name from the skill list first. The answer is still right, and the order is the model's choice, not eve's or relay's.
Fix
To be decided: either the journey accepts a failed call_tool on an unknown name before tool_search (that still proves the search path ran and produced the answer), or the tool-search prompt/skill text steers the model to search first. Changing the journey's assertion is a harness-decision change, so it needs the owner (gate 2).
Decisions made for the owner
- The journey treats a step as a guessed name only when its result parses as JSON whose
errorstarts withunknown tool "(relay's exact refusal). Anot loadedrefusal or any other step beforetool_searchstill fails, and so does free text that merely mentions an unknown tool. - Any number of leading guesses is accepted, not just one. The owner's approval names no limit, and
tool_searchmust still follow. - The detail checks (call_tool names
tides_lookup, direct step shows the nonce) read only steps aftertool_search, so a guess can't satisfy them. - Red on main uses a stand-in commit, 3b70e03. It extracts main's order check into
searchStepunchanged, because the order rule had no unit-reachable door onmain.
Contract
searchStep(steps) -> { index } | { error }anddetailProblem(steps, searchIndex, nonce) -> string | null, both exported fromdevboxverify/journeys-tool-search.js.stepsis[{ name, input, output }]astoolSteps()reads the thread.- Ownership:
devboxverify/journeys-tool-search.js,devboxverify/README.md,test/unit/devboxverify-tool-search.test.js. - Waits: none new. The unit tests are synchronous, and the journey's polls are unchanged.
Activity
barelyworkingcode commentedon Oct 6, 2026
Sighting 2026-10-06T07:56Z: eve PR #203 head 70e1492 (devbox/verify run 3; app code identical to 3e9f98f, which passed this journey twice). Haiku's first tool step was
call_tool(steps: call_tool, tool_search, call_tool).barelyworkingcode commentedon Oct 6, 2026
Seen 2026-10-06 in a devbox/verify run for PR #222 (an e2e-spec-only diff; the same app code passed 36/36 on PRs #218, #219 and #221 the same day). This is a different mode from the one this issue describes: Haiku called tool_search first, then
tides_lookupfor portVerify10d8078f, and the tool returnedTIDE-52d92889, but the journey expectedTIDE-a3e9d2de. The world stub and the harness disagree on the code. The same run's repair step found the world dirty first (acme Acme Corp: unexpected entries: 2) and reset it. Verify comment: #222 (comment). Re-running once; a rerun-to-green links here.barelyworkingcode commentedon Oct 6, 2026
Seen again 2026-10-06 in the PR #222 verify rerun (e2e-spec-only diff), this issue's own mode: steps
call_tool, call_tool, tool_search, call_tool. Verify comment: #222 (comment). A further rerun links here.barelyworkingcode commentedon Oct 8, 2026
Seen again on eve PR #263, head a76b313. The
--postrun went 38/1: chat-tool-search's first tool step wascall_toolwith the guessed namerelay-tides, thentool_search, then the right call. Main's harness against the same head, minutes later, went 39/0. The PR changes no app code (docs, the map, and anareas.jsnothing calls yet).barelyworkingcode commentedon Oct 8, 2026
Owner approved 2026-10-07 in chat (gate 2): the chat-tool-search journey may accept a failed call_tool on an unknown name before tool_search, provided tool_search still runs, loads the tool and the answer is correct. No other assertion is relaxed.
devboxverify: chat-tool-search detail checks read only steps after to…
devboxverify: detailProblem slices after tool_search itself; README w…
Merge pull request #267 from barelyworkingcode/fix/202-tool-search-order