diff --git a/changelog.d/1493-workflow-failed-runs.fixed.md b/changelog.d/1493-workflow-failed-runs.fixed.md index 68f74e4f7..e6b3a758e 100644 --- a/changelog.d/1493-workflow-failed-runs.fixed.md +++ b/changelog.d/1493-workflow-failed-runs.fixed.md @@ -1 +1 @@ -Workflow stage now shows failed sibling runs in the header and timeline, not only in the Runs tab. ([#1493](https://github.com/ThinkFlowLab/FlyBot/issues/1493)) +Workflow stage now shows failed sibling runs in the header and timeline, not only in the Runs tab, and a Runs row shows when the run ended and why it failed. ([#1493](https://github.com/ThinkFlowLab/FlyBot/issues/1493)) diff --git a/docs/subsystems/workflow.md b/docs/subsystems/workflow.md index 684cdb002..442ed88a0 100644 --- a/docs/subsystems/workflow.md +++ b/docs/subsystems/workflow.md @@ -46,8 +46,8 @@ Everything in `plugins/workflow/`: the graph rules, the run store, the matcher, 1. Friends' Bots are not nodes. A `remote` Bot is refused by create, define and the matcher. 2. The canvas and the run share one graph: nodes, edges and positions. A drag commits `workflow/graph`, but only while **Edit** is on — a factory preset is copied first (#1378). Status highlighting reads that same graph. Replacing edges does not recompute who is already running. The Settings canvas paints that graph on the same 960px design plane, scaled to the width the settings page gives it, so the canvas is not a second scroll container inside the page's own scroll; its pointer deltas are divided by the scale, and its wires keep a stronger stroke than a plain divider so an edge can be followed at that size (#1483). -3. **A live run advances only when its ephemeral Bot replies, following the graph's edges in order.** That reply advances the run and may appear on the workflow stage log; it is not delivered into a persistent sidebar Bot conversation (for example Chief). The `workflow-step` send middleware denies those replies after `stepResult` runs. Nodes start only when their forward predecessors finish and edges reach them. **Early termination (#1537, #1538):** When a node completes **or fails** with `needs-human`, `already-claimed`, `tip-unchanged`, or `open-pr` detected in its outputs or reply text (see `checkEarlyEnd` in Key files), the run ends immediately with status `done`, sets `run.earlyEnd` to the condition name, and does not advance to successors, take a success-path `loop`, or burn `onFail` recovery attempts. Preferred signals are a declared OUTPUT equal to the marker, or a whole reply line `EARLY-END `; the engine also honors common phrases (`tip unchanged` / `unchanged tip`, `has open PR` / `already has open PR` / `found open PR`, plus the existing already-claimed heuristics) so a Bot that forgets the structured marker cannot soft-lock Claim↔Verify on the same tip. Factory prompt guidance lives in `EARLY_END_GUIDANCE` (`plugins/workflow/src/template.js`) for Settings graphs that wire Triage / Claim / Verify: Verify must escalate once on an unchanged tip, not re-run the full suite. There is no scripted or seeded run path: `workflow/state` exposes only `live === true` runs. A run does not create a schedule. `scheduleLabel` is display text. When the host can create Bots, starting a live run (`assign`, `run_workflow`, `ingest`) spawns one ephemeral Bot per node from that node's prompt, role, model and tools, sends it the step from the Main Bot, and removes every instance when the run finishes, fails for good, or is reset. A `failOnce` retry keeps them. They are hidden and left out of the sidebar. Loading the plugin does not spawn Bots for the default workflows. Plugin unload (`dispose`) and the load-time `reap` both end a live run still marked `running`: in-flight nodes are failed, the run is archived as `failed`, and any ephemeral Bot ids are released. A run already written as `running` with no ephemeral map (an older unload) is healed the same way on the next load (#1414). Loading is also the one moment a run the host left behind is noticed when its ephemeral map is still present: `reap` releases those instances and records the run as `failed`. Nothing resumes it. A run that already ended keeps its own status (#1388). `workflow/reset` stopping one live run ends it the same way before its Bots are released: nodes still `running` become `failed`, so the strip cannot keep painting a run that is over (#1424). Every wait is bounded (#1530). A dispatch that does not land is not marked handed: the chat names the code and the attempt, and it is retried after a short backoff at most twice, never twice inside one hand-off, so a 429 is not hammered. When it still does not land, the step and the run fail with the reason written into the chat. Each running live step also holds a deadline — its workflow's `stepTimeoutMinutes`, `DEFAULT_STEP_TIMEOUT_MINUTES` (45 minutes) unless the record sets one, an integer from 1 to 120 given at create or through `workflow/step-timeout` (armed even when the host cannot spawn ephemeral Bots; #1530, #1580); when it expires the node and the run fail as `failed`, the run's ephemeral Bots are released, the chat says the step timed out, and `warn` names it once. The deadline is cleared as soon as the step moves, and by Pause, Stop, Reset, delete, `reap` and `dispose`; a paused run holds none, and Resume arms it again from that moment. A timer never outlives the run that started it. A reply that matches no running step — its run ended, or the node already moved — is reported through `warn` once per Bot and run instead of being dropped silently. A reply that says `could not answer your message` or `model call failed` fails that node and does not start its successor. A 429 or rate-limit mention in any other reply does not. -4. Each workflow keeps a FIFO queue and up to `maxConcurrent` live runs (default 2, an integer from 1 to 8 on the record, via `workflow/concurrency`, or in Settings → Workflow graphs). A task that does not fit is enqueued. When a run finishes, every workflow with a free slot is drained, oldest enqueue first. A new assign on a workflow that already has a queue waits behind it. A task does not replace a running run or release that run's Bots. Migrating an old store keeps only runs that were still `running` with ephemeral Bot ids; other legacy scripted rows are dropped from active state (finished tasks may remain in lane history). History records a finished or failed live run. Lowering the cap does not stop a run that has already started. Assigning `{ botId }` claims the card for that local Bot and does not enqueue. A remote Bot is refused. `activeCount` is `runningOf(lane)` — live runs with `live === true` and `status === 'running'`. The stage aggregate chip and the Runs tab (运行记录) must agree: every such run appears as its own running row and progress card, and the chip count does not exceed that list. +3. **A live run advances only when its ephemeral Bot replies, following the graph's edges in order.** That reply advances the run and may appear on the workflow stage log; it is not delivered into a persistent sidebar Bot conversation (for example Chief). The `workflow-step` send middleware denies those replies after `stepResult` runs. Nodes start only when their forward predecessors finish and edges reach them. **Early termination (#1537, #1538):** When a node completes **or fails** with `needs-human`, `already-claimed`, `tip-unchanged`, or `open-pr` detected in its outputs or reply text (see `checkEarlyEnd` in Key files), the run ends immediately with status `done`, sets `run.earlyEnd` to the condition name, and does not advance to successors, take a success-path `loop`, or burn `onFail` recovery attempts. Preferred signals are a declared OUTPUT equal to the marker, or a whole reply line `EARLY-END `; the engine also honors common phrases (`tip unchanged` / `unchanged tip`, `has open PR` / `already has open PR` / `found open PR`, plus the existing already-claimed heuristics) so a Bot that forgets the structured marker cannot soft-lock Claim↔Verify on the same tip. Factory prompt guidance lives in `EARLY_END_GUIDANCE` (`plugins/workflow/src/template.js`) for Settings graphs that wire Triage / Claim / Verify: Verify must escalate once on an unchanged tip, not re-run the full suite. There is no scripted or seeded run path: `workflow/state` exposes only `live === true` runs — the running ones, plus the last run when it is still `live` and ended `failed` — a failed sibling stays on the stage beside whatever is running now, with `run.failure` (`{ code, at, nodeName?, text? }`) naming what ended it, so the room does not have to open Runs to learn that a run failed (#1493, #1494). The failed sibling is left out when a running run carries the same task id: that is a retry, and the live run keeps the row (#1507). History entries carry the run `id` and the same `failure`, so a Runs row shows when the run ended and why, and a row for a run already on the stage is not listed twice. A run does not create a schedule. `scheduleLabel` is display text. When the host can create Bots, starting a live run (`assign`, `run_workflow`, `ingest`) spawns one ephemeral Bot per node from that node's prompt, role, model and tools, sends it the step from the Main Bot, and removes every instance when the run finishes, fails for good, or is reset. A `failOnce` retry keeps them. They are hidden and left out of the sidebar. Loading the plugin does not spawn Bots for the default workflows. Plugin unload (`dispose`) and the load-time `reap` both end a live run still marked `running`: in-flight nodes are failed, the run is archived as `failed`, and any ephemeral Bot ids are released. A run already written as `running` with no ephemeral map (an older unload) is healed the same way on the next load (#1414). Loading is also the one moment a run the host left behind is noticed when its ephemeral map is still present: `reap` releases those instances and records the run as `failed`. Nothing resumes it. A run that already ended keeps its own status (#1388). `workflow/reset` stopping one live run ends it the same way before its Bots are released: nodes still `running` become `failed`, so the strip cannot keep painting a run that is over (#1424). Every wait is bounded (#1530). A dispatch that does not land is not marked handed: the chat names the code and the attempt, and it is retried after a short backoff at most twice, never twice inside one hand-off, so a 429 is not hammered. When it still does not land, the step and the run fail with the reason written into the chat. Each running live step also holds a deadline — its workflow's `stepTimeoutMinutes`, `DEFAULT_STEP_TIMEOUT_MINUTES` (45 minutes) unless the record sets one, an integer from 1 to 120 given at create or through `workflow/step-timeout` (armed even when the host cannot spawn ephemeral Bots; #1530, #1580); when it expires the node and the run fail as `failed`, the run's ephemeral Bots are released, the chat says the step timed out, and `warn` names it once. The deadline is cleared as soon as the step moves, and by Pause, Stop, Reset, delete, `reap` and `dispose`; a paused run holds none, and Resume arms it again from that moment. A timer never outlives the run that started it. A reply that matches no running step — its run ended, or the node already moved — is reported through `warn` once per Bot and run instead of being dropped silently. A reply that says `could not answer your message` or `model call failed` fails that node and does not start its successor. A 429 or rate-limit mention in any other reply does not. +4. Each workflow keeps a FIFO queue and up to `maxConcurrent` live runs (default 2, an integer from 1 to 8 on the record, via `workflow/concurrency`, or in Settings → Workflow graphs). A task that does not fit is enqueued. When a run finishes, every workflow with a free slot is drained, oldest enqueue first. A new assign on a workflow that already has a queue waits behind it. A task does not replace a running run or release that run's Bots. Migrating an old store keeps only runs that were still `running` with ephemeral Bot ids; other legacy scripted rows are dropped from active state (finished tasks may remain in lane history). History records a finished or failed live run. Lowering the cap does not stop a run that has already started. Assigning `{ botId }` claims the card for that local Bot and does not enqueue. A remote Bot is refused. `activeCount` is `runningOf(lane)` — live runs with `live === true` and `status === 'running'`. The stage aggregate chip and the Runs tab (运行记录) must agree: every such run appears as its own running row and progress card with its own `status` (never a hardcoded Running), the chip count does not exceed that list, and a failed sibling appears with its time and reason. 5. A node declares up to eight inputs and eight outputs. A required input is an output of the same name on an ancestor, or `task`, `title`, or `brief` on a root. A start with no provider is refused, and Bots spawned for that attempt are released. On a Bot-backed run a successor stays queued until its required inputs are present. If every forward predecessor is done, failed or skipped and an input is still missing, that node fails and the run stops. The dispatch text carries `Input : `. The Bot reports `OUTPUT : `; a single declared output may be the reply body. Nodes start in edge order. 6. Each node is a real ephemeral Bot for that run, then it is removed. A permanent sidebar Bot (`botId`) is optional and is not required for the step to run. **File and shell rights are the same as a normal sidebar Bot's:** every node Bot gets the `danger-full-access` preset, and `EPHEMERAL_DENIED_TOOLS` still refuses the team-admin tools at the guard and approval layers (#1466, 29 names). **`node.tools` selects connectors only:** `spawn` passes the node's cleaned list to `fb.bot.create`, the host records it as `stepTools`, and `connector-access.js` allows `mcp__github__*` for `github` and `mcp__composio__*` for `composio` — a connector the node did not name is refused, and a node that names none reaches none. The list is also stored in `fields.tools` for the UI. A hidden run Bot cannot show an approval card, so the approval seam answers the same way the guard does. The host stores this on the run Bot as `stepTools` and `permission`. 7. A loop edge (`loop: true`, `max` from 1 to 8, default 3, optional `until: { output, eq }`) is the only cycle for success paths. The body runs again until `until` matches or `max` runs have finished, and the forward edges wait until then. Restarting the body drops the branch choice made inside it and queues a sibling that the previous pass skipped, so the next pass can take the other side. A `when: { output, eq }` edge is taken only when that output equals `eq`. `when: { else: true }` is taken only when no other `when` matched. A node whose forward edges are all closed is `skipped` and does not block a join. A run cannot spin: every loop has a max. An `onFail` edge (`onFail: true`, `max` from 1 to 8, default 3) provides failure recovery: when a node fails, if an `onFail` edge points from that node back to an upstream target and recovery attempts remain (tracked in `failAttempts` per edge), the workflow resets both nodes to their pre-failure states (the target to `running`, the failing node to `queued`) and re-runs from the target through the failing node. After `max` attempts are exhausted, the node fails permanently and the run stops. An `onFail` edge does not interfere with `failOnce` retry (which restarts the same node once) or success-path `loop` edges. Settings → Workflow graphs can add a link, a condition, a loop, or a failure recovery edge. diff --git a/plugins/workflow/client.js b/plugins/workflow/client.js index ef4edb078..750fa8026 100644 --- a/plugins/workflow/client.js +++ b/plugins/workflow/client.js @@ -55,6 +55,11 @@ window.__ModuleLoader__.load({ 'Local Bots': '本机 Bot', 'No workflows yet.': '还没有 Workflow。', 'Waiting for a task': '等待任务', + 'A step failed': '有一步失败', + '{node} failed': '{node} 失败', + Stopped: '已停止', + 'The hand-off failed': '交接失败', + 'The host ended this run': '宿主中断了这次运行', 'Message the workflow': '发消息', Trigger: '触发方式', Manual: '手动，从任务看板分配', @@ -127,6 +132,8 @@ window.__ModuleLoader__.load({ '⏸ Paused · {task}': '⏸ 已暂停 · {task}', '▶ Resumed · {task}': '▶ 已继续 · {task}', '■ Stopped · {task}': '■ 已停止 · {task}', + '✕ Failed · {task}': '✕ 失败 · {task}', + '✕ Failed · {task} · {detail}': '✕ 失败 · {task} · {detail}', '{node} cannot start: missing input {missing}.': '{node} 不能开始：缺少输入 {missing}。', '{node} did not take this step ({code}).': '{node} 没有接到这一步（{code}）。', '{node} did not take this step ({code}); retrying in {seconds}s ({tries}/{total}).': '{node} 没有接到这一步（{code}），{seconds} 秒后重试（{tries}/{total}）。', @@ -219,6 +226,34 @@ window.__ModuleLoader__.load({ } const STATUS_KEY = { queued: 'Queued', running: 'Running', done: 'Done', failed: 'Failed', skipped: 'Skipped', idle: 'Idle' } + // Why a run failed, from the record the host keeps (#1493). A failure it did not + // name still reads Failed: the reason is added information, never a requirement. + const FAILURE_KEY = { + stopped: 'Stopped', + 'dispatch-failed': 'The hand-off failed', + 'host-ended': 'The host ended this run', + } + function failureText(run) { + const failure = run?.failure + if (!failure) return tr('Failed') + if (failure.code === 'step-failed') { + const head = failure.nodeName ? tr('{node} failed', { node: failure.nodeName }) : tr('A step failed') + return failure.text ? `${head} · ${failure.text}` : head + } + return tr(FAILURE_KEY[failure.code] ?? 'Failed') + } + // A row without a time (an older history entry) stays a row; it just shows none. + function runTime(at) { + const ms = Number(at) + if (!Number.isFinite(ms) || ms <= 0) return '' + const when = new Date(ms) + if (Number.isNaN(when.getTime())) return '' + const pad = (value) => String(value).padStart(2, '0') + const clock = `${pad(when.getHours())}:${pad(when.getMinutes())}` + const today = new Date() + const sameDay = when.getFullYear() === today.getFullYear() && when.getMonth() === today.getMonth() && when.getDate() === today.getDate() + return sameDay ? clock : `${pad(when.getMonth() + 1)}-${pad(when.getDate())} ${clock}` + } const COLUMN_KEY = { 'not-started': 'Not started', 'in-progress': 'In progress', done: 'Done' } const COLUMN_OF = { open: 'not-started', claimed: 'not-started', working: 'in-progress', blocked: 'in-progress', 'result-ready': 'in-progress', accepted: 'done', cancelled: 'done', failed: 'done' } const COLORS = { Planner: '#8b5cf6', Writer: '#f0509a', Coder: '#1fbf6a', Reviewer: '#f5a524', Tester: '#14b8a6', Deployer: '#2f7cf6', '研究员': '#a78bfa', '数据分析师': '#ec4899' } @@ -449,7 +484,9 @@ window.__ModuleLoader__.load({ const running = (run.running ?? []).map(id => nodeOf(workflow, id)).filter(Boolean) const chipClass = run.status === 'done' ? 'done' : (run.status === 'failed' ? 'failed' : (run.paused ? 'queued' : (running[0] ? 'running' : 'queued'))) const chipLabel = run.status === 'done' ? tr('Done') : (run.status === 'failed' ? tr('Failed') : (run.paused ? tr('Pause') : (running[0] ? tr('Running') : tr('Queued')))) - const meta = run.paused + const meta = run.status === 'failed' + ? failureText(run) + : run.paused ? `${tr('Pause')} · ${running.map(node => node.name).join(' ‖ ') || '—'} · ${run.doneCount}/${run.nodeCount}` : (running[0] ? `${tr('Running')} · ${running.map(node => node.name).join(' ‖ ')} · ${run.doneCount}/${run.nodeCount}` @@ -648,12 +685,20 @@ window.__ModuleLoader__.load({ const liveRuns = liveStageRuns(workflow) if (liveRuns.length === 0 && history.length === 0 && queued.length === 0) return null const rows = liveRuns.map(run => ({ + id: run.id, task: run.task, - status: 'running', + status: run.status, + at: run.updatedAt, + failure: run.failure, current: workflow.run?.id === run.id, })) for (const task of queued) rows.push({ task, status: 'queued', current: false }) - for (const item of history) rows.push({ ...item, current: false }) + const seen = new Set(rows.map(row => row.id ?? `${row.task?.id ?? ''}|${row.at ?? ''}`)) + for (const item of history) { + const key = item.id ?? `${item.task?.id ?? ''}|${item.at ?? ''}` + if (seen.has(key)) continue + rows.push({ ...item, current: false }) + } return rows } @@ -840,8 +885,13 @@ window.__ModuleLoader__.load({ const rows = runHistoryRows(workflow) if (!rows) return h('div', { className: 'wf-muted' }, tr('No runs yet.')) return h('div', null, rows.map((item, index) => h('div', { key: index, className: 'wf-hist' }, - h('span', { title: item.task?.title ?? '' }, taskSummary(item.task, item)), - h('span', { className: `wf-chip ${item.status === 'done' ? 'done' : item.status === 'running' ? 'running' : item.status === 'failed' ? 'failed' : 'queued'}` }, tr(item.status === 'done' ? 'Done' : item.status === 'running' ? 'Running' : item.status === 'failed' ? 'Failed' : 'Queued'))))) + h('span', { className: 'wf-hist-task', title: item.task?.title ?? '' }, taskSummary(item.task, item)), + h('span', { className: 'wf-hist-meta' }, + item.at ? h('span', { className: 'wf-hist-time' }, runTime(item.at)) : null, + item.status === 'failed' + ? h('span', { className: 'wf-hist-reason', title: failureText(item) }, failureText(item)) + : null), + h('span', { className: `wf-chip ${item.status === 'done' ? 'done' : item.status === 'running' ? 'running' : item.status === 'failed' ? 'failed' : 'queued'}` }, tr(STATUS_KEY[item.status] ?? 'Queued'))))) } function RenameModal({ workflow, onClose, onConfirm }) { @@ -1720,6 +1770,10 @@ window.__ModuleLoader__.load({ .wf-kv{display:flex;justify-content:space-between;gap:16px;padding:8px 0;font-size:12px} .wf-kv span:first-child{color:var(--bt-ink-2,rgba(20,20,20,.6))} .wf-hist{display:flex;justify-content:space-between;align-items:center;gap:8px;padding:10px 0;border-bottom:1px solid var(--bt-line,rgba(20,20,20,.08));font-size:12px} +.wf-hist-task{min-width:0;overflow:hidden;text-overflow:ellipsis;white-space:nowrap} +.wf-hist-meta{display:flex;align-items:center;gap:8px;min-width:0;margin-left:auto} +.wf-hist-time{color:var(--bt-ink-3,#9a9a9a);font-variant-numeric:tabular-nums;white-space:nowrap} +.wf-hist-reason{max-width:220px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;color:#e5484d} .wf-ctl{display:flex;gap:6px;flex-wrap:wrap;margin-top:14px} .wf-mask{position:fixed;inset:0;background:rgba(0,0,0,.28);display:flex;align-items:center;justify-content:center;z-index:80;padding:16px} .wf-modal{width:560px;max-width:100%;max-height:calc(100vh - 32px);background:var(--bt-main,#fcfcfc);color:var(--bt-ink,#141414);border-radius:16px;border:1px solid var(--bt-line,rgba(20,20,20,.08));display:flex;flex-direction:column;overflow:hidden} diff --git a/plugins/workflow/src/service.js b/plugins/workflow/src/service.js index 844f4d901..469b4154f 100644 --- a/plugins/workflow/src/service.js +++ b/plugins/workflow/src/service.js @@ -102,6 +102,7 @@ function publicRun(workflow, run) { updatedAt: run.updatedAt, ...(run.live === true ? { live: true } : {}), ...(run.paused === true ? { paused: true } : {}), + ...(run.failure ? { failure: { ...run.failure } } : {}), ...(run.earlyEnd ? { earlyEnd: run.earlyEnd } : {}), ...(run.ephemeral && Object.keys(run.ephemeral).length > 0 ? { ephemeral: { ...run.ephemeral } } : {}), } @@ -230,7 +231,7 @@ export function createWorkflowService({ params: { node: node?.name ?? nodeId, minutes }, text: `${node?.name ?? nodeId} 超时：${minutes} 分钟没有回包，run 已标记失败。`, }], now()) - endRun(item, run) + endRun(item, run, { code: 'step-failed', nodeName: node?.name, text: `no reply for ${minutes} minutes` }) await releaseRun(workflowId, run) await drainAll() return present() @@ -301,7 +302,7 @@ export function createWorkflowService({ queueDepth: (laneOf(item.id).queue ?? []).length, queue: (laneOf(item.id).queue ?? []).map(task => ({ id: task.id, title: task.title })), history: [...(laneOf(item.id).history ?? [])], - runs: runningOf(laneOf(item.id)).map(run => publicRun(item, run)), + runs: [...runningOf(laneOf(item.id)), ...failedSiblings(laneOf(item.id))].map(run => publicRun(item, run)), preview: rowPreview(item, previewRun(item.id)), run: publicRun(item, displayRun(item.id)), messages: state.logs[item.id] ?? [], @@ -322,6 +323,18 @@ export function createWorkflowService({ return (lane?.active ?? []).filter(run => run && run.live === true && run.status === 'running') } + // A failed run is still the room's business: the stage and the Runs tab show it + // beside whatever is running now, naming what ended it (#1493). A failure of the + // task that is live again is a retry, not a sibling, so the live run keeps the + // row and the older attempt stays in history (#1507). + function failedSiblings(lane) { + const failed = lane?.last + if (failed?.live !== true || failed.status !== 'failed' || !failed.task) return [] + const running = runningOf(lane) + if (running.some(run => run.task?.id && run.task.id === failed.task.id)) return [] + return [failed] + } + function maxOf(item) { const value = Number(item?.maxConcurrent) if (!Number.isInteger(value) || value < 1) return DEFAULT_CONCURRENT @@ -458,6 +471,19 @@ export function createWorkflowService({ ...(earlyEnded ? { earlyEnd: result.earlyEnd } : {}), } if (Object.keys(handed).length === 0) delete next.handed + if (next.status === 'failed') { + const hole = blocked ? gated.blocked[0] : null + const node = nodeById(item, hole ? hole.id : op.nodeId) + const text = hole + ? `Missing input ${hole.missing.join(', ')}` + : String(op.text ?? '').replace(/\s+/g, ' ').trim() + next.failure = { + code: 'step-failed', + at: next.updatedAt, + ...(node ? { nodeName: node.name } : {}), + ...(text ? { text: text.length > 200 ? `${text.slice(0, 197)}…` : text } : {}), + } + } const lines = linesFor(item, working.task, { type: op.type, nodeId: op.nodeId, @@ -551,7 +577,7 @@ export function createWorkflowService({ }], now()) // The run cannot finish without this step: end it the way every other // abandoned run ends (#1424), then let the caller release the Bots. - endRun(item, current) + endRun(item, current, { code: 'dispatch-failed' }) storeRun(item.id, current) break } @@ -575,7 +601,7 @@ export function createWorkflowService({ lane.active = lane.active.filter(item => item.id !== run.id) lane.last = run if (run.status === 'done' || run.status === 'failed') { - lane.history = [...lane.history, { task: run.task, status: run.status, at: run.updatedAt }].slice(-20) + lane.history = [...lane.history, historyRow(run)].slice(-20) } await releaseAll(ids, release, warn) write() @@ -586,8 +612,9 @@ export function createWorkflowService({ // dispose(), the catch around a step that threw, and reset() stopping one run — // so run.status and run.nodeStatus cannot drift apart and a client stops painting // "Running now" (#1424). A run that already ended keeps its own ending (#1388); - // ephemeral ids stay on the record until release strips them. - function endRun(item, run) { + // ephemeral ids stay on the record until release strips them. `failure` names the + // ending so the stage and the Runs tab can say why, not only Failed (#1493). + function endRun(item, run, failure = { code: 'host-ended' }) { if (run.status === 'done' || run.status === 'failed') return run for (const nodeId of item ? orderedIds(item, run.nodeStatus, 'running') : []) { try { @@ -598,14 +625,49 @@ export function createWorkflowService({ } run.status = 'failed' run.updatedAt = now() + run.failure = { ...failure, at: run.updatedAt } return run } + // A run that ended failed says so once in the stage log: the step-failure paths + // already write their own line, these are the endings that wrote none (#1493). + function failureLine(run) { + const label = summarizeTask(run.task) + if (run.failure?.text) { + return { + kind: 'system', + tone: 'bad', + key: '✕ Failed · {task} · {detail}', + params: { task: label, detail: run.failure.text }, + text: `✕ 失败 · ${label} · ${run.failure.text}`, + } + } + return { + kind: 'system', + tone: 'bad', + key: '✕ Failed · {task}', + params: { task: label }, + text: `✕ 失败 · ${label}`, + } + } + function archiveCutoffRun(workflowId, run) { const lane = laneOf(workflowId) lane.active = lane.active.filter(item => item.id !== run.id) lane.last = run - lane.history = [...lane.history, { task: run.task, status: run.status, at: run.updatedAt }].slice(-20) + lane.history = [...lane.history, historyRow(run)].slice(-20) + } + + // The Runs tab reads this: a finished run, when it ended, and — for a failure — + // the reason, so the row says more than "Failed" (#1493). + function historyRow(run) { + return { + id: run.id, + task: run.task, + status: run.status, + at: run.updatedAt, + ...(run.failure ? { failure: { ...run.failure } } : {}), + } } // The oldest queued task whose own workflow still has a slot, or null. @@ -738,7 +800,7 @@ export function createWorkflowService({ try { run = await handOff(item, run) } catch (error) { - endRun(item, run) + endRun(item, run, { code: 'dispatch-failed', text: String(error?.message ?? error).slice(0, 200) }) await releaseRun(item.id, run) return refuse(error) } @@ -1029,7 +1091,7 @@ export function createWorkflowService({ clearRunTimers(run.id) lane.active = lane.active.filter(item => item.id !== run.id) lane.last = run - lane.history = [...lane.history, { task: run.task, status: run.status, at: run.updatedAt }].slice(-20) + lane.history = [...lane.history, historyRow(run)].slice(-20) drainSync() write() } @@ -1142,12 +1204,12 @@ export function createWorkflowService({ // Stopping one run ends it the same way a reap does: the node map settles // with the run, or the strip keeps painting the node it was on (#1424). clearRunTimers(run.id) - const ended = { ...endRun(item, run) } + const ended = { ...endRun(item, run, { code: 'stopped' }) } delete ended.ephemeral delete ended.handed lane.active = lane.active.filter(itemRun => itemRun.id !== runId) lane.last = ended - lane.history = [...lane.history, { task: ended.task, status: ended.status, at: ended.updatedAt }].slice(-20) + lane.history = [...lane.history, historyRow(ended)].slice(-20) push(workflowId, [{ kind: 'system', tone: 'bad', @@ -1335,7 +1397,8 @@ export function createWorkflowService({ try { clearRunTimers(run.id) if (run.ephemeral) stale.push(...Object.values(run.ephemeral)) - endRun(workflow(workflowId), run) + endRun(workflow(workflowId), run, { code: 'host-ended' }) + push(workflowId, [failureLine(run)], now()) delete run.ephemeral delete run.handed archiveCutoffRun(workflowId, run) @@ -1359,7 +1422,7 @@ export function createWorkflowService({ for (const [workflowId, lane] of Object.entries(state.lanes ?? {})) { for (const run of [...(lane.active ?? [])]) { if (run?.live === true && run.status === 'running') { - endRun(workflow(workflowId), run) + endRun(workflow(workflowId), run, { code: 'host-ended' }) archiveCutoffRun(workflowId, run) endedLive = true } diff --git a/plugins/workflow/test/client.test.js b/plugins/workflow/test/client.test.js index 2a3b21406..0746bcd4f 100644 --- a/plugins/workflow/test/client.test.js +++ b/plugins/workflow/test/client.test.js @@ -1619,3 +1619,76 @@ test('a failed store read is reported and retried, not painted as no workflows', } }) }) + +// #1493: the header lists the runs that are still the room's business, so a +// sibling that failed is on the stage beside the one that is running — and the +// Runs tab says when it ended and why. The sibling is not listed twice: the +// failed run is both a live-stage row and a history row. +test('#1493: a failed sibling run stays on the stage, and its row carries time and reason', async () => { + const graph = delivery() + const running = graph.run + running.id = 'run-1445' + running.task = { id: '#1445', title: 'issue #1445: a Workflow run keeps the team task' } + running.taskSummary = '#1445 · issue #1445: a Workflow run keeps the team task' + running.updatedAt = 1_700_000_000_000 + const failed = { + id: 'run-1460', + live: true, + status: 'failed', + task: { id: '#1460', title: 'issue #1460: every stage chat line is capped' }, + taskSummary: '#1460 · issue #1460: every stage chat line is capped', + nodeStatus: { planner: 'done', writer: 'failed' }, + running: [], + doneCount: 1, + nodeCount: 6, + marks: ['done', 'failed', 'queued', 'queued', 'queued'], + steps: 2, + stepBudget: 10, + startedAt: 1_699_999_996_000, + updatedAt: 1_700_000_000_000, + failure: { code: 'step-failed', nodeName: 'Writer', text: 'the reply came back empty' }, + } + graph.runs = [running, failed] + graph.history = [ + { id: 'run-1460', task: failed.task, status: 'failed', at: failed.updatedAt, failure: failed.failure }, + { task: { id: '#1395', title: 'an older finished run' }, status: 'done', at: 1_699_999_000_000 }, + ] + const call = async (name) => { + if (name === 'workflow/state') return { workflows: [graph], templates: [{ id: 'delivery', name: '发布交付流水线' }] } + return { ok: true, workflows: [graph] } + } + const view = mount(client.Section, { roster: { bots: [] }, call, actions: {} }) + await settle() + findAll(view.tree, node => node.type === 'button' && textOf(node).includes('发布交付流水线'))[0].props.onClick() + await settle() + + const strips = findAll(view.tree, node => typeof node.props?.['data-run-id'] === 'string') + assert.deepEqual(strips.map(node => node.props['data-run-id']), ['run-1445', 'run-1460']) + const failedStrip = strips[1] + assert.equal(textOf(failedStrip).includes('Failed'), true, 'the sibling strip does not say Failed') + assert.equal(textOf(failedStrip).includes('Writer failed · the reply came back empty'), true, 'the sibling strip does not say why it failed') + assert.equal(textOf(failedStrip).includes('Waiting for a task'), false) + + // Focusing the failed sibling is how the drawer shows its flow, so its row must be clickable. + failedStrip.props.onClick() + await settle() + assert.equal(findAll(view.tree, node => node.props?.['data-run-id'] === 'run-1460' && node.props.className.includes('wf-strip-focus')).length, 1) + + // The strip's own Progress button is the door to its Runs row. + const progress = findAll(failedStrip, node => node.type === 'button' && textOf(node) === 'Progress')[0] + assert.ok(progress, 'the failed strip has no way into its Runs row') + progress.props.onClick() + await settle() + + const rows = findAll(view.tree, node => node.props?.className === 'wf-hist') + assert.equal(rows.length, 3, `the Runs tab listed ${rows.length} rows for two live rows and two history rows, one of which is the failed run itself`) + const row = rows.find(node => textOf(node).includes('#1460')) + assert.ok(row, 'the failed run has no Runs row') + assert.equal(textOf(row).includes('Failed'), true) + assert.match(textOf(row), /\d{2}-\d{2} \d{2}:\d{2}/, 'the Runs row has no time') + assert.equal(textOf(row).includes('Writer failed · the reply came back empty'), true, 'the Runs row has no reason') + const older = rows.find(node => textOf(node).includes('#1395')) + assert.ok(older, 'a history row was dropped') + assert.equal(textOf(older).includes('Done'), true) + view.unmount() +}) \ No newline at end of file diff --git a/plugins/workflow/test/failed-sibling.test.js b/plugins/workflow/test/failed-sibling.test.js new file mode 100644 index 000000000..8fcd0e5d1 --- /dev/null +++ b/plugins/workflow/test/failed-sibling.test.js @@ -0,0 +1,115 @@ +// #1493: a failed run is not a Runs-tab secret. While another run is live, the +// stage keeps the failed sibling, the timeline says the run ended, and the Runs +// row carries the time and the reason. A failure of the task that is live again +// is a retry, not a sibling, so it stays in history and leaves one row (#1507). +// The silent ending the screenshot shows is the reap: a host that came back with +// the run still marked running archived it as failed and said nothing (#1424). +import assert from 'node:assert/strict' +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { test } from 'node:test' +import { createWorkflowService } from '../src/service.js' +import { DELIVERY_ID } from '../src/template.js' + +const CLOCK = 1_700_000_000_000 + +const roster = [ + { id: 'main', name: 'Chief', role: 'lead', isMain: true, remote: false }, + { id: 'bot-coder', name: 'Coder', role: '实现', remote: false }, +] + +function harness({ dir = null, clock = CLOCK } = {}) { + let seq = 0 + let now = clock + const svc = createWorkflowService({ + ...(dir ? { dir, persist: true } : { persist: false }), + now: () => now, + bots: () => roster, + spawn: async () => { + seq += 1 + return `eph-${seq}` + }, + release: async () => {}, + dispatch: async () => ({ ok: true }), + }) + return { svc, tick(ms = 1000) { now += ms; return now } } +} + +function delivery(svc) { + return svc.state().workflows.find(item => item.id === DELIVERY_ID) +} + +const label = (run) => `${run.task.id} ${run.status}` + +test('#1493: a failed sibling stays on the stage with the reason it ended', async () => { + const { svc, tick } = harness() + svc.setConcurrency({ workflowId: DELIVERY_ID, maxConcurrent: 2 }) + await svc.assign({ workflowId: DELIVERY_ID, taskId: '#1460', title: 'issue #1460: every stage chat line is capped' }) + tick() + await svc.assign({ workflowId: DELIVERY_ID, taskId: '#1445', title: 'issue #1445: a Workflow run keeps the team task' }) + tick() + const before = delivery(svc) + assert.deepEqual(before.runs.map(label), ['#1460 running', '#1445 running']) + const victim = before.runs.find(run => run.task.id === '#1460') + const stopped = await svc.reset({ workflowId: DELIVERY_ID, runId: victim.id }) + assert.equal(stopped.ok, true) + tick() + + const after = delivery(svc) + assert.equal(after.activeCount, 1, 'activeCount still counts only running runs') + assert.equal(after.run.task.id, '#1445', 'the running run stays the focused one') + assert.deepEqual(after.runs.map(label), ['#1445 running', '#1460 failed'], 'the failed sibling is missing from the stage') + const failed = after.runs.find(run => run.status === 'failed') + assert.equal(failed.id, victim.id) + assert.equal(failed.failure?.code, 'stopped', 'the failed run does not say what ended it') + assert.equal(failed.updatedAt, failed.failure.at) + + const row = after.history.find(item => item.task.id === '#1460') + assert.equal(row.status, 'failed') + assert.equal(row.id, victim.id, 'a history row cannot be matched to the run it came from') + assert.equal(row.at, failed.updatedAt) + assert.equal(row.failure?.code, 'stopped', 'the history row dropped the reason') +}) + +test('#1493: a retry of the same task leaves the live run as its only row', async () => { + const { svc, tick } = harness() + svc.setConcurrency({ workflowId: DELIVERY_ID, maxConcurrent: 2 }) + await svc.assign({ workflowId: DELIVERY_ID, taskId: '#1460', title: 'issue #1460: retried' }) + tick() + const first = delivery(svc).runs[0] + await svc.reset({ workflowId: DELIVERY_ID, runId: first.id }) + tick() + await svc.assign({ workflowId: DELIVERY_ID, taskId: '#1460', title: 'issue #1460: retried' }) + tick() + + const after = delivery(svc) + assert.deepEqual(after.runs.map(label), ['#1460 running'], 'the failed attempt of a task that is live again must not add a row (#1507)') + const row = after.history.find(item => item.task.id === '#1460') + assert.equal(row?.status, 'failed', 'the failed attempt left no history row') +}) + +test('#1493: a run the host lost on reload is archived failed with a reason and a line', async () => { + const dir = mkdtempSync(join(tmpdir(), 'wf-failed-sibling-')) + try { + const first = harness({ dir }) + await first.svc.assign({ workflowId: DELIVERY_ID, taskId: '#1460', title: 'issue #1460: the host went away' }) + first.tick(5000) + // A host that came back without disposing: the store still says the run is running. + const after = harness({ dir, clock: CLOCK + 10_000 }) + await after.svc.reap() + const lane = delivery(after.svc) + assert.deepEqual(lane.runs.map(label), ['#1460 failed']) + const failed = lane.runs[0] + assert.equal(failed.failure?.code, 'host-ended') + assert.equal(failed.failure.at, failed.updatedAt) + const line = [...lane.messages].reverse().find(message => message.tone === 'bad') + assert.ok(line, 'the timeline never says the run was lost') + assert.match(line.text, /失败 · #1460/) + const row = lane.history.find(item => item.status === 'failed') + assert.equal(row.id, failed.id) + assert.equal(row.failure?.code, 'host-ended') + } finally { + rmSync(dir, { recursive: true, force: true }) + } +})