diff --git a/LESSONS.md b/LESSONS.md index e6749e07b..aaf545011 100644 --- a/LESSONS.md +++ b/LESSONS.md @@ -835,3 +835,9 @@ Seen 1× (group dispatch and Bot tracking reproduced the bypass). Capture existi ### 2026-11-03 — Validate the composed external proposal, not just its private receipt Seen 1× (production-entry FBA regression). Legacy external proposals preserved title/criteria but dropped canonical impact/scope, leaving the approved action unable to carry its original requirements to Chief. Preserve explicit fields in both the private intent hash and published original Task; test actual authenticated HTTP approval/watch/revoke. Successful startup also requires actual Store writability, not only a Harness login URL. + +### 2026-11-04 — Runner invocation is not runner readiness (Seen 1×) + +`npm test` can start and exit 127 without loading Vitest. Preserve command-start +facts, classify runner startup separately, and strip inherited NODE_TEST_CONTEXT +so independent Node assertions cannot silently skip under the test harness. diff --git a/PROJECT.md b/PROJECT.md index 1984b81c1..62bc962aa 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -327,3 +327,7 @@ Use scripts/backend.mjs to initialize/start the original DSH/FlyBot host plus ex ### Remaining backend convergence — #2292 Task execution pins its associated repository independently of Project defaults; local repository-free decision Tasks preserve normal Sessions/handoff/Chief acceptance without inventing code/tests. Actual Codex validation exposed malformed Chief requirement output: types are now explicit, failed admission is visibly blocked and valid retry retains history. Release Tasks now separate explicit human/watch publication authority, durable external observation and Chief closure; unknown results reconcile without repeated publication. Attempt logs and legacy agent history now return bounded hash-bound pages with lossless source references. Initial execution pins an approved active logical role/capability, catalog revision and Task requirements revision. Default developer remains compatible; a unique custom developer can replace it, while ambiguity needs an explicit target. Chief start Effects, capacity queues and Attempts retain the choice. Catalog/requirements changes block pending work for explicit recovery; active Session conventions remain frozen. + +#2334 follow-up (updated issue): shell 127 / missing test runners are environment +blockers. testsStarted records command invocation; stage distinguishes runner startup +from preparation and tests. Nested Node test markers do not enter host verification. diff --git a/ROADMAP.md b/ROADMAP.md index 0138d3f7f..52c9eed97 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -116,14 +116,7 @@ Broader Claude/mixed-resource/browser-auth/restart checks remain separate future ## Owner-authorized collaboration repairs — 2026-11-01 -[Tracking #33](https://github.com/zy95-12/FlyBot/issues/33): #34 repository preflight → #35 role contracts → #36 per-Bot pools/handoff → #37 ordinary Chief queries → #38 safe/watch delegation → #39 Chief arbitration/configurable schedules → #40 native UI/multi-Bot reuse → #41 actual shared-service acceptance. Component/UI inventory may proceed alongside its foundations; each implementation has its own PR and tested evidence. Public hosting is deferred. A 30-minute Chief inspection is a configurable scheduled-task default, never framework logic. R1 PR #42 passed full local verification; configured-service acceptance is pending. R2 implements complete role-policy delivery and evidence separation and is under verification. R3 dynamic pool/claim/handoff implementation is under verification, including non-DAG SDK contract-process rework. R4 conversation/query/status repair is under verification; R5 persistent safe/watch delegation is under verification; R6–R8 remain pending; R4 integrates conversation bridging and R6 completes Chief arbitration/configurable schedules. The owner requests backend repairs first, then real human-style assignment through FlyBot to exercise role handoffs and Chief fallback. - -R6 implementation checkpoint: Chief now adjudicates completion/uncertainty and may -intervene through audited routes. Inspection reuses original configurable DSH -schedules; there is no Chief cadence in framework timers. After exact-tip full -verification and its own draft PR, upgrade the backed-up shared backend, assign -#40 through actual FlyBot and collect real Codex/handoff/Chief/restart evidence in -#41. Protocol fixtures alone do not satisfy those integration criteria. +Original R1–R8 implementation checkpoints are preserved in the dated delivery history below. Current collaboration and frontend acceptance boundaries remain here. Earlier R6/R7 implementation, recovery and frontend checkpoint narratives are [archived verbatim](docs/history/2026-11-03-collaboration-checkpoints.md). @@ -142,39 +135,11 @@ complete from these historical checkpoints or backend convergence alone. ### Resident Chief and Task-role context — 2026-11-01 Implemented Task-role logical identity, binding-checked SDK resume across rework, retained cwd ownership/cleanup and current candidate inputs. Added original-store Chief topic-memory read/write/replace and same-thread scoped retrieval rounds, plus immutable context archives instead of re-sending unbounded history. Native DSH keeps its original parts/steward/compaction; SDK uses its own runtime compaction. Verification must include real resource-user/runtime continuation and Chief memory before claiming live delivery. Current native workflow carriers remain per execution; their logical Task identity and retrievable facts are retained, not falsely described as SDK backend threads. -Task40 continues with synchronized native CSS/JS loading and a scoped composer -exported from the original composer module, sharing dictation cleanup and captured -chat sends. Focused regression evidence is separate from complete UI acceptance; -rich tool/approval, personal running state and deployed browser acceptance remain open. - -Task40 native interaction continuation implements personal questions/one-shot approval, -stop/continue, own-chat streaming, bounded tool output and original group member-work -controls. SDK execution input reuses original approval/composer components. Proceed to -exact-candidate host tests and independent Review; only operator evidence can satisfy -actual deployed three-account/weak-network and visual acceptance. - -Task40 host-return repair preserves legacy listener registration while retaining the -explicit shared native stream observer. Reproduced all four reported failures and -added a fail-without-fix mode-boundary regression. Continue through host tests/Review -and operator acceptance on the same Task/PR48. - -Task40 review-return continuation repairs live-part navigation and replaces plain human -paragraphs with original person bubbles. Native file/image sends and scoped historical -attachment reads are implemented through public DSH APIs with explicit shared-entry -byte limits. Proceed through host verification/independent Review; operator deployed -three-account, weak-network and screenshot evidence remains separately required. - -Task40 refresh repair retains scoped native roots and updates props instead of -unmounting on each poll. Bounded mounted-workbench regressions cover focus, selection, -drafts, sends and teardown for personal/public/private/room views. Continue through -new candidate verification and Review without borrowing earlier candidate evidence. - -Task40/PR48 integrates preserved frontend 0e72dc60 with operator backend b920cae8, -including S1 and verified workspace/Session exit proofs. Compact board status -uses the same live-activity evidence as the full S1 view. Integrated focused tests -pass; exact sealed candidate host verification and independent Review come next. -Actual shared-service browser/weak-network/screenshot acceptance stays outstanding. - +Earlier Task40 frontend implementation checkpoints are +[preserved verbatim](docs/history/2026-11-04-collaboration-delivery-checkpoints.md). +They do not complete actual shared-service, three-account, weak-network, screenshot +or original native UI compatibility acceptance. Preserve Task40/PR48 and healthy +execution; current changes remain in the foreground backend session. Issue60 follow-up restores the completed-Chief deferral checkpoint discovered by real S4 frontend recovery. Keep all exit/lease guards and the current unsealed @@ -203,3 +168,7 @@ current authority checks before execution and stop on revoked rights; unknown exit retains capacity. Real-provider parallel/restart/migration/UI and Release operational acceptance remain distinct pending evidence under #2292. Previously deferred multi-repository automation is not reactivated by this recovery work. + +Updated #2334 acceptance adds actual npm missing-vitest/127 classification and +clean/prepare/same-candidate retest coverage; it remains a source/test follow-up, +not live npm ci provisioning or production Vitest acceptance. diff --git a/changelog.d/2334-runner.fixed.md b/changelog.d/2334-runner.fixed.md new file mode 100644 index 000000000..7db582a54 --- /dev/null +++ b/changelog.d/2334-runner.fixed.md @@ -0,0 +1 @@ +Classify missing test runners and shell exit127 as environment blockers, expose verification stage, and prevent inherited Node test markers from skipping independent assertions. diff --git a/core/src/host/project-chief-control.js b/core/src/host/project-chief-control.js index 8c72e1a98..8294b3784 100644 --- a/core/src/host/project-chief-control.js +++ b/core/src/host/project-chief-control.js @@ -142,7 +142,7 @@ export function createProjectChiefControl(rt) { repositoryRecovery: await rt.projectDelivery.recoveryFacts?.(task) ?? null, integrationInputs: structuredClone(attempt?.integrationInputs ?? []), attemptStatus: attempt?.status ?? null, generation: task.collaboration?.generation ?? 0, destination: task.collaboration?.botId ?? null, attemptId: attempt?.id ?? null, candidateSha: attempt?.candidateSha ?? null, evidence: { implementation: attempt?.evidence.implementation ? { sha: attempt.evidence.implementation.sha, paths: attempt.evidence.implementation.paths } : null, - tests: attempt?.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, status: attempt.evidence.tests.status ?? null, reason: attempt.evidence.tests.reason ?? null, testsStarted: attempt.evidence.tests.testsStarted ?? null, preparation: attempt.evidence.tests.preparation ? { id: attempt.evidence.tests.preparation.id, status: attempt.evidence.tests.preparation.status, executor: attempt.evidence.tests.preparation.executor, sha256: attempt.evidence.tests.preparation.sha256 } : null, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, + tests: attempt?.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, status: attempt.evidence.tests.status ?? null, stage: attempt.evidence.tests.stage ?? null, reason: attempt.evidence.tests.reason ?? null, testsStarted: attempt.evidence.tests.testsStarted ?? null, preparation: attempt.evidence.tests.preparation ? { id: attempt.evidence.tests.preparation.id, status: attempt.evidence.tests.preparation.status, executor: attempt.evidence.tests.preparation.executor, sha256: attempt.evidence.tests.preparation.sha256 } : null, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, review: attempt?.evidence.review ? { sha: attempt.evidence.review.sha, verdict: attempt.evidence.review.verdict, summary: attempt.evidence.review.summary, repositoryVerdict: attempt.evidence.review.report?.repositoryVerdict ?? 'INC' } : null, outputs: attempt?.evidence.developer?.outputs ?? null, pr: attempt?.pr ? { number: attempt.pr.number, sha: attempt.pr.sha, url: attempt.pr.url } : null }, inspection: facts } })) diff --git a/core/src/host/project-delivery.js b/core/src/host/project-delivery.js index ab497b3ec..52a8ceb8e 100644 --- a/core/src/host/project-delivery.js +++ b/core/src/host/project-delivery.js @@ -36,7 +36,7 @@ const requirementsOf = (task, repository) => ({ title: task.title, criteria: tas ...(task.projectTask ? { projectTaskRevision: task.projectTask.requirements.revision, impact: task.projectTask.requirements.impact, scope: task.projectTask.requirements.scope } : {}) }) const reject = message => Object.assign(new Error(message), { code: 'bot/delivery' }) const inside = (root, path) => { const rel = relative(root, path); return rel !== '' && rel !== '..' && !rel.startsWith(`..${sep}`) && !isAbsolute(rel) } -const cleanEnv = () => Object.fromEntries(Object.entries(process.env).filter(([key]) => !/TOKEN|PASSWORD|SECRET|API_KEY|GIT_CONFIG|NODE_OPTIONS/.test(key))) +const cleanEnv = () => Object.fromEntries(Object.entries(process.env).filter(([key]) => !/TOKEN|PASSWORD|SECRET|API_KEY|GIT_CONFIG|NODE_OPTIONS|NODE_TEST_CONTEXT/.test(key))) async function git(path, args, preserveWhitespace = false) { const { stdout } = await exec('git', ['-c', 'core.hooksPath=/dev/null', '-c', 'commit.gpgsign=false', ...args], { cwd: path, timeout: 30_000, maxBuffer: 2 * 1024 * 1024, env: { ...cleanEnv(), GIT_TERMINAL_PROMPT: '0', @@ -126,7 +126,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { if (source !== attempt && tests.sha !== attempt.seedSha) return null const text = await readFile(tests.log, 'utf8') if (hash(text) !== tests.sha256) throw reject('Failure diagnostic log changed') - return { status: tests.status ?? 'test-process-failed', reason: tests.reason ?? null, executor: tests.executor, testsStarted: tests.testsStarted ?? true, preparation: tests.preparation ? { id: tests.preparation.id, status: tests.preparation.status, sha256: tests.preparation.sha256, executor: tests.preparation.executor } : null, sourceAttemptId: source.id, sha: tests.sha, logSha256: tests.sha256, excerpt: testFailureExcerpt(text, process.env), completeLogRetained: true, source: 'project-attempt-log; bounded host-verified diagnostic, not a passing result' } + return { status: tests.status ?? 'test-process-failed', stage: tests.stage ?? null, reason: tests.reason ?? null, executor: tests.executor, testsStarted: tests.testsStarted ?? true, preparation: tests.preparation ? { id: tests.preparation.id, status: tests.preparation.status, sha256: tests.preparation.sha256, executor: tests.preparation.executor } : null, sourceAttemptId: source.id, sha: tests.sha, logSha256: tests.sha256, excerpt: testFailureExcerpt(text, process.env), completeLogRetained: true, source: 'project-attempt-log; bounded host-verified diagnostic, not a passing result' } } async function archiveForNative(context, attempt, role, suffix) { if (!context.archive) return @@ -620,7 +620,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { if (error && !output) output = 'Verification did not complete; inspect the structured outcome and preparation receipt' await writeFile(log, output, { mode: 0o600 }) if (attempt.evidence.tests) (attempt.testHistory ??= []).push(structuredClone(attempt.evidence.tests)) - attempt.evidence.tests = { logicalRole: deliveryRole(attempt, 'tester'), roleSessionId: attempt.roleSessions?.[deliveryRole(attempt, 'tester')]?.id ?? null, executor: 'independent-host-controller', workspaceId: workspace.id, resourceId: workspace.resourceId, sha: attempt.candidateSha, passed: !error, status: outcome.status, reason: outcome.reason, exitConfirmed: outcome.exitConfirmed, testsStarted: stage === 'tests', preparation, command: settings.testCommand, log, sha256: hash(output), startedAt, at: Date.now() } + attempt.evidence.tests = { logicalRole: deliveryRole(attempt, 'tester'), roleSessionId: attempt.roleSessions?.[deliveryRole(attempt, 'tester')]?.id ?? null, executor: 'independent-host-controller', workspaceId: workspace.id, resourceId: workspace.resourceId, sha: attempt.candidateSha, passed: !error, status: outcome.status, stage: outcome.stage ?? stage, reason: outcome.reason, exitConfirmed: outcome.exitConfirmed, testsStarted: stage === 'tests', preparation, command: settings.testCommand, log, sha256: hash(output), startedAt, at: Date.now() } if (error) throw Object.assign(reject(outcome.status === 'environment-blocked' ? 'Test environment is blocked; verification is incomplete' : outcome.status === 'execution-timeout' ? 'Test execution timed out; verification is incomplete' : 'Configured tests failed; see Attempt evidence'), { outcome: outcome.status }) } async function review(attempt, outputs) { @@ -961,7 +961,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { else await workspaceGit(workspace, ['reset', '--hard', attempt.candidateSha]) } if (project.repository) { await prepareWorkspace(attempt, workspace, settings); await prepareIntegrations(attempt, workspace) } - context = await buildProjectContext(rt, { retrieval: true, policy, projectId: project.id, taskId: task.id, role, revision: pinned.revision, resourceId: lease?.resourceId, definition, memory: definition.memory, knowledge: ownEntry(state.projectKnowledge ?? {}, project.id) ?? [], requirement: attempt.requirement, feedback: attempt.feedback, projectFacts: { environmentPreparation: workspace.environmentPreparation ? { id: workspace.environmentPreparation.id, status: workspace.environmentPreparation.status, executor: workspace.environmentPreparation.executor, identitySha256: workspace.environmentPreparation.identitySha256 ?? null, sha256: workspace.environmentPreparation.sha256 } : null, workspaceMaterialization: workspace.materialization ?? null, ...(pinned.change ? { roleDefinitionChange: pinned.change } : {}), integrationInputs: structuredClone(attempt.integrationInputs ?? []), testFailure: await failureContext(attempt), publicationBinding: attempt.publicationBinding ?? null, integrationInstruction: attempt.publicationBinding ? 'Merge the pinned upstreamSha into the published candidate using --no-commit, resolve and stage conflicts before handoff. Host sealing retains both trusted inputs; never fabricate parents or discard backend changes.' : null, workingDraft: !attempt.candidateSha && workspaceFor(attempt, 'developer') ? { kind: 'unverified-workspace-reference', resourceId: workspaceFor(attempt, 'developer').resourceId, path: workspaceFor(attempt, 'developer').resourceId === workspace.resourceId ? workspaceFor(attempt, 'developer').path : null, note: 'Unsealed source, not candidate or verification evidence. Cross-resource drafts require a sealed artifact before verification.' } : null, handoff: task.collaboration?.history.at(-1) ? { purpose: task.collaboration.history.at(-1).purpose, reason: task.collaboration.history.at(-1).reason } : null, candidateSha: attempt.candidateSha ?? null, tests: attempt.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, status: attempt.evidence.tests.status ?? null, reason: attempt.evidence.tests.reason ?? null, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, review: attempt.evidence.review ? { sha: attempt.evidence.review.sha, verdict: attempt.evidence.review.verdict, summary: attempt.evidence.review.summary } : null }, repository: await repositoryContext(project.id, attempt.baseSha, project) }) + context = await buildProjectContext(rt, { retrieval: true, policy, projectId: project.id, taskId: task.id, role, revision: pinned.revision, resourceId: lease?.resourceId, definition, memory: definition.memory, knowledge: ownEntry(state.projectKnowledge ?? {}, project.id) ?? [], requirement: attempt.requirement, feedback: attempt.feedback, projectFacts: { environmentPreparation: workspace.environmentPreparation ? { id: workspace.environmentPreparation.id, status: workspace.environmentPreparation.status, executor: workspace.environmentPreparation.executor, identitySha256: workspace.environmentPreparation.identitySha256 ?? null, sha256: workspace.environmentPreparation.sha256 } : null, workspaceMaterialization: workspace.materialization ?? null, ...(pinned.change ? { roleDefinitionChange: pinned.change } : {}), integrationInputs: structuredClone(attempt.integrationInputs ?? []), testFailure: await failureContext(attempt), publicationBinding: attempt.publicationBinding ?? null, integrationInstruction: attempt.publicationBinding ? 'Merge the pinned upstreamSha into the published candidate using --no-commit, resolve and stage conflicts before handoff. Host sealing retains both trusted inputs; never fabricate parents or discard backend changes.' : null, workingDraft: !attempt.candidateSha && workspaceFor(attempt, 'developer') ? { kind: 'unverified-workspace-reference', resourceId: workspaceFor(attempt, 'developer').resourceId, path: workspaceFor(attempt, 'developer').resourceId === workspace.resourceId ? workspaceFor(attempt, 'developer').path : null, note: 'Unsealed source, not candidate or verification evidence. Cross-resource drafts require a sealed artifact before verification.' } : null, handoff: task.collaboration?.history.at(-1) ? { purpose: task.collaboration.history.at(-1).purpose, reason: task.collaboration.history.at(-1).reason } : null, candidateSha: attempt.candidateSha ?? null, tests: attempt.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, status: attempt.evidence.tests.status ?? null, stage: attempt.evidence.tests.stage ?? null, reason: attempt.evidence.tests.reason ?? null, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, review: attempt.evidence.review ? { sha: attempt.evidence.review.sha, verdict: attempt.evidence.review.verdict, summary: attempt.evidence.review.summary } : null }, repository: await repositoryContext(project.id, attempt.baseSha, project) }) } finally { if (preparation && !state.projectResourceLeases?.[lease.id]?.preparation?.exitUnknown) await rt.projectResources.endPreparation(preparation) } } catch (error) { const registered = lease && ownEntry(state.projectResourceLeases ?? {}, lease.id) diff --git a/core/src/host/project-execution-environment.js b/core/src/host/project-execution-environment.js index 3b1e94ec2..790192807 100644 --- a/core/src/host/project-execution-environment.js +++ b/core/src/host/project-execution-environment.js @@ -23,6 +23,7 @@ export function executionOutcome(error, output = '', { stage = 'tests', remote = if (remote && (timedOut || typeof error.code !== 'number' || [125, 255].includes(error.code))) return { status: 'execution-unknown', reason: 'resource-command-exit-unconfirmed', exitConfirmed: false } if (timedOut) return { status: 'execution-timeout', reason: 'command-deadline', exitConfirmed: true } if (stage === 'preparation') return { status: 'environment-blocked', reason: 'preparation-command-failed', exitConfirmed: true } + if (error.code === 127 || error.code === 'ENOENT' || /(?:^|[\s/])[^\s:]+:\s*(?:command )?not found\b/i.test(output)) return { status: 'environment-blocked', reason: 'test-runner-unavailable', stage: 'runner-startup', exitConfirmed: true } if (['ENOENT', 'EACCES'].includes(error.code) || /\b(?:ENOTCACHED|only-if-cached|ERR_MODULE_NOT_FOUND|MODULE_NOT_FOUND|EAI_AGAIN|ENOTFOUND|ENETUNREACH|ECONNREFUSED)\b|Cannot find (?:module|package)|command not found|network (?:is unreachable|access.*denied)|registry.*(?:denied|unavailable)/i.test(output)) return { status: 'environment-blocked', reason: 'dependency-tool-cache-or-network-unavailable', exitConfirmed: true } if (typeof error.code !== 'number') return { status: 'execution-unknown', reason: 'test-exit-unconfirmed', exitConfirmed: false } return { status: /AssertionError|ERR_ASSERTION|assertion failed/i.test(output) ? 'assertion-failed' : 'test-process-failed', reason: 'nonzero-test-exit', exitConfirmed: true } diff --git a/core/src/host/project-service-views.js b/core/src/host/project-service-views.js index 1f87883a1..1147faeca 100644 --- a/core/src/host/project-service-views.js +++ b/core/src/host/project-service-views.js @@ -63,7 +63,7 @@ export function attemptMetadataView(attempt) { if (attempt.evidence?.tests?.preparation) result.environment = pick(attempt.evidence.tests.preparation, ['id', 'status', 'executor', 'workspaceId', 'resourceId', 'identitySha256', 'sha256']) if (attempt.workspaces) result.workspaces = Object.fromEntries(Object.entries(attempt.workspaces).slice(0, 32).map(([role, row]) => [role, pick(row, ['id', 'status', 'role', 'resourceId', 'materialization'])])) if (attempt.agentDefinitions) result.agentDefinitions = { revision: attempt.agentDefinitions.revision, roles: (attempt.agentDefinitions.roles ?? []).slice(0, 32).map(row => ({ ...pick(row, ['id', 'role', 'policyRole', 'name', 'model']), skills: (row.skills ?? []).slice(0, 20).map(value => short(value, 300)), execution: pick(row.execution, ['harness', 'model', 'resourceIds']) })) } - function evidence(row) { return { ...pick(row, ['sha', 'sha256', 'passed', 'status', 'reason', 'executor', 'testsStarted', 'exitConfirmed', 'verdict', 'harness', 'sdkVersion', 'resourceId', 'sessionId', 'model', 'startedAt', 'completedAt']), ...(row.summary ? { summary: short(row.summary) } : {}) } } + function evidence(row) { return { ...pick(row, ['sha', 'sha256', 'passed', 'status', 'stage', 'reason', 'executor', 'testsStarted', 'exitConfirmed', 'verdict', 'harness', 'sdkVersion', 'resourceId', 'sessionId', 'model', 'startedAt', 'completedAt']), ...(row.summary ? { summary: short(row.summary) } : {}) } } result.evidence = {} for (const [key, row] of Object.entries(attempt.evidence ?? {}).slice(0, 32)) { if (key === 'execution') result.evidence.execution = Object.fromEntries(Object.entries(row).slice(0, 32).map(([role, value]) => [role, evidence(value)])) diff --git a/core/test/project-delivery.test.js b/core/test/project-delivery.test.js index 293071a58..aae6a0d45 100644 --- a/core/test/project-delivery.test.js +++ b/core/test/project-delivery.test.js @@ -991,6 +991,42 @@ test('independent verification preserves dependency diagnostics without declarin assert.match(readFileSync(tests.log, 'utf8'), /ENOTCACHED/) }) +test('npm test missing vitest after clean is runner-startup blockage and preparation reruns real assertions on the same candidate', async t => { + const f = await setup(t, undefined, { pool: true }) + f.rt.projectPools.connect({ validate: () => ({ ok: true }) }) + const settings = f.rt.config.projectRepositories['example/demo'] + settings.allowedPaths.push('package.json') + const wrapper = 'const cp=require("node:child_process");const r=cp.spawnSync("npm",["--offline","test"],{stdio:"inherit",env:{...process.env,HOME:process.argv[1],npm_config_cache:process.argv[1]+"/npm-cache"}});process.exit(r.status??1)' + settings.testCommand = [process.execPath, '-e', wrapper, f.directory] + await f.start('missing-vitest') + const attempt = f.attempt(), task = f.rt.state.tasks[f.task.id], roles = f.rt.projectAgents.deliveryDefinitions('p1').roles + await prepareRole(f.rt, task, { id: 'missing-runner-dev', attemptId: attempt.id, generation: 1 }, roles.find(row => row.role === 'developer')) + writeFileSync(join(attempt.workspaces.developer.path, 'code.mjs'), 'export const add=(a,b)=>a+b;\n') + writeFileSync(join(attempt.workspaces.developer.path, 'package.json'), JSON.stringify({ scripts: { test: 'vitest run' } })) + await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: attempt.id }, role: 'developer', outputs: { implementation: 'Implement addition' } }) + await prepareRole(f.rt, task, { id: 'missing-runner-test', attemptId: attempt.id, generation: 2 }, roles.find(row => row.role === 'tester')) + const sha = attempt.candidateSha, path = attempt.workspaces.tester.path + mkdirSync(join(path, 'node_modules/.bin'), { recursive: true }) + writeFileSync(join(path, 'node_modules/stale'), 'Must be cleaned') + const finish = () => f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: attempt.id }, role: 'tester', outputs: { tests: 'Request host verification' } }) + await assert.rejects(finish(), /Configured tests failed|environment is blocked/) + const failed = structuredClone(attempt.evidence.tests) + assert.equal(failed.status, 'environment-blocked'); assert.equal(failed.stage, 'runner-startup') + assert.equal(failed.testsStarted, true); assert.equal(failed.preparation, null); assert.equal(failed.passed, false) + assert.match(readFileSync(failed.log, 'utf8'), /vitest: (?:command )?not found/) + assert.equal((await import('node:fs')).existsSync(join(path, 'node_modules/stale')), false) + // A transport runner fixture delegates to the real Node assertions; this is + // not an installed Vitest or a paid-model acceptance claim. + const runner = '#!/usr/bin/env node\nconst r=require("node:child_process").spawnSync(process.execPath,["--test","--test-reporter=tap","code.test.mjs"],{stdio:"inherit"});process.exit(r.status??1)\n' + settings.workspacePrepareCommand = [process.execPath, '-e', `const fs=require('fs');fs.mkdirSync('node_modules/.bin',{recursive:true});fs.writeFileSync('node_modules/.bin/vitest',${JSON.stringify(runner)},{mode:0o755})`] + await finish() + assert.equal(attempt.candidateSha, sha); assert.equal(attempt.evidence.tests.sha, sha) + assert.equal(attempt.evidence.tests.status, 'passed'); assert.equal(attempt.evidence.tests.stage, 'tests') + assert.equal(attempt.evidence.tests.preparation.status, 'ready') + assert.deepEqual(attempt.testHistory[0], failed) + assert.match(readFileSync(attempt.evidence.tests.log, 'utf8'), /# pass 1/) +}) + test('independent test timeout remains incomplete verification with an immutable log', async t => { const f = await setup(t, [process.execPath, '-e', 'setTimeout(()=>{},10000)']) await f.start('test-timeout') diff --git a/core/test/project-execution-environment.test.js b/core/test/project-execution-environment.test.js index 6ca52fb75..d654fc0c1 100644 --- a/core/test/project-execution-environment.test.js +++ b/core/test/project-execution-environment.test.js @@ -18,6 +18,12 @@ test('environment outcomes retain non-success and separate dependency, assertion assert.equal(executionOutcome({ killed: true }, '', { remote: true }).exitConfirmed, false) assert.equal(executionOutcome({ code: 'ECONNRESET' }, '', { remote: true }).status, 'execution-unknown') assert.equal(executionOutcome({ code: 37 }, '', { stage: 'preparation' }).status, 'environment-blocked') + for (const output of ['sh: 1: vitest: not found', '/bin/sh: vitest: command not found']) { + assert.equal(executionOutcome({ code: 127 }, output).status, 'environment-blocked') + assert.equal(executionOutcome({ code: 127 }, output).stage, 'runner-startup') + } + assert.equal(executionOutcome({ code: 127 }, '').reason, 'test-runner-unavailable') + assert.equal(executionOutcome({ code: 1 }, 'AssertionError: expected true to be false').status, 'assertion-failed') }) test('dependency contract rejects escaping paths, arbitrary network modes and malformed commands', () => { for (const spec of [{ lockFiles: ['../package-lock.json'] }, { npmCachePath: '/owner/cache' }, { tools: [{ argv: 'npm --version' }] }, { network: 'full-access' }, { timeoutMs: 0 }, { arbitraryAuthority: true }]) assert.throws(() => dependencyEnvironmentCheck(spec), { code: 'bot/repository-config' }) diff --git a/docs/backend-operations.md b/docs/backend-operations.md index fb47dd4e0..be0eba52e 100644 --- a/docs/backend-operations.md +++ b/docs/backend-operations.md @@ -159,3 +159,9 @@ not just passed=false. The original Task/Chief controls decide retry or consulta fixing environment can rerun the same candidate while retaining old logs. Unknown remote exit holds capacity until trusted recovery evidence. Use attempt.context (testHistory/preparationFailures) and attempt.log(environmentRole) for bounded reads. + +Missing `vitest` after candidate clean is an environment blocker even when npm +starts: inspect stage=runner-startup and reason=test-runner-unavailable. Exit127 is +not a business assertion failure. Configure an authorized workspacePrepareCommand +for every host-verifier checkout; a manual npm ci or Docker preparation is not a +replacement for this hook, and configuration alone is not a same-candidate test PASS. diff --git a/docs/history/2026-11-04-collaboration-delivery-checkpoints.md b/docs/history/2026-11-04-collaboration-delivery-checkpoints.md new file mode 100644 index 000000000..757820fd6 --- /dev/null +++ b/docs/history/2026-11-04-collaboration-delivery-checkpoints.md @@ -0,0 +1,49 @@ +# Earlier collaboration and frontend implementation checkpoints + +Archived verbatim from ROADMAP.md to keep required role policy within its existing +384000-byte limit. No authorization, evidence gate or pending acceptance changed. + +[Tracking #33](https://github.com/zy95-12/FlyBot/issues/33): #34 repository preflight → #35 role contracts → #36 per-Bot pools/handoff → #37 ordinary Chief queries → #38 safe/watch delegation → #39 Chief arbitration/configurable schedules → #40 native UI/multi-Bot reuse → #41 actual shared-service acceptance. Component/UI inventory may proceed alongside its foundations; each implementation has its own PR and tested evidence. Public hosting is deferred. A 30-minute Chief inspection is a configurable scheduled-task default, never framework logic. R1 PR #42 passed full local verification; configured-service acceptance is pending. R2 implements complete role-policy delivery and evidence separation and is under verification. R3 dynamic pool/claim/handoff implementation is under verification, including non-DAG SDK contract-process rework. R4 conversation/query/status repair is under verification; R5 persistent safe/watch delegation is under verification; R6–R8 remain pending; R4 integrates conversation bridging and R6 completes Chief arbitration/configurable schedules. The owner requests backend repairs first, then real human-style assignment through FlyBot to exercise role handoffs and Chief fallback. + +R6 implementation checkpoint: Chief now adjudicates completion/uncertainty and may +intervene through audited routes. Inspection reuses original configurable DSH +schedules; there is no Chief cadence in framework timers. After exact-tip full +verification and its own draft PR, upgrade the backed-up shared backend, assign +#40 through actual FlyBot and collect real Codex/handoff/Chief/restart evidence in +#41. Protocol fixtures alone do not satisfy those integration criteria. + + +Task40 continues with synchronized native CSS/JS loading and a scoped composer +exported from the original composer module, sharing dictation cleanup and captured +chat sends. Focused regression evidence is separate from complete UI acceptance; +rich tool/approval, personal running state and deployed browser acceptance remain open. + +Task40 native interaction continuation implements personal questions/one-shot approval, +stop/continue, own-chat streaming, bounded tool output and original group member-work +controls. SDK execution input reuses original approval/composer components. Proceed to +exact-candidate host tests and independent Review; only operator evidence can satisfy +actual deployed three-account/weak-network and visual acceptance. + +Task40 host-return repair preserves legacy listener registration while retaining the +explicit shared native stream observer. Reproduced all four reported failures and +added a fail-without-fix mode-boundary regression. Continue through host tests/Review +and operator acceptance on the same Task/PR48. + +Task40 review-return continuation repairs live-part navigation and replaces plain human +paragraphs with original person bubbles. Native file/image sends and scoped historical +attachment reads are implemented through public DSH APIs with explicit shared-entry +byte limits. Proceed through host verification/independent Review; operator deployed +three-account, weak-network and screenshot evidence remains separately required. + +Task40 refresh repair retains scoped native roots and updates props instead of +unmounting on each poll. Bounded mounted-workbench regressions cover focus, selection, +drafts, sends and teardown for personal/public/private/room views. Continue through +new candidate verification and Review without borrowing earlier candidate evidence. + +Task40/PR48 integrates preserved frontend 0e72dc60 with operator backend b920cae8, +including S1 and verified workspace/Session exit proofs. Compact board status +uses the same live-activity evidence as the full S1 view. Integrated focused tests +pass; exact sealed candidate host verification and independent Review come next. +Actual shared-service browser/weak-network/screenshot acceptance stays outstanding. + + diff --git a/docs/openapi/flybot.openapi.json b/docs/openapi/flybot.openapi.json index 7b7bba738..ec3131c4c 100644 --- a/docs/openapi/flybot.openapi.json +++ b/docs/openapi/flybot.openapi.json @@ -4587,6 +4587,14 @@ "reason": { "type": "string" }, + "stage": { + "type": "string", + "enum": [ + "preparation", + "runner-startup", + "tests" + ] + }, "testsStarted": { "type": "boolean" }, @@ -30583,7 +30591,7 @@ "notes": "Although a query, the downstream integration checkpoint gate requires manager authority and confirmed release.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" } }, { @@ -30661,7 +30669,7 @@ "notes": "UTF-8 byte offsets, not character offsets; nextOffset preserves character boundaries. Rechecks rights after source I/O. No caller path. environmentRole selects that role workspace latest preparation log and is mutually exclusive with SDK execution role. Preparation is not tests or inference.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" } }, { @@ -30822,7 +30830,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-chief-control.js", - "sha256": "6bc492f92f46142a208149ea54cdc11cdcdd3c48c8f825c1e780fed98a2d9aff" + "sha256": "58f96d2f25aa925e806b00e547e6eefcaf99549d5d5d71abe99e77abecc7a65e" } }, { @@ -32710,7 +32718,7 @@ "notes": "New Project Task contracts require actual Chief adjudication; third-party clients cannot forge Chief credentials.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" } }, { @@ -32818,7 +32826,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" } }, { @@ -32925,7 +32933,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" } }, { @@ -33019,7 +33027,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" } }, { @@ -33113,7 +33121,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" } }, { @@ -33222,7 +33230,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" } }, { @@ -33797,7 +33805,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-chief-control.js", - "sha256": "6bc492f92f46142a208149ea54cdc11cdcdd3c48c8f825c1e780fed98a2d9aff" + "sha256": "58f96d2f25aa925e806b00e547e6eefcaf99549d5d5d71abe99e77abecc7a65e" } }, { @@ -33897,7 +33905,7 @@ "notes": "30-minute inspection is a configurable schedule default, not a framework deadline. correctionBudget defaults to 2 (0–4); protocol correction reuses the Chief Session. Exhaustion is incomplete judgement, never automatic consultation or default routing. inspection.lifecycle configures queue/execution deadlines, bounded recovery delay/count and health overdue threshold; timeout is not exit proof.", "implementation": { "path": "core/src/host/project-chief-control.js", - "sha256": "6bc492f92f46142a208149ea54cdc11cdcdd3c48c8f825c1e780fed98a2d9aff" + "sha256": "58f96d2f25aa925e806b00e547e6eefcaf99549d5d5d71abe99e77abecc7a65e" } }, { @@ -43639,7 +43647,7 @@ "notes": "New Project Task contracts require actual Chief adjudication; third-party clients cannot forge Chief credentials.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" }, "aliasOf": "development.command.task.accept" }, @@ -43732,7 +43740,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" }, "aliasOf": "development.command.task.reject" }, @@ -43824,7 +43832,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" }, "aliasOf": "development.command.attempt.stop" }, @@ -43916,7 +43924,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" }, "aliasOf": "development.command.attempt.cleanup" }, @@ -44295,7 +44303,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" }, "aliasOf": "development.command.attempt.publication-retry" }, @@ -44371,7 +44379,7 @@ "notes": "UTF-8 byte offsets, not character offsets; nextOffset preserves character boundaries. Rechecks rights after source I/O. No caller path. environmentRole selects that role workspace latest preparation log and is mutually exclusive with SDK execution role. Preparation is not tests or inference.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273" + "sha256": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890" }, "aliasOf": "development.query.attempt.log" }, @@ -44679,7 +44687,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-chief-control.js", - "sha256": "6bc492f92f46142a208149ea54cdc11cdcdd3c48c8f825c1e780fed98a2d9aff" + "sha256": "58f96d2f25aa925e806b00e547e6eefcaf99549d5d5d71abe99e77abecc7a65e" }, "aliasOf": "development.command.chief.inspect" }, @@ -44777,7 +44785,7 @@ "notes": "30-minute inspection is a configurable schedule default, not a framework deadline. correctionBudget defaults to 2 (0–4); protocol correction reuses the Chief Session. Exhaustion is incomplete judgement, never automatic consultation or default routing. inspection.lifecycle configures queue/execution deadlines, bounded recovery delay/count and health overdue threshold; timeout is not exit proof.", "implementation": { "path": "core/src/host/project-chief-control.js", - "sha256": "6bc492f92f46142a208149ea54cdc11cdcdd3c48c8f825c1e780fed98a2d9aff" + "sha256": "58f96d2f25aa925e806b00e547e6eefcaf99549d5d5d71abe99e77abecc7a65e" }, "aliasOf": "development.command.chief.configure" }, @@ -44853,7 +44861,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-chief-control.js", - "sha256": "6bc492f92f46142a208149ea54cdc11cdcdd3c48c8f825c1e780fed98a2d9aff" + "sha256": "58f96d2f25aa925e806b00e547e6eefcaf99549d5d5d71abe99e77abecc7a65e" }, "aliasOf": "development.query.chief.state" }, @@ -45717,7 +45725,7 @@ "core/src/host/project-agents.js": "9edc8a4af53347b2b14d5f26b739a7f6683a4767358b86e520ddcfea1162fa9c", "core/src/host/project-assistant.js": "4fa4309d4aa7abf78b6ba4e55ba754f320384f931e0e4a5c6620c859e1b50676", "core/src/host/project-board.js": "dfb0b4bc05ae42335632e3edbba883a03c5da2fb17899b23e8226320ea08aec4", - "core/src/host/project-chief-control.js": "6bc492f92f46142a208149ea54cdc11cdcdd3c48c8f825c1e780fed98a2d9aff", + "core/src/host/project-chief-control.js": "58f96d2f25aa925e806b00e547e6eefcaf99549d5d5d71abe99e77abecc7a65e", "core/src/host/project-chief-lifecycle.js": "e854d2fb22b4ea5041ea63a7f072f5af61863c9703ed063b82d0858a15129c68", "core/src/host/project-chief-protocol.js": "a09d85e04171b22625d3eed131dd6a5227dc1a946ad39d8a59f017bb75f71936", "core/src/host/project-chief.js": "a3f01a053177fd8f10d4b778d79ac98c25aee70aa3c4d967dc7e1da2aea2cf6e", @@ -45731,14 +45739,14 @@ "core/src/host/project-credentials.js": "18fe61057fb920664fc5077d81eee0957f7eaf64821dd10d2d2a3cd463f5fdea", "core/src/host/project-delegations.js": "d4d86192551994a3419e0d2ca0f64bc4d50a75960970d2b79ed9f97b82940903", "core/src/host/project-delivery-template.js": "daa5cc8185abc1ec84421ae3fcd5574a50573cfd1a7c4cb10941158e29c79a13", - "core/src/host/project-delivery.js": "12a4f42aaec09dc79580f73f15c2787fc65c1a4f085de65c73cddaf317be0273", + "core/src/host/project-delivery.js": "0e2de57798d9d121ccd1e6b93466fe791cd9baa41b5ae59ac52fd76fa7ad0890", "core/src/host/project-development-service.js": "857847e416da641c8699a2c02e4c02b80ca448fcd908cdc0192cc7cf6aefdf2b", "core/src/host/project-domain.js": "6d6a20727905005f7a8de0247a429a10ffbd1436773ee96a5d7615b2fb763e82", "core/src/host/project-effects.js": "5557782333a0b3437dacde2b6924fe12ea54225098571fa8e404d79a432be549", "core/src/host/project-execution-authority.js": "9c5ffb4bc9153d8f9d9e7901f7aef2aee90cacc19f19d8a0726ac0adb93e3e36", "core/src/host/project-execution-boundary.js": "9e7afe568024e19517a3d5430a0483a6ea5e2c910a53de056d83db3896d3eb16", "core/src/host/project-execution-capacity.js": "751b48e26f6ed1ad4c000337174b39ab4c113c494cbb7bd5a2a8369cbae10aa4", - "core/src/host/project-execution-environment.js": "26c68a7a5d456d6287081ef9f3edaf487327ef3b7bb23b35fa159003d8f0458c", + "core/src/host/project-execution-environment.js": "bab932919fe2aa6ef9cae2021c3f1b3ffc365b6a9ebb6663ce2a70bf61b5360c", "core/src/host/project-execution-exit.js": "50f94881a40c2d0c6288fd081c885c7cabb71f8abec3a8c918e1c0bc852a04f7", "core/src/host/project-execution-profile.js": "7106593979d6e21b9e1dab567c8628a7d0252976ed47d42dace337944725da59", "core/src/host/project-git-sandbox.js": "b579b01dcce0708275e9f735d08932481369803382b8f7d24af69639608d724f", @@ -45772,7 +45780,7 @@ "core/src/host/project-scheduler.js": "9419010ed18fcfe801cab23562ea2ad53cb4dc500078ff1ec2af4438cc6726a6", "core/src/host/project-service-events.js": "71917d1ab367a9db8bef8c1d1003228255a57528e84a57e76de3a39d08ac28ec", "core/src/host/project-service-sse.js": "df97d6b11156da885a6a47d50a4a77c8c99b6af5eb0066de14312cc512762297", - "core/src/host/project-service-views.js": "5eb689249adccf55440d196c7e4b17e0e0f0232c8338bc1b3ea3abc0e3d84146", + "core/src/host/project-service-views.js": "5d04f3e21e27d85697de1aba8e009ab3ad19234b1d18796fa2a1f7f302ed7fbc", "core/src/host/project-service.js": "8ba30c1d7c066867eae06a6612d5bbba3bc2278f34fcf7f1c2242876565dba8f", "core/src/host/project-sources.js": "cf4c11c8e853f3091c4d159b850d06fe7e8bf291fdc616d251eb23d071d63403", "core/src/host/project-state-format.js": "4838b344439e585287fdc25ef15cea44d7aff4199c7a380ebbfa8cfcdf24f678", diff --git a/docs/subsystems/project-chief-control.md b/docs/subsystems/project-chief-control.md index de6b306c6..0d390d7b8 100644 --- a/docs/subsystems/project-chief-control.md +++ b/docs/subsystems/project-chief-control.md @@ -259,3 +259,6 @@ empty inspections can succeed without changing a Task. Persisted state remains t source of truth, not a misleading enabled flag or an in-memory waiter. #2334 supplies structured test outcome, testsStarted, actual executor and preparation receipt references plus bounded SDK dependency observations. Chief chooses recovery; an environment blockage is neither an assertion failure nor automatic escalation. + +Missing-runner facts additionally include stage=runner-startup. testsStarted is +configured command invocation; Chief must not interpret it as assertion execution. diff --git a/docs/subsystems/project-delivery.md b/docs/subsystems/project-delivery.md index bd94d539e..cfcd55140 100644 --- a/docs/subsystems/project-delivery.md +++ b/docs/subsystems/project-delivery.md @@ -627,3 +627,13 @@ client does not prove its container command exited. Existing hash-bound pages expose attempt.context testHistory/preparationFailures; attempt.log environmentRole selects the latest preparation for a registered role, mutually exclusive with execution role. No caller filesystem path is accepted. + +#2334 updated acceptance: shell exit127, `vitest: not found` and `command not found` +are environment-blocked with reason test-runner-unavailable and stage runner-startup. +`testsStarted=true` means the configured process was invoked, never that assertions +ran. Preparation failure has testsStarted=false/stage=preparation; other configured +verification is stage=tests. Remove the upstream NODE_TEST_CONTEXT marker from the +independent execution environment so nested Node tests execute real assertions. +Actual missing-runner npm tests and an explicit local runner fixture delegating to +Node assertions verify clean→prepare→same-candidate retry; the latter is not installed +Vitest or live npm-ci/provider acceptance. diff --git a/docs/subsystems/project-service-contracts.md b/docs/subsystems/project-service-contracts.md index d3216fca4..23b9d7863 100644 --- a/docs/subsystems/project-service-contracts.md +++ b/docs/subsystems/project-service-contracts.md @@ -140,3 +140,7 @@ Operation names scope requestId, so different operations may reuse a key. When a #2325 adds inspect/integrate purpose to scoped approved input metadata and sanitized workspace materialization facts (base/seed/expected/actual SHA). No private paths, key material or workspace contents are added to Attempt projections. #2334 includes verification status/reason/executor/testsStarted/exitConfirmed in Attempt summaries. Hash-bound attempt.context also serves testHistory and preparationFailures; attempt.log.environmentRole selects registered preparation evidence without accepting a filesystem path. API components document EnvironmentPreparation and TestVerification; role SDK observations remain advisory. + +TestVerification.stage distinguishes preparation, runner-startup and tests. The +optional field is present in Attempt summaries, Chief facts and source context; +legacy records may omit it. testsStarted describes configured-command invocation. diff --git a/patches/ds-bot/files/src/host/project-chief-control.js b/patches/ds-bot/files/src/host/project-chief-control.js index 8c72e1a98..8294b3784 100644 --- a/patches/ds-bot/files/src/host/project-chief-control.js +++ b/patches/ds-bot/files/src/host/project-chief-control.js @@ -142,7 +142,7 @@ export function createProjectChiefControl(rt) { repositoryRecovery: await rt.projectDelivery.recoveryFacts?.(task) ?? null, integrationInputs: structuredClone(attempt?.integrationInputs ?? []), attemptStatus: attempt?.status ?? null, generation: task.collaboration?.generation ?? 0, destination: task.collaboration?.botId ?? null, attemptId: attempt?.id ?? null, candidateSha: attempt?.candidateSha ?? null, evidence: { implementation: attempt?.evidence.implementation ? { sha: attempt.evidence.implementation.sha, paths: attempt.evidence.implementation.paths } : null, - tests: attempt?.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, status: attempt.evidence.tests.status ?? null, reason: attempt.evidence.tests.reason ?? null, testsStarted: attempt.evidence.tests.testsStarted ?? null, preparation: attempt.evidence.tests.preparation ? { id: attempt.evidence.tests.preparation.id, status: attempt.evidence.tests.preparation.status, executor: attempt.evidence.tests.preparation.executor, sha256: attempt.evidence.tests.preparation.sha256 } : null, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, + tests: attempt?.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, status: attempt.evidence.tests.status ?? null, stage: attempt.evidence.tests.stage ?? null, reason: attempt.evidence.tests.reason ?? null, testsStarted: attempt.evidence.tests.testsStarted ?? null, preparation: attempt.evidence.tests.preparation ? { id: attempt.evidence.tests.preparation.id, status: attempt.evidence.tests.preparation.status, executor: attempt.evidence.tests.preparation.executor, sha256: attempt.evidence.tests.preparation.sha256 } : null, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, review: attempt?.evidence.review ? { sha: attempt.evidence.review.sha, verdict: attempt.evidence.review.verdict, summary: attempt.evidence.review.summary, repositoryVerdict: attempt.evidence.review.report?.repositoryVerdict ?? 'INC' } : null, outputs: attempt?.evidence.developer?.outputs ?? null, pr: attempt?.pr ? { number: attempt.pr.number, sha: attempt.pr.sha, url: attempt.pr.url } : null }, inspection: facts } })) diff --git a/patches/ds-bot/files/src/host/project-delivery.js b/patches/ds-bot/files/src/host/project-delivery.js index ab497b3ec..52a8ceb8e 100644 --- a/patches/ds-bot/files/src/host/project-delivery.js +++ b/patches/ds-bot/files/src/host/project-delivery.js @@ -36,7 +36,7 @@ const requirementsOf = (task, repository) => ({ title: task.title, criteria: tas ...(task.projectTask ? { projectTaskRevision: task.projectTask.requirements.revision, impact: task.projectTask.requirements.impact, scope: task.projectTask.requirements.scope } : {}) }) const reject = message => Object.assign(new Error(message), { code: 'bot/delivery' }) const inside = (root, path) => { const rel = relative(root, path); return rel !== '' && rel !== '..' && !rel.startsWith(`..${sep}`) && !isAbsolute(rel) } -const cleanEnv = () => Object.fromEntries(Object.entries(process.env).filter(([key]) => !/TOKEN|PASSWORD|SECRET|API_KEY|GIT_CONFIG|NODE_OPTIONS/.test(key))) +const cleanEnv = () => Object.fromEntries(Object.entries(process.env).filter(([key]) => !/TOKEN|PASSWORD|SECRET|API_KEY|GIT_CONFIG|NODE_OPTIONS|NODE_TEST_CONTEXT/.test(key))) async function git(path, args, preserveWhitespace = false) { const { stdout } = await exec('git', ['-c', 'core.hooksPath=/dev/null', '-c', 'commit.gpgsign=false', ...args], { cwd: path, timeout: 30_000, maxBuffer: 2 * 1024 * 1024, env: { ...cleanEnv(), GIT_TERMINAL_PROMPT: '0', @@ -126,7 +126,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { if (source !== attempt && tests.sha !== attempt.seedSha) return null const text = await readFile(tests.log, 'utf8') if (hash(text) !== tests.sha256) throw reject('Failure diagnostic log changed') - return { status: tests.status ?? 'test-process-failed', reason: tests.reason ?? null, executor: tests.executor, testsStarted: tests.testsStarted ?? true, preparation: tests.preparation ? { id: tests.preparation.id, status: tests.preparation.status, sha256: tests.preparation.sha256, executor: tests.preparation.executor } : null, sourceAttemptId: source.id, sha: tests.sha, logSha256: tests.sha256, excerpt: testFailureExcerpt(text, process.env), completeLogRetained: true, source: 'project-attempt-log; bounded host-verified diagnostic, not a passing result' } + return { status: tests.status ?? 'test-process-failed', stage: tests.stage ?? null, reason: tests.reason ?? null, executor: tests.executor, testsStarted: tests.testsStarted ?? true, preparation: tests.preparation ? { id: tests.preparation.id, status: tests.preparation.status, sha256: tests.preparation.sha256, executor: tests.preparation.executor } : null, sourceAttemptId: source.id, sha: tests.sha, logSha256: tests.sha256, excerpt: testFailureExcerpt(text, process.env), completeLogRetained: true, source: 'project-attempt-log; bounded host-verified diagnostic, not a passing result' } } async function archiveForNative(context, attempt, role, suffix) { if (!context.archive) return @@ -620,7 +620,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { if (error && !output) output = 'Verification did not complete; inspect the structured outcome and preparation receipt' await writeFile(log, output, { mode: 0o600 }) if (attempt.evidence.tests) (attempt.testHistory ??= []).push(structuredClone(attempt.evidence.tests)) - attempt.evidence.tests = { logicalRole: deliveryRole(attempt, 'tester'), roleSessionId: attempt.roleSessions?.[deliveryRole(attempt, 'tester')]?.id ?? null, executor: 'independent-host-controller', workspaceId: workspace.id, resourceId: workspace.resourceId, sha: attempt.candidateSha, passed: !error, status: outcome.status, reason: outcome.reason, exitConfirmed: outcome.exitConfirmed, testsStarted: stage === 'tests', preparation, command: settings.testCommand, log, sha256: hash(output), startedAt, at: Date.now() } + attempt.evidence.tests = { logicalRole: deliveryRole(attempt, 'tester'), roleSessionId: attempt.roleSessions?.[deliveryRole(attempt, 'tester')]?.id ?? null, executor: 'independent-host-controller', workspaceId: workspace.id, resourceId: workspace.resourceId, sha: attempt.candidateSha, passed: !error, status: outcome.status, stage: outcome.stage ?? stage, reason: outcome.reason, exitConfirmed: outcome.exitConfirmed, testsStarted: stage === 'tests', preparation, command: settings.testCommand, log, sha256: hash(output), startedAt, at: Date.now() } if (error) throw Object.assign(reject(outcome.status === 'environment-blocked' ? 'Test environment is blocked; verification is incomplete' : outcome.status === 'execution-timeout' ? 'Test execution timed out; verification is incomplete' : 'Configured tests failed; see Attempt evidence'), { outcome: outcome.status }) } async function review(attempt, outputs) { @@ -961,7 +961,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { else await workspaceGit(workspace, ['reset', '--hard', attempt.candidateSha]) } if (project.repository) { await prepareWorkspace(attempt, workspace, settings); await prepareIntegrations(attempt, workspace) } - context = await buildProjectContext(rt, { retrieval: true, policy, projectId: project.id, taskId: task.id, role, revision: pinned.revision, resourceId: lease?.resourceId, definition, memory: definition.memory, knowledge: ownEntry(state.projectKnowledge ?? {}, project.id) ?? [], requirement: attempt.requirement, feedback: attempt.feedback, projectFacts: { environmentPreparation: workspace.environmentPreparation ? { id: workspace.environmentPreparation.id, status: workspace.environmentPreparation.status, executor: workspace.environmentPreparation.executor, identitySha256: workspace.environmentPreparation.identitySha256 ?? null, sha256: workspace.environmentPreparation.sha256 } : null, workspaceMaterialization: workspace.materialization ?? null, ...(pinned.change ? { roleDefinitionChange: pinned.change } : {}), integrationInputs: structuredClone(attempt.integrationInputs ?? []), testFailure: await failureContext(attempt), publicationBinding: attempt.publicationBinding ?? null, integrationInstruction: attempt.publicationBinding ? 'Merge the pinned upstreamSha into the published candidate using --no-commit, resolve and stage conflicts before handoff. Host sealing retains both trusted inputs; never fabricate parents or discard backend changes.' : null, workingDraft: !attempt.candidateSha && workspaceFor(attempt, 'developer') ? { kind: 'unverified-workspace-reference', resourceId: workspaceFor(attempt, 'developer').resourceId, path: workspaceFor(attempt, 'developer').resourceId === workspace.resourceId ? workspaceFor(attempt, 'developer').path : null, note: 'Unsealed source, not candidate or verification evidence. Cross-resource drafts require a sealed artifact before verification.' } : null, handoff: task.collaboration?.history.at(-1) ? { purpose: task.collaboration.history.at(-1).purpose, reason: task.collaboration.history.at(-1).reason } : null, candidateSha: attempt.candidateSha ?? null, tests: attempt.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, status: attempt.evidence.tests.status ?? null, reason: attempt.evidence.tests.reason ?? null, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, review: attempt.evidence.review ? { sha: attempt.evidence.review.sha, verdict: attempt.evidence.review.verdict, summary: attempt.evidence.review.summary } : null }, repository: await repositoryContext(project.id, attempt.baseSha, project) }) + context = await buildProjectContext(rt, { retrieval: true, policy, projectId: project.id, taskId: task.id, role, revision: pinned.revision, resourceId: lease?.resourceId, definition, memory: definition.memory, knowledge: ownEntry(state.projectKnowledge ?? {}, project.id) ?? [], requirement: attempt.requirement, feedback: attempt.feedback, projectFacts: { environmentPreparation: workspace.environmentPreparation ? { id: workspace.environmentPreparation.id, status: workspace.environmentPreparation.status, executor: workspace.environmentPreparation.executor, identitySha256: workspace.environmentPreparation.identitySha256 ?? null, sha256: workspace.environmentPreparation.sha256 } : null, workspaceMaterialization: workspace.materialization ?? null, ...(pinned.change ? { roleDefinitionChange: pinned.change } : {}), integrationInputs: structuredClone(attempt.integrationInputs ?? []), testFailure: await failureContext(attempt), publicationBinding: attempt.publicationBinding ?? null, integrationInstruction: attempt.publicationBinding ? 'Merge the pinned upstreamSha into the published candidate using --no-commit, resolve and stage conflicts before handoff. Host sealing retains both trusted inputs; never fabricate parents or discard backend changes.' : null, workingDraft: !attempt.candidateSha && workspaceFor(attempt, 'developer') ? { kind: 'unverified-workspace-reference', resourceId: workspaceFor(attempt, 'developer').resourceId, path: workspaceFor(attempt, 'developer').resourceId === workspace.resourceId ? workspaceFor(attempt, 'developer').path : null, note: 'Unsealed source, not candidate or verification evidence. Cross-resource drafts require a sealed artifact before verification.' } : null, handoff: task.collaboration?.history.at(-1) ? { purpose: task.collaboration.history.at(-1).purpose, reason: task.collaboration.history.at(-1).reason } : null, candidateSha: attempt.candidateSha ?? null, tests: attempt.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, status: attempt.evidence.tests.status ?? null, stage: attempt.evidence.tests.stage ?? null, reason: attempt.evidence.tests.reason ?? null, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, review: attempt.evidence.review ? { sha: attempt.evidence.review.sha, verdict: attempt.evidence.review.verdict, summary: attempt.evidence.review.summary } : null }, repository: await repositoryContext(project.id, attempt.baseSha, project) }) } finally { if (preparation && !state.projectResourceLeases?.[lease.id]?.preparation?.exitUnknown) await rt.projectResources.endPreparation(preparation) } } catch (error) { const registered = lease && ownEntry(state.projectResourceLeases ?? {}, lease.id) diff --git a/patches/ds-bot/files/src/host/project-execution-environment.js b/patches/ds-bot/files/src/host/project-execution-environment.js index 3b1e94ec2..790192807 100644 --- a/patches/ds-bot/files/src/host/project-execution-environment.js +++ b/patches/ds-bot/files/src/host/project-execution-environment.js @@ -23,6 +23,7 @@ export function executionOutcome(error, output = '', { stage = 'tests', remote = if (remote && (timedOut || typeof error.code !== 'number' || [125, 255].includes(error.code))) return { status: 'execution-unknown', reason: 'resource-command-exit-unconfirmed', exitConfirmed: false } if (timedOut) return { status: 'execution-timeout', reason: 'command-deadline', exitConfirmed: true } if (stage === 'preparation') return { status: 'environment-blocked', reason: 'preparation-command-failed', exitConfirmed: true } + if (error.code === 127 || error.code === 'ENOENT' || /(?:^|[\s/])[^\s:]+:\s*(?:command )?not found\b/i.test(output)) return { status: 'environment-blocked', reason: 'test-runner-unavailable', stage: 'runner-startup', exitConfirmed: true } if (['ENOENT', 'EACCES'].includes(error.code) || /\b(?:ENOTCACHED|only-if-cached|ERR_MODULE_NOT_FOUND|MODULE_NOT_FOUND|EAI_AGAIN|ENOTFOUND|ENETUNREACH|ECONNREFUSED)\b|Cannot find (?:module|package)|command not found|network (?:is unreachable|access.*denied)|registry.*(?:denied|unavailable)/i.test(output)) return { status: 'environment-blocked', reason: 'dependency-tool-cache-or-network-unavailable', exitConfirmed: true } if (typeof error.code !== 'number') return { status: 'execution-unknown', reason: 'test-exit-unconfirmed', exitConfirmed: false } return { status: /AssertionError|ERR_ASSERTION|assertion failed/i.test(output) ? 'assertion-failed' : 'test-process-failed', reason: 'nonzero-test-exit', exitConfirmed: true } diff --git a/patches/ds-bot/files/src/host/project-service-views.js b/patches/ds-bot/files/src/host/project-service-views.js index 1f87883a1..1147faeca 100644 --- a/patches/ds-bot/files/src/host/project-service-views.js +++ b/patches/ds-bot/files/src/host/project-service-views.js @@ -63,7 +63,7 @@ export function attemptMetadataView(attempt) { if (attempt.evidence?.tests?.preparation) result.environment = pick(attempt.evidence.tests.preparation, ['id', 'status', 'executor', 'workspaceId', 'resourceId', 'identitySha256', 'sha256']) if (attempt.workspaces) result.workspaces = Object.fromEntries(Object.entries(attempt.workspaces).slice(0, 32).map(([role, row]) => [role, pick(row, ['id', 'status', 'role', 'resourceId', 'materialization'])])) if (attempt.agentDefinitions) result.agentDefinitions = { revision: attempt.agentDefinitions.revision, roles: (attempt.agentDefinitions.roles ?? []).slice(0, 32).map(row => ({ ...pick(row, ['id', 'role', 'policyRole', 'name', 'model']), skills: (row.skills ?? []).slice(0, 20).map(value => short(value, 300)), execution: pick(row.execution, ['harness', 'model', 'resourceIds']) })) } - function evidence(row) { return { ...pick(row, ['sha', 'sha256', 'passed', 'status', 'reason', 'executor', 'testsStarted', 'exitConfirmed', 'verdict', 'harness', 'sdkVersion', 'resourceId', 'sessionId', 'model', 'startedAt', 'completedAt']), ...(row.summary ? { summary: short(row.summary) } : {}) } } + function evidence(row) { return { ...pick(row, ['sha', 'sha256', 'passed', 'status', 'stage', 'reason', 'executor', 'testsStarted', 'exitConfirmed', 'verdict', 'harness', 'sdkVersion', 'resourceId', 'sessionId', 'model', 'startedAt', 'completedAt']), ...(row.summary ? { summary: short(row.summary) } : {}) } } result.evidence = {} for (const [key, row] of Object.entries(attempt.evidence ?? {}).slice(0, 32)) { if (key === 'execution') result.evidence.execution = Object.fromEntries(Object.entries(row).slice(0, 32).map(([role, value]) => [role, evidence(value)])) diff --git a/patches/ds-bot/files/test/project-delivery.test.js b/patches/ds-bot/files/test/project-delivery.test.js index 293071a58..aae6a0d45 100644 --- a/patches/ds-bot/files/test/project-delivery.test.js +++ b/patches/ds-bot/files/test/project-delivery.test.js @@ -991,6 +991,42 @@ test('independent verification preserves dependency diagnostics without declarin assert.match(readFileSync(tests.log, 'utf8'), /ENOTCACHED/) }) +test('npm test missing vitest after clean is runner-startup blockage and preparation reruns real assertions on the same candidate', async t => { + const f = await setup(t, undefined, { pool: true }) + f.rt.projectPools.connect({ validate: () => ({ ok: true }) }) + const settings = f.rt.config.projectRepositories['example/demo'] + settings.allowedPaths.push('package.json') + const wrapper = 'const cp=require("node:child_process");const r=cp.spawnSync("npm",["--offline","test"],{stdio:"inherit",env:{...process.env,HOME:process.argv[1],npm_config_cache:process.argv[1]+"/npm-cache"}});process.exit(r.status??1)' + settings.testCommand = [process.execPath, '-e', wrapper, f.directory] + await f.start('missing-vitest') + const attempt = f.attempt(), task = f.rt.state.tasks[f.task.id], roles = f.rt.projectAgents.deliveryDefinitions('p1').roles + await prepareRole(f.rt, task, { id: 'missing-runner-dev', attemptId: attempt.id, generation: 1 }, roles.find(row => row.role === 'developer')) + writeFileSync(join(attempt.workspaces.developer.path, 'code.mjs'), 'export const add=(a,b)=>a+b;\n') + writeFileSync(join(attempt.workspaces.developer.path, 'package.json'), JSON.stringify({ scripts: { test: 'vitest run' } })) + await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: attempt.id }, role: 'developer', outputs: { implementation: 'Implement addition' } }) + await prepareRole(f.rt, task, { id: 'missing-runner-test', attemptId: attempt.id, generation: 2 }, roles.find(row => row.role === 'tester')) + const sha = attempt.candidateSha, path = attempt.workspaces.tester.path + mkdirSync(join(path, 'node_modules/.bin'), { recursive: true }) + writeFileSync(join(path, 'node_modules/stale'), 'Must be cleaned') + const finish = () => f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: attempt.id }, role: 'tester', outputs: { tests: 'Request host verification' } }) + await assert.rejects(finish(), /Configured tests failed|environment is blocked/) + const failed = structuredClone(attempt.evidence.tests) + assert.equal(failed.status, 'environment-blocked'); assert.equal(failed.stage, 'runner-startup') + assert.equal(failed.testsStarted, true); assert.equal(failed.preparation, null); assert.equal(failed.passed, false) + assert.match(readFileSync(failed.log, 'utf8'), /vitest: (?:command )?not found/) + assert.equal((await import('node:fs')).existsSync(join(path, 'node_modules/stale')), false) + // A transport runner fixture delegates to the real Node assertions; this is + // not an installed Vitest or a paid-model acceptance claim. + const runner = '#!/usr/bin/env node\nconst r=require("node:child_process").spawnSync(process.execPath,["--test","--test-reporter=tap","code.test.mjs"],{stdio:"inherit"});process.exit(r.status??1)\n' + settings.workspacePrepareCommand = [process.execPath, '-e', `const fs=require('fs');fs.mkdirSync('node_modules/.bin',{recursive:true});fs.writeFileSync('node_modules/.bin/vitest',${JSON.stringify(runner)},{mode:0o755})`] + await finish() + assert.equal(attempt.candidateSha, sha); assert.equal(attempt.evidence.tests.sha, sha) + assert.equal(attempt.evidence.tests.status, 'passed'); assert.equal(attempt.evidence.tests.stage, 'tests') + assert.equal(attempt.evidence.tests.preparation.status, 'ready') + assert.deepEqual(attempt.testHistory[0], failed) + assert.match(readFileSync(attempt.evidence.tests.log, 'utf8'), /# pass 1/) +}) + test('independent test timeout remains incomplete verification with an immutable log', async t => { const f = await setup(t, [process.execPath, '-e', 'setTimeout(()=>{},10000)']) await f.start('test-timeout') diff --git a/patches/ds-bot/files/test/project-execution-environment.test.js b/patches/ds-bot/files/test/project-execution-environment.test.js index 6ca52fb75..d654fc0c1 100644 --- a/patches/ds-bot/files/test/project-execution-environment.test.js +++ b/patches/ds-bot/files/test/project-execution-environment.test.js @@ -18,6 +18,12 @@ test('environment outcomes retain non-success and separate dependency, assertion assert.equal(executionOutcome({ killed: true }, '', { remote: true }).exitConfirmed, false) assert.equal(executionOutcome({ code: 'ECONNRESET' }, '', { remote: true }).status, 'execution-unknown') assert.equal(executionOutcome({ code: 37 }, '', { stage: 'preparation' }).status, 'environment-blocked') + for (const output of ['sh: 1: vitest: not found', '/bin/sh: vitest: command not found']) { + assert.equal(executionOutcome({ code: 127 }, output).status, 'environment-blocked') + assert.equal(executionOutcome({ code: 127 }, output).stage, 'runner-startup') + } + assert.equal(executionOutcome({ code: 127 }, '').reason, 'test-runner-unavailable') + assert.equal(executionOutcome({ code: 1 }, 'AssertionError: expected true to be false').status, 'assertion-failed') }) test('dependency contract rejects escaping paths, arbitrary network modes and malformed commands', () => { for (const spec of [{ lockFiles: ['../package-lock.json'] }, { npmCachePath: '/owner/cache' }, { tools: [{ argv: 'npm --version' }] }, { network: 'full-access' }, { timeoutMs: 0 }, { arbitraryAuthority: true }]) assert.throws(() => dependencyEnvironmentCheck(spec), { code: 'bot/repository-config' }) diff --git a/scripts/api-contracts.mjs b/scripts/api-contracts.mjs index 46d96dea3..d06d06981 100644 --- a/scripts/api-contracts.mjs +++ b/scripts/api-contracts.mjs @@ -341,5 +341,5 @@ for (const operation of ['create', 'graph']) { // Execution environment outcomes are independent of model-role completion. schemas.EnvironmentPreparation = record({ version: lit('project-environment/v1'), id, projectId: id, attemptId: id, workspaceId: id, role, resourceId: nullable(id), principalId: nullable(id), candidateSha: nullable(sha), executor: choice('local-resource', 'docker-resource', 'independent-host-controller'), status: choice('not-declared', 'ready', 'environment-blocked', 'execution-timeout', 'execution-unknown'), reason: str(), exitConfirmed: bool, configurationSha256: hash, identitySha256: hash, locks: map(hash), tools: array(json, 16), cache: nullable(str()), sha256: hash, startedAt: int(), completedAt: int() }) -schemas.TestVerification = record({ sha, passed: bool, status: choice('passed', 'environment-blocked', 'assertion-failed', 'test-process-failed', 'execution-timeout', 'execution-unknown'), reason: str(), testsStarted: bool, exitConfirmed: bool, executor: str(), command: array(str()), preparation: nullable(ref('EnvironmentPreparation')), sha256: hash, workspaceId: id, resourceId: nullable(id), startedAt: int(), at: int() }) +schemas.TestVerification = record({ sha, passed: bool, status: choice('passed', 'environment-blocked', 'assertion-failed', 'test-process-failed', 'execution-timeout', 'execution-unknown'), reason: str(), stage: choice('preparation', 'runner-startup', 'tests'), testsStarted: bool, exitConfirmed: bool, executor: str(), command: array(str()), preparation: nullable(ref('EnvironmentPreparation')), sha256: hash, workspaceId: id, resourceId: nullable(id), startedAt: int(), at: int() }) schemas.Attempt.properties.evidence = record({ tests: ref('TestVerification'), execution: map(json), review: json, implementation: json })