diff --git a/LESSONS.md b/LESSONS.md index c920ef7c1..d00f618f9 100644 --- a/LESSONS.md +++ b/LESSONS.md @@ -13,6 +13,14 @@ Dated, short lessons from real FlyBot work. Agents read this **before coding**. ## Open lessons +### 2026-11-04 — Missing workerId does not define admission cleanup (Seen 1×) + +Binding can fail before execute returns; register reservation ownership and cleanup +across preparation/binding/spawn. Persist the spawn fence before external effects. +Only proven pre-spawn failure permits rollback; retain unknown occupancy and original +context, and serialize late startup against reclamation. + + ### 2026-11-04 — An enabled schedule is not a completed inspection (Seen 1×) A process-local waiter disappears on restart; correlate durable occurrence and Run diff --git a/PROJECT.md b/PROJECT.md index 2edf9469c..e5b5c15b5 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -8,6 +8,14 @@ architecture lives in [docs/architecture.md](docs/architecture.md) and Current stage and priorities: [ROADMAP.md](ROADMAP.md). How decisions are made: [docs/governance.md](docs/governance.md). Dated decisions: [decisions/](decisions/README.md). +## Follow-up reliability delivery — 2026-11-04 + +Owner authorizes sequential #2336/#2337/#2334/#2346/#2338/#2347/#2331/#2342 +implementation, one PR each, tests/memory/docs and validated merge to the delivery +branch. #2336 establishes owned reservation rollback and a durable pre-spawn fence; +unknown execution, workspace/context and current authority remain protected. +No live task/configuration/credential mutation or deployment is implied. + ## Execution recovery — 2026-11-04 #2325 aligns actual reused SDK/resource checkouts with pinned baseline/seed facts, diff --git a/ROADMAP.md b/ROADMAP.md index 4287e56e8..7886b506f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,5 +1,12 @@ # FlyBot — Roadmap +## Follow-up reliability — 2026-11-04 + +Implement #2336 reservation cleanup first; then #2337 CLI configuration, #2334 +environment attribution, #2346 context migration, #2338 receipts, #2347 recovery +routing, #2331 controlled merge and #2342 operational-agent collaboration. Separate +PRs retain existing UI/control-plane, authority and exact exit/evidence contracts. + ## Recovery delivery — 2026-11-04 Implement and merge #2325 → #2327 → #2328 → #2330 sequentially into @@ -52,41 +59,11 @@ Vendor-specific adapters and live external-provider acceptance remain separate. Backend delivery does not complete original frontend #41 acceptance or merge staged PRs. -S6/#55 integrates current service stages with original native frontend work and -adds repeatable isolated fault/replay acceptance. Actual PR feedback→original Task→ -Codex Dev→Tester has run; final candidate/Review/Chief evidence and compatible UI -rollout are recorded separately. Do not replace #41 or claim its whole visual/ -dual-mode matrix from this backend service acceptance. - -S5/#54 adds the trusted-private compatibility profile and execution-boundary query, -without per-Bot OS infrastructure or live identity/sandbox migration. Current-role, -safe/watch, private-data and concurrency regression suites guard existing rights. -Exact-head full checks and dedicated-account metadata probes are separate from -S6/#55 integrated GitHub/SDK/recovery/UI acceptance, which remains required. - -S4/#53 implements scoped GitHub feedback→existing Task inputs with persistent -revisions, bounded source sweeps and native schedule configuration. Full exact-head -verification, compatible checkpoint rollout and actual GitHub→Bot handoff evidence -must precede delivery claims. S5/#54 trust/capability boundary and S6/#55 integrated -fault/recovery acceptance follow. S1–S3/#60 are deployed; original frontend Task40/ -PR48 now advances its new sealed candidate through native Tester/Reviewer roles. - -Immediate #60 foreground repair completes approved extra-merge host sealing before -S2/S3 batch rollout and current frontend Attempt resumption. Keep the existing -index/MERGE_HEAD, shared-ref restriction, Task/PR and independent candidate gates. -The first batch is deployed; live source admission found a historical-Chief -scope regression. PR62 includes its real-state reproduction and focused guard -repair before resuming the original frontend Attempt. No full frontend acceptance yet. - -Service #49 implementation order: S1/#50 (PR56, deployed), S2/#51 (PR58, verified, -awaiting compatible batch rollout), S3/#52 (PR61,4374 checks and real Codex receipt/pool acceptance verified), then source subscriptions/#53, trusted execution boundary/#54 and -integrated fault/recovery acceptance/#55. #57/PR59 compatibility repair is deployed; -original Chief resumed actual frontend Dev without changing Task40/PR48 ownership. -Each stage keeps memory, architecture and regression evidence with its own PR. - -Immediate compatibility repair #57 removes the verified prior-invocation workspace -and Session resumption blocker before continuing service #51–#55. Preserve the -current frontend Task/Attempt/PR and unknown-exit protections. +Earlier service chronology are [preserved in the dated history](docs/history/2026-11-04-earlier-delivery-roadmap.md). +Current authorization, Task/Session/role constraints and original capability compatibility +remain in PROJECT.md and canonical subsystem contracts. Historical test evidence does +not complete outstanding real-provider, mixed-resource, release or frontend acceptance; +no new deployment, privilege expansion or automatic mission change is authorized. Current execution order (Owner update, 2026-11-01): keep the existing frontend #40 Task/PR48 autonomous while implementing backend #49–#55 directly in the @@ -98,53 +75,11 @@ live integration, frontend restoration and R8 acceptance retain their existing issues; no passing unit suite alone completes them. -The owner-approved plan is [fork #1](https://github.com/zy95-12/FlyBot/issues/1), recorded -in [the fork decision](decisions/2026-10-31-project-first-fork.md). Historical upstream -stages below do not restrict the fork's approved Project and frontend refactor. - -| Milestone | Deliverable | Status / issue | -| --- | --- | --- | -| M0 | Isolated DSH/tool/cwd/lifecycle and unchanged-platform HTTP checks | Merged PR #9; [#2](https://github.com/zy95-12/FlyBot/issues/2) | -| M1 | Project domain and direct Task/Run/Workspace ownership | PR #10 verified locally, awaiting review; [#3](https://github.com/zy95-12/FlyBot/issues/3) | -| M2 | Shared membership, platform identity adapter and backend authorization | PR #11 verified locally, awaiting review; [#4](https://github.com/zy95-12/FlyBot/issues/4) | -| M3 | Worktree, code/tests/PR/review evidence and human acceptance | PR #13 locally verified; integrated real-provider acceptance passed in M6; [#5](https://github.com/zy95-12/FlyBot/issues/5) | -| M4 | Public project Bots and private user/project agents | PR #14 verified locally, awaiting review; [#6](https://github.com/zy95-12/FlyBot/issues/6) | -| M5 | Rebuild task navigation, Project board, details and resource view | PR #15 locally verified, awaiting review; [#7](https://github.com/zy95-12/FlyBot/issues/7) | -| M6 | Repeatable two-user end-to-end acceptance | PR #16: real-model/GitHub/browser acceptance passed; awaiting review; [#8](https://github.com/zy95-12/FlyBot/issues/8) | - -The owner authorized sequential M1–M6 implementation, one PR per milestone. Each PR -updates long-term memory, architecture and tests. Merge status is tracked separately. - -Reuse platform services unchanged first; new project logic/adapters belong in FlyBot. -M0's deterministic model and existing shared-write denials do not imply M2/M3 completion. -GPU pools, multi-host scheduling, auto-merge and exact crash resume remain deferred. -The 30–43 person-day estimate is provisional pending interfaces/frontend design. - -## Authorized M7 sequence - -| Phase | Deliverable | State | -| --- | --- | --- | -| [M7.1 #21](https://github.com/zy95-12/FlyBot/issues/21) | Durable Chief/role memory and temporary Session snapshots | PR #26; full verify and actual DSH context checks passed | -| [M7.2 #22](https://github.com/zy95-12/FlyBot/issues/22) | Execution resource registry, candidate pools and leases | PR #27; full verify + actual local/Docker probe; integration #30 | -| [M7.3 #23](https://github.com/zy95-12/FlyBot/issues/23) | Native/Codex/Claude Agent SDK adapters | Draft PR #28; full verify passed; real authorization acceptance pending | -| [M7.4 #24](https://github.com/zy95-12/FlyBot/issues/24) | Owner configuration, probes and authorized resource terminal | Draft PR #29; full verify + actual PTY checks; browser integration #30 | -| [M7.5 #25](https://github.com/zy95-12/FlyBot/issues/25) | Cross-resource code/evidence transfer and real browser acceptance | Draft PR #30; code/resource/browser checks implemented, authenticated mixed-Harness acceptance pending | - -User explicitly authorized sequential implementation and a separate PR for every issue, -including long-term memory, architecture and tests. Merge/deployment is separate. - -## Automatic development service sequence - -[#49](https://github.com/zy95-12/FlyBot/issues/49) tracks S1 #50 service/TaskView → -S2 #51 durable Effects/outbox → S3 #52 input/role receipts → S4 #53 external -Task subscriptions. S5 #54 trusted deployment/authority follows S1 and applies -throughout; S6 #55 integrated recovery acceptance follows all five. One PR per -implementation issue; no extra tracking-only PR. S1 PR56 has a deployed initial facade (673c741b); long-history follow-up -b9cceb02 passed full4333 verification, deployed at a released checkpoint and -passed three-account scoped HTTP/pagination checks, preserving original UI bytes. -S2/#51 effect/outbox code and fault regressions are implemented; full verification -and source-bound rollout follow before S3 inputs/read receipts. -Independent repository review is not claimed. Original #34–#41 residuals remain. +Earlier M0–M7 and service checkpoints are [preserved in the dated history](docs/history/2026-11-04-earlier-delivery-roadmap.md). +Current authorization, Task/Session/role constraints and original capability compatibility +remain in PROJECT.md and canonical subsystem contracts. Historical test evidence does +not complete outstanding real-provider, mixed-resource, release or frontend acceptance; +no new deployment, privilege expansion or automatic mission change is authorized. ## Historical upstream roadmap @@ -250,48 +185,11 @@ in-process service and incomplete real watch/UI acceptance remain explicit; no standalone frontend, OpenAPI service or runtime migration is implied. -### Backend convergence #2184 — 2026-11-03 - -SDK Task-role binding hardening: pinned shared definition snapshots, latest observed -partial-thread recovery, verified legacy-key adoption, durable dispatch intent, -explicit incompatible recovery, current rights/resource checks and truthful adapter -capabilities. Native Task-role context restoration, parent waits, transactional Effects -and external command receipts remain dependent acceptance; no live-provider or deploy -claim from controlled process tests. README/API rewrite remains last (#2189). - - -### Collaboration contract convergence — 2026-11-03 (#2175) - -Expose durable Task handoffs through one scoped contract projection, with distinct -pending/accepted/processed/revoked/invalidated states, source/target context references, -actual claim/worker binding, initiative fences and predecessor/successor history. -Idempotent trusted offer/accept ports reuse pools; unresolved old writers cannot be -bypassed by Chief rerouting to another resource. Consultation is a separate private -protocol over retained transcripts, without Task creation or initiative transfer. -Chief prefers an active personal agent for user questions: safe answers require user -publication, watch answers use existing inbox rights; only committed Effects may wake -an authorized native watch. External clients have isolated mailboxes, not fabricated -provider sessions. SDK process/HTTP tests and real-provider acceptance remain distinct. -Native restoration/context bounds, transaction events and parent waits stay on their -original dependent issues; no control-plane, deployment or frontend changes. - - -### Context/memory convergence — 2026-11-03 (#2177) - -Unify model/Harness context budgeting and bounded source tools across persistent -Chief and temporary roles. Unknown capacity stays explicit; full rules/current -constraints remain required, retrieval yields before overflow, and native/SDK input -is checked before delivery. Working summaries use original public role-memory -records with exact source/evidence/uncertainty, separate resident Chief topics and -rollback of uncommitted staging. Facts have source/classification/version/archive; -role-authored experience cannot declare facts or overwrite human control. Private -notes retain history under bounded views. Codex tool history has an explicit cap; -SDK compaction remains runtime-managed, without claiming aggregate usage is current -window pressure. No new memory platform, executor or timer. Native Task-role physical -restore2184 and normalized SQL storage2182 remain separate acceptance; real SDK -verification must name exact runtime source hashes, not borrow deterministic tests. - -#2177 follow-up: validate two sequential native Chief jobs against fresh per-turn facts while retaining the same resident Session; dependent Task admission runtime acceptance remains #2179. +Earlier convergence implementation notes are [preserved in the dated history](docs/history/2026-11-04-earlier-delivery-roadmap.md). +Current authorization, Task/Session/role constraints and original capability compatibility +remain in PROJECT.md and canonical subsystem contracts. Historical test evidence does +not complete outstanding real-provider, mixed-resource, release or frontend acceptance; +no new deployment, privilege expansion or automatic mission change is authorized. ### Completed convergence checkpoints diff --git a/changelog.d/2336.fixed.md b/changelog.d/2336.fixed.md new file mode 100644 index 000000000..27e8b045a --- /dev/null +++ b/changelog.d/2336.fixed.md @@ -0,0 +1 @@ +Reclaim proven orphan reservations and roll back SDK admission failures before external spawn. Preserve unknown execution capacity, original Task-role context and workspaces with durable ownership/startup evidence. diff --git a/core/src/host/hooks.js b/core/src/host/hooks.js index e101d41e9..adca68bbf 100644 --- a/core/src/host/hooks.js +++ b/core/src/host/hooks.js @@ -1393,6 +1393,13 @@ export function install(rt) { ready({ task, definition }) { try { rt.projectResources.plan(task.projectId, { roles: [definition] }); return true } catch { return false } }, + async abort({ task, claim, reason }) { + const attempt = rt.state.projectAttempts?.[claim.attemptId] + for (const session of Object.values(attempt?.roleSessions ?? {})) if (session.poolClaimId === claim.id && session.botId) { + try { await lifecycle.release(session.botId) } catch { /* Do not assume unknown exit. */ } + } + return rt.projectDelivery.abortPoolAdmission(task, claim, reason) + }, async execute({ task, claim, definition }) { return lifecycle.spawn(await rt.projectDelivery.preparePoolRole(task, claim, definition)) }, async dispatch({ task, claim, workerId }) { const taskInbox = await rt.projectInputs?.prepare(task.id, claim) diff --git a/core/src/host/project-delivery.js b/core/src/host/project-delivery.js index b2f26f008..172cfa53c 100644 --- a/core/src/host/project-delivery.js +++ b/core/src/host/project-delivery.js @@ -900,6 +900,13 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { rt.taskDeliveryState?.(attempt.taskId, attempt.id, 'failed', attempt.failure.message) } } + async function abortPoolAdmission(task, claim, reason) { + const attempt = ownEntry(attempts(), claim.attemptId) + if (!attempt || attempt.taskId !== task.id || attempt.projectId !== task.projectId) return false + const rows = Object.values(state.projectResourceLeases ?? {}).filter(row => row.attemptId === attempt.id && row.reservationOwner?.claimId === claim.id && row.reservationOwner.generation === claim.generation) + for (const lease of rows) await rt.projectResources.abortReservation(lease.id, lease.generation, reason) + return rows.every(row => row.status === 'released') + } async function preparePoolRole(task, claim, definition) { const attempt = ownEntry(attempts(), claim.attemptId) if (!attempt || attempt.taskId !== task.id || attempt.protocol !== 'project-handoff/v1' || attempt.status !== 'running') throw reject('Pool Attempt is unavailable') @@ -915,10 +922,12 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { const policy = await repositoryRolePolicy(project.id, attempt.baseSha, role, policyRole, project) const sessionId = `role-session-${randomUUID()}` const execution = definition.execution - const lease = execution.resourceIds?.length ? await rt.projectResources.reserve({ projectId: project.id, attemptId: attempt.id, role, sessionKey: sessionId, ...execution }) : null + const lease = execution.resourceIds?.length ? await rt.projectResources.reserve({ projectId: project.id, attemptId: attempt.id, role, sessionKey: sessionId, reservationOwner: { taskId: task.id, claimId: claim.id, generation: claim.generation, sessionId }, ...execution }) : null if (lease) attempt.resourceLeases[role] = lease let context, workspace + const preparation = lease ? await rt.projectResources.beginPreparation(lease.id, lease.generation) : null try { + try { const freshWorkspace = !attempt.workspaces[role] await allocate(attempt, project, settings, [role], { [role]: policyRole }) workspace = attempt.workspaces[role] @@ -930,6 +939,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { } if (project.repository) { await prepareWorkspace(attempt, workspace, settings); await prepareIntegrations(attempt, workspace) } context = await buildProjectContext(rt, { retrieval: true, policy, projectId: project.id, taskId: task.id, role, revision: pinned.revision, resourceId: lease?.resourceId, definition, memory: definition.memory, knowledge: ownEntry(state.projectKnowledge ?? {}, project.id) ?? [], requirement: attempt.requirement, feedback: attempt.feedback, projectFacts: { workspaceMaterialization: workspace.materialization ?? null, ...(pinned.change ? { roleDefinitionChange: pinned.change } : {}), integrationInputs: structuredClone(attempt.integrationInputs ?? []), testFailure: await failureContext(attempt), publicationBinding: attempt.publicationBinding ?? null, integrationInstruction: attempt.publicationBinding ? 'Merge the pinned upstreamSha into the published candidate using --no-commit, resolve and stage conflicts before handoff. Host sealing retains both trusted inputs; never fabricate parents or discard backend changes.' : null, workingDraft: !attempt.candidateSha && workspaceFor(attempt, 'developer') ? { kind: 'unverified-workspace-reference', resourceId: workspaceFor(attempt, 'developer').resourceId, path: workspaceFor(attempt, 'developer').resourceId === workspace.resourceId ? workspaceFor(attempt, 'developer').path : null, note: 'Unsealed source, not candidate or verification evidence. Cross-resource drafts require a sealed artifact before verification.' } : null, handoff: task.collaboration?.history.at(-1) ? { purpose: task.collaboration.history.at(-1).purpose, reason: task.collaboration.history.at(-1).reason } : null, candidateSha: attempt.candidateSha ?? null, tests: attempt.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, review: attempt.evidence.review ? { sha: attempt.evidence.review.sha, verdict: attempt.evidence.review.verdict, summary: attempt.evidence.review.summary } : null }, repository: await repositoryContext(project.id, attempt.baseSha, project) }) + } finally { if (preparation) await rt.projectResources.endPreparation(preparation) } } catch (error) { const registered = lease && ownEntry(state.projectResourceLeases ?? {}, lease.id) const unbound = registered && registered.generation === lease.generation && !registered.worker && !registered.processScope && !Object.values(state.projectHarnessWorkers ?? {}).some(row => row.leaseId === lease.id) && !Object.values(attempt.roleSessions ?? {}).some(row => row.id === sessionId) @@ -1112,7 +1122,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { } } const ownsWorkspace = (projectId, path) => Object.values(state.projectWorkspaces ?? {}).find(workspace => workspace.projectId === projectId && workspace.path === path && DELIVERY_ROLES.includes(workspace.policyRole ?? workspace.role) && ['worktree', 'resource-clone', 'decision-directory'].includes(workspace.kind) && workspace.status === 'allocated' && ongoing.has(ownEntry(attempts(), workspace.attemptId)?.status)) - return { releases, recoveryFacts, admitChiefInputs, reconcileExecutions: () => reconcileExecutionAuthority(rt), exitConfirmed: workerId => executionExitConfirmed(rt, workerId), nativeSessions, endpoint, context, admitChildInputs: (taskId, claim) => serialAttempt(claim.attemptId, () => childInputs.admit(taskId, claim)), preparePoolRole, completePoolRole, chiefAccept, reworkPool, recordExecution, repositoryContext, repositoryRolePolicy, bindSession, releaseSession, sessionFor, recover, runReported, ownsWorkspace, gate: input => serialAttempt(current(input.taskId)?.id ?? input.taskId, () => gate(input)) } + return { releases, recoveryFacts, admitChiefInputs, reconcileExecutions: () => reconcileExecutionAuthority(rt), exitConfirmed: workerId => executionExitConfirmed(rt, workerId), nativeSessions, endpoint, context, admitChildInputs: (taskId, claim) => serialAttempt(claim.attemptId, () => childInputs.admit(taskId, claim)), preparePoolRole, abortPoolAdmission, completePoolRole, chiefAccept, reworkPool, recordExecution, repositoryContext, repositoryRolePolicy, bindSession, releaseSession, sessionFor, recover, runReported, ownsWorkspace, gate: input => serialAttempt(current(input.taskId)?.id ?? input.taskId, () => gate(input)) } } export function install(rt) { diff --git a/core/src/host/project-harness.js b/core/src/host/project-harness.js index 631f7b549..f3302f749 100644 --- a/core/src/host/project-harness.js +++ b/core/src/host/project-harness.js @@ -72,6 +72,27 @@ export function createProjectHarness(rt) { return JSON.parse(text) } async function spawn(spec) { + const workspace = spec.preparedChief?.workspace ?? rt.projectDelivery.ownsWorkspace(spec.executionProjectId, spec.executionWorkDir) + const session = spec.preparedChief?.session ?? (workspace && rt.projectDelivery.sessionFor(workspace)) + let preparation + try { + if (session?.resourceLease) preparation = await rt.projectResources.beginPreparation(session.resourceLease.id, session.resourceLease.generation) + const id = await spawnPrepared(spec) + if (preparation) { await rt.projectResources.endPreparation(preparation); preparation = null } + return id + } + catch (error) { + if (preparation) { const token = preparation; preparation = null; try { await rt.projectResources.endPreparation(token) } catch { /* Keep unconfirmed preparation capacity; preserve the admission failure. */ } } + const lease = session?.resourceLease + if (lease) { + const rows = Object.values(workers()).filter(row => row.leaseId === lease.id && row.generation === lease.generation) + for (const row of rows) if (active.has(row.id)) { try { await release(row.id) } catch { /* Unknown external exit retains capacity. */ } } + try { await rt.projectResources.abortReservation(lease.id, lease.generation, `SDK admission failed: ${error.code ?? 'bot/harness'}`) } catch { /* Preserve the original failure and any unresolved reservation. */ } + } + throw error + } + } + async function spawnPrepared(spec) { const chief = spec.preparedChief if (chief && !rt.projectAgents.isDelegationActive(chief.agentId, chief.delegationId)) throw refuse('Chief delegation is no longer active') if (!spec.executionWorkDir && !chief) return null @@ -103,7 +124,7 @@ export function createProjectHarness(rt) { const row = { id, ...(conversation?.roomId ? { roomId: conversation.roomId } : {}), model: session.execution.model ?? selected.selected.model ?? null, chiefRunId: chief?.runId ?? null, chiefInvocation: chief?.runId ? rt.state.projectExecutions?.[chief.runId]?.invocation ?? null : null, visibility: conversation?.visibility ?? 'project', ...(conversation?.visibility === 'private' ? { memberId: conversation.memberId } : {}), projectId: session.projectId, taskId: session.taskId, attemptId: session.attemptId, role: session.role, policyRole: session.policyRole ?? session.definition?.policyRole ?? session.role, sessionKey: session.id, resourceId: session.resourceLease.resourceId, leaseId: session.resourceLease.id, generation: session.resourceLease.generation, harness: session.execution.harness, sdkVersion: null, expectedSdkVersion: VERSIONS[session.execution.harness], principalId: selected.principalId, - cwd: workspace.path, scratchDirectory, status: 'prepared', executionStartedAt: null, createdAt: Date.now(), eventCount: 0 } + cwd: workspace.path, scratchDirectory, spawnPhase: 'registered', status: 'prepared', executionStartedAt: null, createdAt: Date.now(), eventCount: 0 } if (logical) Object.assign(row, { logicalSessionId: logical.id, bindingKey, bindingVersion: 'task-role-binding/v2', definitionKey, ...(previous ? { backendSessionId: previous.sessionId, continuedFrom: previous.workerId, ...(previous.key !== bindingKey ? { migratedBindingKey: previous.key } : {}) } : { rotationReason: 'First Task-role execution' }) }) else if (chief) Object.assign(row, { logicalSessionId: chief.agentId, bindingKey, ...(chief.resumeSessionId ? { backendSessionId: chief.resumeSessionId, continuedFrom: chief.previousWorkerId } : {}) }) setEntry(workers(), id, row) @@ -121,8 +142,10 @@ export function createProjectHarness(rt) { row.contextArchive = { sha256: session.context.archiveSha256, file: source } } row.eventLog = join(directory, `${evidenceName}-sdk.jsonl`) + await rt.projectResources.fenceSpawn(row.leaseId, row.generation, id) const process = rt.projectResources.startProcess(row.projectId, row.resourceId, { argv: ['node', workerPath], cwd: workspace.path }) const control = { process, chief, contextBudget, resumeSessionId: chief?.resumeSessionId ?? previous?.sessionId, continuedPolicyHash: previous?.policySha256, archivePath, logical, bindingKey, env: selected.env, profile: { ...selected.selected, ...(session.execution.model ? { model: session.execution.model } : {}) }, session, prompt: spec.prompt, outputs: [], closed: false, ready: false } + row.spawnPhase = 'spawned' active.set(id, control) let readyResolve, readyReject control.handshake = new Promise((resolve, reject) => { readyResolve = resolve; readyReject = reject }) diff --git a/core/src/host/project-pools.js b/core/src/host/project-pools.js index 2d20659b9..4e7b56012 100644 --- a/core/src/host/project-pools.js +++ b/core/src/host/project-pools.js @@ -415,6 +415,7 @@ export function createProjectPools(rt) { if (!failedClaim) continue failedClaim.exitConfirmed = false if (failedClaim.workerId) { try { failedClaim.exitConfirmed = await adapter.release(failedClaim.workerId) } catch { failedClaim.exitConfirmed = false } } + else if (adapter.abort) { try { failedClaim.exitConfirmed = await adapter.abort({ task, claim: failedClaim, reason }) } catch { /* Retain unconfirmed preparation occupancy. */ } } await serial(async () => { const c = task.collaboration if (c.claim?.id !== token.id) return diff --git a/core/src/host/project-resources.js b/core/src/host/project-resources.js index 3541227b8..52255a12a 100644 --- a/core/src/host/project-resources.js +++ b/core/src/host/project-resources.js @@ -18,6 +18,8 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} const { state, config } = rt const leases = () => (state.projectResourceLeases ??= {}) const resources = () => config.projectResources ?? {} + const preparing = new Map() + let effectStore let recovered = false let tail = Promise.resolve() const serial = fn => { const result = tail.then(fn, fn); tail = result.then(() => {}, () => {}); return result } @@ -80,7 +82,110 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} } return pending } + const workerRecords = lease => [...Object.values(state.projectHarnessWorkers ?? {}), ...Object.values(state.projectNativeWorkers ?? {})].filter(row => row.leaseId === lease.id) + function unbound(lease) { + return !preparing.has(lease.id) && lease.preparation?.status !== 'active' && !lease.worker && !lease.processScope && !workerRecords(lease).length && !Object.values(state.projectAttempts?.[lease.attemptId]?.roleSessions ?? {}).some(row => row.resourceLease?.id === lease.id && row.botId) && !Object.values(state.bots ?? {}).some(row => row.resourceLease?.id === lease.id || row.ephemeral && row.attemptId === lease.attemptId && row.role === lease.role) + } + function retiredReservation(lease) { + const attempt = state.projectAttempts?.[lease.attemptId], owner = lease.reservationOwner + if (!attempt || attempt.projectId !== lease.projectId) return false + if (['returned', 'cancelled', 'accepted', 'failed'].includes(attempt.status) || attempt.supersededBy) return true + if (!owner) return false + const task = state.tasks?.[owner.taskId], claim = task?.collaboration?.claim + return !task || task.projectId !== lease.projectId || task.deliveryAttemptId !== lease.attemptId || !claim || claim.id !== owner.claimId || claim.generation !== owner.generation || task.collaboration.generation !== owner.generation || claim.status !== 'active' + } + async function beginPreparation(id, generation) { + const lease = requireLease(id, generation) + if (lease.status === 'released' || retiredReservation(lease) || preparing.has(id) || lease.preparation?.status === 'active') throw failure('Resource preparation is released, stale or already active') + const token = { leaseId: id, generation, id: randomUUID() }, previous = lease.preparation + preparing.set(id, token.id) + try { + lease.preparation = { ...token, status: 'active', processScope: await captureProcessScope(), at: Date.now() } + if (rt.projectEffects) { + effectHandlers() + const intent = rt.projectEffects.stage({ kind: 'resource.preparation', key: `${id}:${token.id}`, projectId: lease.projectId, payload: token }) + try { await rt.projectEffects.persist(intent) } catch (error) { rt.projectEffects.discardUncommitted(intent); throw error } + } else await rt.save() + if (lease.status === 'released' || retiredReservation(lease)) throw failure('Resource preparation owner changed before commands') + return token + } catch (error) { + preparing.delete(id) + if (previous === undefined) delete lease.preparation; else lease.preparation = previous + throw error + } + } + async function endPreparation(token) { + const lease = requireLease(token.leaseId, token.generation) + if (preparing.get(lease.id) !== token.id || lease.preparation?.id !== token.id) throw failure('Preparation completion is stale') + lease.preparation.status = 'settled'; lease.preparation.finishedAt = Date.now() + try { await rt.save() } catch (error) { lease.preparation.status = 'active'; throw error } + preparing.delete(lease.id) + } + async function abortReservation(id, generation, reason) { + const lease = requireLease(id, generation) + if (lease.status === 'released') return true + if (preparing.has(id) || lease.preparation?.status === 'active') return false + const records = workerRecords(lease) + let changedSession, beforeSession + const before = structuredClone(lease), worker = records.length === 1 && records[0], beforeWorker = worker && structuredClone(worker) + if (!unbound(lease)) { + const row = records.length === 1 && records[0] + // Only the new bind-before-spawn protocol proves that registration never crossed spawn. + if (!row || row.spawnPhase !== 'registered' || row.status !== 'prepared' || row.executionStartedAt != null || lease.worker?.kind !== 'sdk' || lease.worker.handle !== row.id || row.generation !== generation) return false + const proof = { kind: 'sdk-worker-not-started', workerId: row.id, generation, at: Date.now() } + row.spawnPhase = 'aborted'; row.status = 'released'; row.releasedAt = proof.at; row.releaseProof = proof + lease.status = 'released'; lease.releasedAt = proof.at; lease.releaseProof = proof + const attempt = state.projectAttempts?.[row.attemptId], session = attempt?.roleSessions?.[row.role] + if (session?.botId === row.id) { changedSession = session; beforeSession = structuredClone(session); session.status = 'released'; session.releasedAt = proof.at } + } else { + lease.status = 'released'; lease.releasedAt = Date.now(); lease.releaseProof = { kind: 'unbound-reservation-release', at: lease.releasedAt } + } + lease.reclamation = { reason: String(reason).slice(0, 400), at: Date.now(), owner: structuredClone(lease.reservationOwner ?? null) } + let intent + try { + if (rt.projectEffects) { + effectHandlers() + intent = rt.projectEffects.stage({ kind: 'resource.reclaim', key: `${id}:${generation}`, projectId: lease.projectId, payload: { leaseId: id, generation } }) + lease.reclamation.effect = intent; await rt.projectEffects.persist(intent) + } else await rt.save() + } catch (error) { + if (!intent || rt.projectEffects.discardUncommitted(intent)) { + for (const key of Object.keys(lease)) delete lease[key] + Object.assign(lease, before) + if (worker) { for (const key of Object.keys(worker)) delete worker[key]; Object.assign(worker, beforeWorker) } + if (changedSession) { for (const key of Object.keys(changedSession)) delete changedSession[key]; Object.assign(changedSession, beforeSession) } + } + throw error + } + rt.projectScheduler?.wake() + return true + } + async function fenceSpawn(id, generation, workerId) { + const lease = requireLease(id, generation), row = state.projectHarnessWorkers?.[workerId] + if (lease.status !== 'running' || !row || row.spawnPhase !== 'registered' || lease.worker?.handle !== workerId || row.leaseId !== id || row.generation !== generation || retiredReservation(lease)) throw failure('Resource startup reservation is released, stale or retired') + row.spawnPhase = 'starting' + try { + if (rt.projectEffects) { + effectHandlers() + const token = rt.projectEffects.stage({ kind: 'resource.spawn', key: `${id}:${generation}:${workerId}`, projectId: lease.projectId, payload: { leaseId: id, generation, workerId } }) + row.spawnEffect = token + try { await rt.projectEffects.persist(token) } catch (error) { if (rt.projectEffects.discardUncommitted(token)) row.spawnPhase = 'registered'; throw error } + } else await rt.save() + } catch (error) { if (row.spawnPhase === 'starting') row.spawnPhase = 'registered'; throw error } // The caller has not crossed external spawn when the fence itself throws. + requireBinding(lease.projectId, lease) + if (lease.status !== 'running' || row.spawnPhase !== 'starting' || retiredReservation(lease)) throw failure('Resource startup changed before external spawn') + } + function effectHandlers() { + if (!rt.projectEffects || rt.projectEffects === effectStore) return + rt.projectEffects?.register('resource.spawn', { run: row => ({ workerId: row.payload.workerId, status: 'spawn-fenced' }) }, { replace: true }) + rt.projectEffects.register('resource.preparation', { run: row => ({ leaseId: row.payload.leaseId, status: 'preparation-fenced' }) }, { replace: true }) + rt.projectEffects.register('resource.reclaim', { run: row => ({ leaseId: row.payload.leaseId, status: 'reclaimed-before-spawn' }) }, { replace: true }) + effectStore = rt.projectEffects + } + effectHandlers() async function reserve(request) { + const owner = request.reservationOwner + if (owner && (typeof owner.taskId !== 'string' || typeof owner.claimId !== 'string' || !Number.isSafeInteger(owner.generation) || owner.generation < 1 || owner.sessionId !== undefined && owner.sessionId !== request.sessionKey)) throw failure('Invalid reservation ownership') const existing = Object.values(leases()).find(row => row.sessionKey === request.sessionKey) if (existing) { if (existing.projectId !== request.projectId || existing.attemptId !== request.attemptId || existing.role !== request.role || existing.harness !== request.harness || existing.status === 'released') throw failure('Execution resource Session key is already bound or released') @@ -89,19 +194,28 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} const selected = select(request) const lease = { id: `resource-lease-${randomUUID()}`, generation: 1, projectId: request.projectId, attemptId: request.attemptId, role: request.role, sessionKey: request.sessionKey, purpose: request.purpose ?? 'execution', resourceId: selected.id, harness: request.harness, principalId: rt.readProject(request.projectId).ownerUserId, runtimeUser: selected.item.type === 'docker' ? selected.item.user : `uid:${process.getuid?.() ?? 'unknown'}`, - bindingHash: bindingHash(selected.item), status: 'reserved', createdAt: Date.now() } + ...(request.reservationOwner ? { reservationOwner: structuredClone(request.reservationOwner) } : {}), bindingHash: bindingHash(selected.item), status: 'reserved', createdAt: Date.now() } setEntry(leases(), lease.id, lease) - await rt.save() + try { await rt.save() } catch (error) { + // No caller received this reservation, so no external startup was authorized. + lease.status = 'released'; lease.releasedAt = Date.now(); lease.releaseProof = { kind: 'unbound-reservation-release', at: lease.releasedAt }; lease.reclamation = { reason: 'Initial reservation save failed before admission returned', at: lease.releasedAt } + throw error + } return structuredClone(lease) } async function reserveRoles(projectId, attemptId, definitions) { plan(projectId, definitions) const reserved = {} - for (const role of definitions?.roles ?? []) { - if (role.role === 'chief' || !role.execution?.resourceIds?.length) continue - reserved[role.role] = await reserve({ projectId, attemptId, role: role.role, sessionKey: `${attemptId}:${role.role}`, ...role.execution }) + try { + for (const role of definitions?.roles ?? []) { + if (role.role === 'chief' || !role.execution?.resourceIds?.length) continue + reserved[role.role] = await reserve({ projectId, attemptId, role: role.role, sessionKey: `${attemptId}:${role.role}`, ...role.execution }) + } + return reserved + } catch (error) { + for (const lease of Object.values(reserved)) { try { await abortReservation(lease.id, lease.generation, 'Batch admission failed before returning reservations') } catch { /* Reconciliation retains any uncommitted cleanup. */ } } + throw error } - return reserved } function requireLease(id, generation) { const lease = ownEntry(leases(), id) @@ -111,8 +225,10 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} async function markStarted(id, generation, worker) { const lease = requireLease(id, generation) if (lease.status === 'released') throw failure('Execution resource lease was released') + if (retiredReservation(lease)) throw failure('Execution reservation owner is no longer current') if (lease.worker && JSON.stringify(lease.worker) !== JSON.stringify(worker)) throw failure('Execution resource worker is already bound') const scope = setting(lease.projectId, lease.resourceId).type === 'local' ? await captureProcessScope() : null + if (lease.status === 'released' || retiredReservation(lease)) throw failure('Execution reservation changed before binding') lease.worker = structuredClone(worker); lease.status = 'running' if (scope) lease.processScope = scope await rt.save() @@ -126,7 +242,7 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} const supervisor = worker?.kind === 'native' && supervisorProofMatches(scope, confirmation?.proof) && await priorExit(scope) if (requireLease(id, generation) !== lease || lease.worker !== worker || lease.processScope !== scope) throw failure('Execution resource binding changed during exit inspection') if (lease.status === 'released') return - const confirmed = supervisor || !worker || worker.kind === 'native' && !ownEntry(state.bots ?? {}, worker.botId) && native?.leaseId === lease.id && nativeExitConfirmed(state, native) || worker.kind !== 'native' && typeof worker.handle === 'string' && worker.handle.length > 0 && confirmation?.handle === worker.handle && confirmation?.exited === true + const confirmed = supervisor || !worker && unbound(lease) || worker.kind === 'native' && !ownEntry(state.bots ?? {}, worker.botId) && native?.leaseId === lease.id && nativeExitConfirmed(state, native) || worker.kind !== 'native' && typeof worker.handle === 'string' && worker.handle.length > 0 && confirmation?.handle === worker.handle && confirmation?.exited === true lease.status = confirmed ? 'released' : 'stopping' if (confirmed) { lease.releasedAt = Date.now(); lease.releaseProof ??= confirmation?.proof ?? native?.releaseProof ?? { kind: worker ? 'host-confirmed-worker-exit' : 'unbound-reservation-release', at: lease.releasedAt } } await rt.save() @@ -166,7 +282,16 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} if (lease.processScope?.hostPid === process.pid) { if (lease.worker?.kind === 'native' && rt.projectDelivery?.nativeSessions?.currentHost(lease.worker.botId) || lease.worker?.kind === 'sdk' && rt.projectHarness?.hasController(lease.worker.handle) || lease.worker?.kind === 'native-turn' && rt.projectAgents?.hasExchange(lease.worker.handle)) continue } - if (!lease.worker) { if (lease.status === 'unreachable') await release(lease.id, lease.generation); continue } + if (lease.preparation?.status === 'active') { + if (preparing.has(lease.id)) continue + if (setting(lease.projectId, lease.resourceId).type !== 'local' || !await priorExit(lease.preparation.processScope)) continue + lease.preparation.status = 'settled'; lease.preparation.recoveryProof = { kind: 'systemd-prior-invocation-exited', previousInvocationId: lease.preparation.processScope.invocationId, at: Date.now() }; await rt.save() + } + if (workerRecords(lease).some(row => row.spawnPhase === 'registered') && retiredReservation(lease)) { await serial(() => abortReservation(lease.id, lease.generation, 'Registered worker lost its startup owner before spawn')); continue } + if (!lease.worker) { + if (unbound(lease) && (lease.status === 'unreachable' || retiredReservation(lease))) await serial(() => unbound(lease) && active(lease) && (lease.status === 'unreachable' || retiredReservation(lease)) ? abortReservation(lease.id, lease.generation, 'No current reservation owner remains before binding') : undefined) + continue + } if (lease.worker.kind === 'native' && !ownEntry(state.bots ?? {}, lease.worker.botId)) await release(lease.id, lease.generation) else if (lease.worker.kind !== 'native' && lease.status !== 'unreachable') { lease.status = 'unreachable'; changed = true } } @@ -221,7 +346,7 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} return structuredClone(row) } return { setting, requireBinding, uploadFile, downloadFile, list, checkPool, plan, command, startProcess, probe, occupied, reconcile, recover, - reserveRoles: (...args) => serial(() => reserveRoles(...args)), + beginPreparation: (...args) => serial(() => beginPreparation(...args)), endPreparation: (...args) => serial(() => endPreparation(...args)), abortReservation: (...args) => serial(() => abortReservation(...args)), fenceSpawn: (...args) => serial(() => fenceSpawn(...args)), reserveRoles: (...args) => serial(() => reserveRoles(...args)), reserve: request => serial(() => reserve(request)), markStarted: (...args) => serial(() => markStarted(...args)), release: (...args) => serial(() => release(...args)) } } export function install(rt) { rt.projectResources = createProjectResources(rt) } diff --git a/core/src/host/project-service-views.js b/core/src/host/project-service-views.js index 2ce5d1b27..50350dcc9 100644 --- a/core/src/host/project-service-views.js +++ b/core/src/host/project-service-views.js @@ -83,7 +83,7 @@ export function executionMetadataView(state, row) { const lease = row.leaseId && state.projectResourceLeases?.[row.leaseId] const exit = native ? nativeExitConfirmed(state, row) : confirmedWorkerExit(state, row) return { ...pick(row, ['id', 'role', 'resourceId', 'generation', 'sessionKey', 'logicalSessionId', 'createdAt', 'executionStartedAt', 'releasedAt']), - harness: native ? 'flybot' : row.harness ?? null, status: row.status, + harness: native ? 'flybot' : row.harness ?? null, status: row.status, startupPhase: native ? null : row.spawnPhase ?? null, policyRole: session?.policyRole ?? row.policyRole ?? row.role ?? null, definitionRevision: session?.definitionRevision ?? null, activationEffectId: native ? row.activationEffect?.id ?? null : null, diff --git a/core/src/host/project-task-sessions.js b/core/src/host/project-task-sessions.js index 5de63cc19..4d5211f20 100644 --- a/core/src/host/project-task-sessions.js +++ b/core/src/host/project-task-sessions.js @@ -115,6 +115,10 @@ export function confirmedWorkerExit(state, worker) { lease.worker?.kind === 'sdk' && lease.worker.handle === worker.id && lease.releaseProof?.kind === proof.kind && lease.releaseProof.workerId === worker.id && lease.releaseProof.generation === worker.generation && lease.releaseProof.at === proof.at) } const proof = worker.releaseProof + if (proof?.kind === 'sdk-worker-not-started') { + const lease = ownEntry(state.projectResourceLeases ?? {}, worker.leaseId) + return worker.spawnPhase === 'aborted' && worker.executionStartedAt == null && proof.workerId === worker.id && proof.generation === worker.generation && lease?.status === 'released' && lease.generation === worker.generation && lease.worker?.handle === worker.id && lease.projectId === worker.projectId && lease.attemptId === worker.attemptId && lease.resourceId === worker.resourceId && lease.role === worker.role && lease.releaseProof?.kind === proof.kind && lease.releaseProof.workerId === worker.id && lease.releaseProof.at === proof.at + } if (proof?.kind === 'sdk-stop-ack-and-process-exit') return proof.generation === worker.generation if (proof?.kind !== 'systemd-prior-invocation-exited' || !/^[a-f0-9]{32}$/.test(proof.previousInvocationId ?? '')) return false const lease = ownEntry(state.projectResourceLeases ?? {}, worker.leaseId) diff --git a/core/test/project-delivery.test.js b/core/test/project-delivery.test.js index 2ecae88b5..130d9fed6 100644 --- a/core/test/project-delivery.test.js +++ b/core/test/project-delivery.test.js @@ -62,6 +62,16 @@ async function setup(t, testCommand = [process.execPath, '--test', 'code.test.mj return { rt, home, task, start, attempt, gate, repository, published, publisher, workflow, directory } } +// Direct preparation probes emulate the pool's current claim before using a resource. +function prepareRole(rt, task, claim, definition) { + if (definition.execution?.resourceIds?.length && task.collaboration?.claim?.id !== claim.id) { + const c = task.collaboration + c.generation = claim.generation; c.attemptId = claim.attemptId; c.botId = definition.id; c.status = 'claimed' + c.claim = { ...claim, taskId: task.id, projectId: task.projectId, botId: definition.id, role: definition.role, policyRole: definition.policyRole ?? definition.role, status: 'active', createdAt: Date.now() } + } + return rt.projectDelivery.preparePoolRole(task, claim, definition) +} + async function implement(fixture) { writeFileSync(join(fixture.attempt().workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await fixture.gate('developer') @@ -621,10 +631,10 @@ test('a transferred unsealed implementation is a labelled draft reference, not a const attempt = f.attempt(), task = f.rt.state.tasks[f.task.id] const defs = f.rt.projectAgents.deliveryDefinitions('p1').roles const dev = defs.find(row => row.role === 'developer'), tester = defs.find(row => row.role === 'tester') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'draft-dev', attemptId: attempt.id, generation: 1 }, dev) + await prepareRole(f.rt, task, { id: 'draft-dev', attemptId: attempt.id, generation: 1 }, dev) const path = attempt.workspaces.developer.path writeFileSync(join(path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'draft-test', attemptId: attempt.id, generation: 2 }, tester) + await prepareRole(f.rt, task, { id: 'draft-test', attemptId: attempt.id, generation: 2 }, tester) const facts = JSON.parse(attempt.roleSessions.tester.context.text.split('\n')[1].split('\n')[0]).projectFacts assert.equal(facts.candidateSha, null) assert.equal(facts.workingDraft.kind, 'unverified-workspace-reference') @@ -638,10 +648,10 @@ test('restart preserves a tested pool checkpoint waiting for Chief only when eve const f = await setup(t, undefined, { pool: true }); f.rt.projectPools.connect({ validate: () => ({ ok: true }) }); await f.start('checkpoint-' + variant) const a = f.attempt(), task = f.rt.state.tasks[f.task.id] const roles = f.rt.projectAgents.deliveryDefinitions('p1').roles - await f.rt.projectDelivery.preparePoolRole(task, { id: 'dev', attemptId: a.id, generation: 1 }, roles.find(r => r.role === 'developer')) + await prepareRole(f.rt, task, { id: 'dev', attemptId: a.id, generation: 1 }, roles.find(r => r.role === 'developer')) writeFileSync(join(a.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: a.id }, role: 'developer', outputs: {} }) - await f.rt.projectDelivery.preparePoolRole(task, { id: 'test', attemptId: a.id, generation: 2 }, roles.find(r => r.role === 'tester')) + await prepareRole(f.rt, task, { id: 'test', attemptId: a.id, generation: 2 }, roles.find(r => r.role === 'tester')) await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: a.id }, role: 'tester', outputs: {} }) task.collaboration.status = 'blocked' task.collaboration.claim = { botId: 'p1:chief', status: variant === 'active-claim' ? 'active' : 'waiting-human', workerId: 'chief-worker', chiefRunId: 'chief-run' } @@ -668,7 +678,7 @@ test('published pool rework retains its PR binding and seals a descendant of the await f.start('published-rework') const task = f.rt.state.tasks[f.task.id], first = f.attempt() const developer = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'first-dev', attemptId: first.id, generation: 1 }, developer) + await prepareRole(f.rt, task, { id: 'first-dev', attemptId: first.id, generation: 1 }, developer) writeFileSync(join(first.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: first.id }, role: 'developer', outputs: {} }) const original = first.candidateSha @@ -682,7 +692,7 @@ test('published pool rework retains its PR binding and seals a descendant of the const id = await f.rt.projectDelivery.reworkPool({ task, claim: { id: 'chief-return', attemptId: first.id, botId: 'p1:chief' }, reason: 'Maintain the existing PR after its base advanced' }) const next = f.rt.state.projectAttempts[id] assert.deepEqual(next.publicationBinding, { number: 48, repository: 'example/demo', branch: first.branch, previousSha: original, upstreamSha: first.baseSha, upstreamRef: 'main' }) - await f.rt.projectDelivery.preparePoolRole(task, { id: 'next-dev', attemptId: id, generation: 2 }, developer) + await prepareRole(f.rt, task, { id: 'next-dev', attemptId: id, generation: 2 }, developer) writeFileSync(join(next.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => Number(a)+Number(b); // intermediate\n') git(next.workspaces.developer.path, ['-c', 'user.name=Fixture', '-c', 'user.email=fixture@localhost', 'commit', '-am', 'Private intermediate work']) writeFileSync(join(next.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => Number(a)+Number(b);\n') @@ -716,7 +726,7 @@ test('published rework pins the actual target and seals both published and resol f.rt.projectPools.connect({ validate: () => ({ ok: true }) }); await f.start('upstream-rework') const task = f.rt.state.tasks[f.task.id], first = f.attempt() const developer = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'dev-before', attemptId: first.id, generation: 1 }, developer) + await prepareRole(f.rt, task, { id: 'dev-before', attemptId: first.id, generation: 1 }, developer) writeFileSync(join(first.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: first.id }, role: 'developer', outputs: {} }) const seed = first.candidateSha @@ -737,7 +747,7 @@ test('published rework pins the actual target and seals both published and resol const id = await f.rt.projectDelivery.reworkPool({ task, claim: { id: 'upstream-return', attemptId: first.id, botId: 'p1:chief' }, reason: 'Integrate current backend without losing published history' }) const next = f.rt.state.projectAttempts[id] assert.equal(next.publicationBinding.upstreamSha, upstream) - await f.rt.projectDelivery.preparePoolRole(task, { id: 'dev-after', attemptId: id, generation: 2 }, developer) + await prepareRole(f.rt, task, { id: 'dev-after', attemptId: id, generation: 2 }, developer) const path = next.workspaces.developer.path await assert.rejects(f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: id }, role: 'developer', outputs: {} }), /merge the pinned upstream/) assert.equal(git(path, ['rev-parse', 'HEAD']), seed) @@ -755,7 +765,7 @@ test('published rework pins the actual target and seals both published and resol assert.equal(first.candidateSha, seed) assert.equal(next.evidence.upstreamIntegration.sha, upstream) const tester = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'tester') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'test-after', attemptId: id, generation: 3 }, tester) + await prepareRole(f.rt, task, { id: 'test-after', attemptId: id, generation: 3 }, tester) await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: id }, role: 'tester', outputs: {} }) assert.equal(next.evidence.tests.passed, true) }) @@ -795,12 +805,12 @@ test('context admission checkpoints require host rejection and no external execu test('failed context admission preserves an existing unsealed developer workspace',async t=>{ const f=await setup(t,undefined,{pool:true}); f.rt.projectPools.connect({validate:()=>({ok:true})}); await f.start('context-retry') const a=f.attempt(),task=f.rt.state.tasks[f.task.id],dev=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r=>r.role==='developer') - await f.rt.projectDelivery.preparePoolRole(task,{id:'first',attemptId:a.id,generation:1},dev) + await prepareRole(f.rt, task,{id:'first',attemptId:a.id,generation:1},dev) const path=a.workspaces.developer.path writeFileSync(join(path,'code.mjs'),'export const preserved = true;\n') a.seedSha=a.baseSha a.feedback='x'.repeat(49000) // Required current Task input cannot be summarized away. - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task,{id:'second',attemptId:a.id,generation:2},dev),/exceeds/) + await assert.rejects(prepareRole(f.rt, task,{id:'second',attemptId:a.id,generation:2},dev),/exceeds/) assert.equal(readFileSync(join(path,'code.mjs'),'utf8'),'export const preserved = true;\n') assert.equal(a.preparationFailures.at(-1).releasedBeforeExternalStart,false) assert.equal(a.roleSessions.developer.poolClaimId,'first') @@ -814,7 +824,7 @@ test('context failure releases an actual unbound reservation before registering await f.start('unbound-context-failure') const a=f.attempt(),task=f.rt.state.tasks[f.task.id],dev=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r=>r.role==='developer') dev.instructions='x'.repeat(49000) // Required persona cannot be summarized away. - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task,{id:'rejected',attemptId:a.id,generation:1},dev),/exceeds/) + await assert.rejects(prepareRole(f.rt, task,{id:'rejected',attemptId:a.id,generation:1},dev),/exceeds/) const lease=Object.values(f.rt.state.projectResourceLeases).find(x=>x.attemptId===a.id) assert.equal(lease.status,'released') assert.equal(lease.releaseProof.kind,'unbound-reservation-release') @@ -831,7 +841,7 @@ test('a preparation-only successor retains its inherited sealed seed across anot await f.start('retain-seed') const first = f.attempt(), task = f.rt.state.tasks[f.task.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, {id: 'original-dev', attemptId: first.id, generation: 1}, dev) + await prepareRole(f.rt, task, {id: 'original-dev', attemptId: first.id, generation: 1}, dev) writeFileSync(join(first.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({task, claim: {attemptId: first.id}, role: 'developer', outputs: {implementation: 'Fixed'}}) const seed = first.candidateSha @@ -842,7 +852,7 @@ test('a preparation-only successor retains its inherited sealed seed across anot const finalId = await f.rt.projectDelivery.reworkPool({task, claim: {attemptId: nextId, id: 'preparation-retry', botId: 'chief'}, reason: 'Context preparation repaired', force: true}) const final = f.rt.state.projectAttempts[finalId] assert.equal(final.seedSha, seed) - await f.rt.projectDelivery.preparePoolRole(task, {id: 'recovered-dev', attemptId: finalId, generation: 3}, dev) + await prepareRole(f.rt, task, {id: 'recovered-dev', attemptId: finalId, generation: 3}, dev) assert.match(readFileSync(join(final.workspaces.developer.path, 'code.mjs'), 'utf8'), /a\+b/) assert.equal(next.status, 'interrupted', 'Original failed preparation history remains') }) @@ -854,25 +864,25 @@ test('a rework receiver gets hash-verified bounded failed-test facts instead of await f.start('failure-feedback') const a = f.attempt(), task = f.rt.state.tasks[f.task.id], roles = f.rt.projectAgents.deliveryDefinitions('p1').roles const dev = roles.find(row => row.role === 'developer'), tester = roles.find(row => row.role === 'tester') - await f.rt.projectDelivery.preparePoolRole(task, {id: 'dev', attemptId: a.id, generation: 1}, dev) + await prepareRole(f.rt, task, {id: 'dev', attemptId: a.id, generation: 1}, dev) writeFileSync(join(a.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({task, claim: {attemptId: a.id}, role: 'developer', outputs: {}}) - await f.rt.projectDelivery.preparePoolRole(task, {id: 'test', attemptId: a.id, generation: 2}, tester) + await prepareRole(f.rt, task, {id: 'test', attemptId: a.id, generation: 2}, tester) await assert.rejects(f.rt.projectDelivery.completePoolRole({task, claim: {attemptId: a.id}, role: 'tester', outputs: {tests: 'Run host tests'}}), /tests failed/) const id = await f.rt.projectDelivery.reworkPool({task, claim: {attemptId: a.id, id: 'failed-test-return', botId: 'chief'}, reason: 'Fix actual failure', force: true}) const next = f.rt.state.projectAttempts[id] - await f.rt.projectDelivery.preparePoolRole(task, {id: 'repair', attemptId: id, generation: 3}, dev) + await prepareRole(f.rt, task, {id: 'repair', attemptId: id, generation: 3}, dev) const context = next.roleSessions.developer.context.text assert.ok(context.includes('popup layer regression')) assert.ok(context.includes(a.evidence.tests.sha256)) assert.ok(context.length < 48000) const deepId = await f.rt.projectDelivery.reworkPool({task, claim: {attemptId: id, id: 'preparation-only-return', botId: 'chief'}, reason: 'Preserve failure across preparation retry', force: true}) - await f.rt.projectDelivery.preparePoolRole(task, {id: 'deep-repair', attemptId: deepId, generation: 4}, dev) + await prepareRole(f.rt, task, {id: 'deep-repair', attemptId: deepId, generation: 4}, dev) assert.ok(f.rt.state.projectAttempts[deepId].roleSessions.developer.context.text.includes('popup layer regression')) writeFileSync(a.evidence.tests.log, 'tampered failure') - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, {id: 'stale-log', attemptId: deepId, generation: 5}, dev), /log changed/) + await assert.rejects(prepareRole(f.rt, task, {id: 'stale-log', attemptId: deepId, generation: 5}, dev), /log changed/) a.projectId = 'another-project' - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, {id: 'foreign-log', attemptId: deepId, generation: 6}, dev), /another Task or Project/) + await assert.rejects(prepareRole(f.rt, task, {id: 'foreign-log', attemptId: deepId, generation: 6}, dev), /another Task or Project/) }) test('workspace allocation failure releases an actual reservation before any external execution', async t => { @@ -888,7 +898,7 @@ test('workspace allocation failure releases an actual reservation before any ext // A real filesystem obstacle after reservation and before git worktree creation. writeFileSync(join(f.directory, 'delivery'), 'Not a directory') const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'failed-allocation', attemptId: a.id, generation: 1 }, dev), /ENOTDIR|EEXIST/) + await assert.rejects(prepareRole(f.rt, task, { id: 'failed-allocation', attemptId: a.id, generation: 1 }, dev), /ENOTDIR|EEXIST/) const lease = Object.values(f.rt.state.projectResourceLeases).find(row => row.attemptId === a.id) assert.equal(lease.status, 'released') assert.equal(lease.releaseProof.kind, 'unbound-reservation-release') @@ -907,7 +917,7 @@ test('every pool receiver is host-prepared in its own checkout before registerin const a = f.attempt(), task = f.rt.state.tasks[f.task.id] for (const [index, role] of ['developer', 'tester', 'reviewer'].entries()) { const definition = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === role) - const node = await f.rt.projectDelivery.preparePoolRole(task, { id: `prepare-${role}`, attemptId: a.id, generation: index + 1 }, definition) + const node = await prepareRole(f.rt, task, { id: `prepare-${role}`, attemptId: a.id, generation: index + 1 }, definition) assert.equal(readFileSync(join(node.executionWorkDir, 'node_modules/ready'), 'utf8'), node.executionWorkDir) assert.equal(a.workspacePreparations.at(-1).workspaceId, a.workspaces[role].id) assert.equal(a.roleSessions[role].status, 'prepared') @@ -917,7 +927,7 @@ test('every pool receiver is host-prepared in its own checkout before registerin settings.workspacePrepareCommand = [process.execPath, '-e', 'process.exit(37)'] const previous = structuredClone(a.roleSessions.reviewer) const reviewer = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'reviewer') - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'prepare-failed', attemptId: a.id, generation: 9 }, reviewer), error => error.code === 37) + await assert.rejects(prepareRole(f.rt, task, { id: 'prepare-failed', attemptId: a.id, generation: 9 }, reviewer), error => error.code === 37) assert.deepEqual(a.roleSessions.reviewer, previous) assert.equal(a.workspacePreparations.length, 3) assert.equal(a.preparationFailures.at(-1).role, 'reviewer') @@ -949,14 +959,14 @@ test('bound-resource preparation executes before SDK registration and failure re await f.start('resource-preparation') const a = f.attempt(), task = f.rt.state.tasks[f.task.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'prepared-resource', attemptId: a.id, generation: 1 }, dev) + await prepareRole(f.rt, task, { id: 'prepared-resource', attemptId: a.id, generation: 1 }, dev) assert.equal(a.workspacePreparations.at(-1).resourceId, 'one') const original = structuredClone(a.roleSessions.developer) const first = a.resourceLeases.developer await f.rt.projectResources.release(first.id, first.generation) const preparationCommand = settings.workspacePrepareCommand settings.workspacePrepareCommand = [process.execPath, '-e', 'process.exit(37)'] - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'resource-command-failed', attemptId: a.id, generation: 2 }, dev), error => error.code === 37) + await assert.rejects(prepareRole(f.rt, task, { id: 'resource-command-failed', attemptId: a.id, generation: 2 }, dev), error => error.code === 37) assert.equal(f.rt.state.projectResourceLeases[a.resourceLeases.developer.id].status, 'released') assert.equal(a.preparationFailures.at(-1).releasedBeforeExternalStart, true) settings.workspacePrepareCommand = preparationCommand @@ -967,7 +977,8 @@ test('bound-resource preparation executes before SDK registration and failure re await command(...args) throw new Error('Connection lost after external registration') } - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'unknown-preparation', attemptId: a.id, generation: 2 }, dev), /Connection lost/) + const unknownClaim = { id: 'unknown-preparation', attemptId: a.id, generation: 2 } + await assert.rejects(prepareRole(f.rt, task, unknownClaim, dev), /Connection lost/) const held = f.rt.state.projectResourceLeases[a.resourceLeases.developer.id] assert.equal(held.status, 'running') assert.equal(held.worker.handle, 'unknown-worker') @@ -982,7 +993,7 @@ test('unstarted bound delivery refreshes a verified descendant target without re const a=f.attempt(),task=f.rt.state.tasks[f.task.id],dev=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r=>r.role==='developer'),base=a.baseSha a.seedSha=base;a.publicationBinding={number:48,repository:'example/demo',branch:'original-delivery',previousSha:base,upstreamSha:base,upstreamRef:'main'} const originalInstructions=dev.instructions;dev.instructions='x'.repeat(49000) - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task,{id:'before',attemptId:a.id,generation:1},dev),/exceeds/) + await assert.rejects(prepareRole(f.rt, task,{id:'before',attemptId:a.id,generation:1},dev),/exceeds/) dev.instructions=originalInstructions const path=a.workspaces.developer.path;writeFileSync(join(path,'code.mjs'),'export const preservedDraft = true;\n') writeFileSync(join(f.repository,'backend.mjs'),'export const backend = 2;\n');git(f.repository,['add','backend.mjs']);git(f.repository,['commit','-m','Advance trusted target']) @@ -1005,7 +1016,7 @@ test('unstarted bound delivery refreshes a verified descendant target without re if(variant==='missing')f.publisher.read=async()=>({sha:base,state:'open',baseRef:'main',baseSha:'1'.repeat(40)}) if(variant==='moved-head')f.publisher.read=async()=>({sha:target,state:'open',baseRef:'main',baseSha:target}) if(variant==='moved-ref')f.publisher.read=async()=>({sha:base,state:'open',baseRef:'other',baseSha:target}) - const prepare=()=>f.rt.projectDelivery.preparePoolRole(task,{id:'after',attemptId:a.id,generation:2},dev) + const prepare=()=>prepareRole(f.rt, task,{id:'after',attemptId:a.id,generation:2},dev) if(variant==='safe'){ await prepare();assert.equal(a.publicationBinding.upstreamSha,target);assert.equal(a.publicationBindingHistory.length,1) assert.deepEqual(a.publicationBindingHistory[0].previous,old);assert.deepEqual(a.publicationBindingHistory[0].current,a.publicationBinding) @@ -1097,7 +1108,7 @@ test('SDK Task rework preserves an unsealed same-base draft and old cleanup cann await f.rt.projectEndpoint('project-agent-definition',{projectId:'p1',role:'developer',execution:{harness:'codex',resourceIds:['one']}},alice) await f.start('retained-draft') const previous=f.attempt(),task=f.rt.state.tasks[f.task.id],dev=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r=>r.role==='developer') - await f.rt.projectDelivery.preparePoolRole(task,{id:'first',attemptId:previous.id,generation:1},dev) + await prepareRole(f.rt, task,{id:'first',attemptId:previous.id,generation:1},dev) const cwd=previous.workspaces.developer.path writeFileSync(join(cwd,'code.mjs'),'export const preservedDraft = true;\n') const lease=previous.resourceLeases.developer @@ -1105,7 +1116,7 @@ test('SDK Task rework preserves an unsealed same-base draft and old cleanup cann previous.status='interrupted' const id=await f.rt.projectDelivery.reworkPool({task,claim:{id:'recover',attemptId:previous.id,botId:'p1:chief'},reason:'Resume confirmed idle draft',force:true}) const next=f.rt.state.projectAttempts[id] - await f.rt.projectDelivery.preparePoolRole(task,{id:'second',attemptId:id,generation:2},dev) + await prepareRole(f.rt, task,{id:'second',attemptId:id,generation:2},dev) assert.equal(next.workspaces.developer.path,cwd) assert.equal(next.workspaces.developer.retainedDraft,true) assert.equal(readFileSync(join(cwd,'code.mjs'),'utf8'),'export const preservedDraft = true;\n') @@ -1125,14 +1136,14 @@ test('a returned SDK draft reuses its Task workspace after exact supervisor exit await f.start('supervised-draft') const previous = f.attempt(), task = f.rt.state.tasks[f.task.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'old', attemptId: previous.id, generation: 1 }, dev) + await prepareRole(f.rt, task, { id: 'old', attemptId: previous.id, generation: 1 }, dev) const workspace = previous.workspaces.developer, lease = f.rt.state.projectResourceLeases[previous.resourceLeases.developer.id] writeFileSync(join(workspace.path, 'code.mjs'), 'export const add = (a,b) => a+b; // retained draft\n') const proof = { kind: 'systemd-prior-invocation-exited', previousInvocationId: 'a'.repeat(32), at: Date.now() } Object.assign(lease, { status: 'released', worker: { kind: 'sdk', handle: 'old-worker' }, processScope: { kind: 'systemd-control-group', killMode: 'control-group', invocationId: proof.previousInvocationId }, releaseProof: proof }) f.rt.state.projectHarnessWorkers = { 'old-worker': { id: 'old-worker', projectId: 'p1', taskId: task.id, attemptId: previous.id, role: 'developer', leaseId: lease.id, resourceId: 'one', generation: lease.generation, status: 'released', releaseProof: proof } } const id = await f.rt.projectDelivery.reworkPool({ task, claim: { id: 'return', attemptId: previous.id, botId: dev.id }, reason: 'Resume the retained draft after supervisor restart', force: true }) - await f.rt.projectDelivery.preparePoolRole(task, { id: 'next', attemptId: id, generation: 2 }, dev) + await prepareRole(f.rt, task, { id: 'next', attemptId: id, generation: 2 }, dev) const current = f.rt.state.projectAttempts[id].workspaces.developer assert.equal(current.path, workspace.path) assert.equal(current.reusesWorkspaceId, workspace.id) @@ -1164,7 +1175,7 @@ async function additionalIntegrationFixture(t, { conflict = false, merge = true attempt.seedSha=seed attempt.publicationBinding={number:7,repository:'example/demo',branch:'published-ui',previousSha:seed,upstreamSha:base,upstreamRef:'main'} const developer=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row=>row.role==='developer') - await f.rt.projectDelivery.preparePoolRole(task,{id:'integration-claim',attemptId:attempt.id,generation:1},developer) + await prepareRole(f.rt, task,{id:'integration-claim',attemptId:attempt.id,generation:1},developer) const path=attempt.workspaces.developer.path if(merge){if(conflict)assert.throws(()=>git(path,['merge','--no-commit','--no-ff',source]));else git(path,['merge','--no-commit','--no-ff',source])} return {...f,base,source,ref,seed,path,task,attempt,developer} @@ -1204,7 +1215,7 @@ test('manager explicitly pins integration input through service snapshot without assert.equal(replay.result.admission.id,approved.result.admission.id) assert.equal(f.attempt.integrationAdmissions.length,1) assert.equal(f.attempt.candidateSha,undefined) - const node=await f.rt.projectDelivery.preparePoolRole(f.task,{id:'resumed-integration',attemptId:f.attempt.id,generation:99},f.developer) + const node=await prepareRole(f.rt, f.task,{id:'resumed-integration',attemptId:f.attempt.id,generation:99},f.developer) assert.match(node.prompt,/do not git commit/) const text=f.attempt.roleSessions.developer.context.text assert.ok(text.includes(f.source)) @@ -1307,7 +1318,7 @@ test('an approved operator object reaches a resource clone through the existing const query=await f.rt.projectEndpoint('project-development',{version:'project-development/v1',kind:'query',name:'attempt.integration-state',projectId:'p1',payload:{attemptId:attempt.id}},alice) await f.rt.projectEndpoint('project-development',{version:'project-development/v1',kind:'command',name:'attempt.integration-bind',projectId:'p1',requestId:'remote-source',payload:{attemptId:attempt.id,inputs:[{ref,sha}],generation:query.result.generation,expectedWorkspace:null}},alice) const developer=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row=>row.role==='developer') - await f.rt.projectDelivery.preparePoolRole(task,{id:'remote-claim',attemptId:attempt.id,generation:9},developer) + await prepareRole(f.rt, task,{id:'remote-claim',attemptId:attempt.id,generation:9},developer) const workspace=attempt.workspaces.developer assert.equal(workspace.kind,'resource-clone') assert.equal(git(workspace.path,['cat-file','-t',sha]),'commit') @@ -1551,7 +1562,7 @@ test('approved custom role capabilities preserve logical identities while sealin const attempt = f.attempt(), task = f.rt.state.tasks[f.task.id] const prepare = async (policyRole, generation) => { const role = names[policyRole], definition = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === role) - await f.rt.projectDelivery.preparePoolRole(task, { id: `custom-${role}`, attemptId: attempt.id, generation }, definition) + await prepareRole(f.rt, task, { id: `custom-${role}`, attemptId: attempt.id, generation }, definition) return role } const dev = await prepare('developer', 1) @@ -1560,7 +1571,7 @@ test('approved custom role capabilities preserve logical identities while sealin const backup = { ...structuredClone(f.rt.state.projectAgentDefinitions.p1.roles.find(row => row.role === 'implementer')), id: 'p1:backup', role: 'backup' } f.rt.state.projectAgentDefinitions.p1.roles.push(backup); ++f.rt.state.projectAgentDefinitions.p1.revision f.rt.config.projectRepositories['example/demo'].requiredRolePolicySources = { backup: ['MISSING.md'] } - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'bad-backup', attemptId: attempt.id, generation: 2 }, f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'backup')), /Required role policy source is missing/) + await assert.rejects(prepareRole(f.rt, task, { id: 'bad-backup', attemptId: attempt.id, generation: 2 }, f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'backup')), /Required role policy source is missing/) assert.equal(attempt.roleAssignments.developer, 'implementer', 'Failed preparation cannot replace the actual artifact producer') assert.match(attempt.roleSessions[dev].policy.text, /Never invent passing tests/) writeFileSync(join(workspace.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') @@ -1593,12 +1604,12 @@ test('Task execution selects an associated repository and retains it after the P f.rt.state.projects.p1.repository = 'unprovisioned/changed-default' const attempt = f.rt.state.projectAttempts[started.attempt.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r => r.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(f.task, { id: 'repo-dev', attemptId: attempt.id, generation: 1 }, dev) + await prepareRole(f.rt, f.task, { id: 'repo-dev', attemptId: attempt.id, generation: 1 }, dev) assert.equal(attempt.repositoryBinding.repository, 'example/second') assert.equal(attempt.workspaces.developer.repository, second) const association = Object.values(f.rt.state.projectRepositories).find(row => row.repository === 'example/second') delete f.rt.state.projectRepositories[association.id] - await assert.rejects(f.rt.projectDelivery.preparePoolRole(f.task, { id: 'revoked-repo', attemptId: attempt.id, generation: 2 }, dev), /not currently associated/) + await assert.rejects(prepareRole(f.rt, f.task, { id: 'revoked-repo', attemptId: attempt.id, generation: 2 }, dev), /not currently associated/) assert.equal(attempt.requirement.repository, 'example/second') }) @@ -1611,7 +1622,7 @@ test('a repository-free decision Task prepares a scoped workspace and records ve assert.equal(started.attempt.status, 'running') const attempt = f.rt.state.projectAttempts[started.attempt.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r => r.role === 'developer') - const spec = await f.rt.projectDelivery.preparePoolRole(f.task, { id: 'decision-dev', attemptId: attempt.id, generation: 1 }, dev) + const spec = await prepareRole(f.rt, f.task, { id: 'decision-dev', attemptId: attempt.id, generation: 1 }, dev) assert.equal(attempt.baseSha, undefined) assert.equal(attempt.workspaces.developer.kind, 'decision-directory') assert.match(spec.prompt, /no repository/i) @@ -1659,7 +1670,7 @@ test('Release Task closes only after explicit durable authorization and current await f.start('release-task') const a = f.attempt() const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'release-dev', attemptId: a.id, generation: 1 }, dev) + await prepareRole(f.rt, task, { id: 'release-dev', attemptId: a.id, generation: 1 }, dev) await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: a.id }, role: 'developer', outputs: { releasePlan: 'Publish the explicitly authorized base commit' } }) const envelope = { version: 'project-development/v1', kind: 'command', projectId: 'p1', name: 'release.request', requestId: 'publish-release', payload: { attemptId: a.id, tag: 'v1.0.0', sha: a.baseSha, mode: 'publish' } } await assert.rejects(f.rt.projectEndpoint('project-development', envelope, bob), /denied/i) @@ -1712,14 +1723,15 @@ async function baselineResourceFixture(t, kind = 'local') { await f.rt.projectEndpoint('project-agent-definition', { projectId: 'p1', role: 'developer', execution: { harness: 'codex', resourceIds: ['baseline'] } }, alice) await f.start('baseline-recovery') const previous = f.attempt(), task = f.rt.state.tasks[f.task.id], definition = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'first-baseline', attemptId: previous.id, generation: 1 }, definition) + await prepareRole(f.rt, task, { id: 'first-baseline', attemptId: previous.id, generation: 1 }, definition) const workspace = previous.workspaces.developer, lease = previous.resourceLeases.developer await f.rt.projectResources.release(lease.id, lease.generation); previous.status = 'interrupted' + task.collaboration.claim.status = 'failed'; task.collaboration.status = 'blocked' writeFileSync(join(f.repository, 'host-repair.txt'), 'Updated trusted baseline\n'); git(f.repository, ['add', 'host-repair.txt']); git(f.repository, ['commit', '-m', 'Advance trusted base']) const latest = git(f.repository, ['rev-parse', 'HEAD']) const recover = async () => { const id = await f.rt.projectDelivery.reworkPool({ task, claim: { id: 'baseline-rework', attemptId: previous.id, botId: 'p1:chief' }, reason: 'Use repaired baseline', force: true }) - return { next: f.rt.state.projectAttempts[id], prepare: () => f.rt.projectDelivery.preparePoolRole(task, { id: 'next-baseline', attemptId: id, generation: 2 }, definition) } + return { next: f.rt.state.projectAttempts[id], prepare: () => prepareRole(f.rt, task, { id: 'next-baseline', attemptId: id, generation: 2 }, definition) } } return { ...f, previous, task, definition, workspace, latest, recover } } @@ -1795,7 +1807,7 @@ test('actual Chief route admits a pinned inspect input and Dev receives the PR o assert.equal(run.status, 'done', run.failure) assert.equal(attempt.integrationInputs[0].purpose, 'inspect') assert.equal(task.collaboration.botId, 'p1:developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'inspect-after-chief', attemptId: attempt.id, generation: task.collaboration.generation }, f.definition) + await prepareRole(f.rt, task, { id: 'inspect-after-chief', attemptId: attempt.id, generation: task.collaboration.generation }, f.definition) assert.equal(git(attempt.workspaces.developer.path, ['cat-file', '-t', sha]), 'commit') assert.equal(git(attempt.workspaces.developer.path, ['rev-parse', 'HEAD']), attempt.baseSha) assert.equal(attempt.candidateSha, undefined) @@ -1808,6 +1820,6 @@ test('a newly created unsealed successor after failure pins current host HEAD ev const id = await f.rt.projectDelivery.reworkPool({ task: f.task, claim: { id: 'failed-route', attemptId: f.previous.id, botId: 'p1:chief' }, reason: 'Host repair is complete; route back', force: false }) const next = f.rt.state.projectAttempts[id] assert.notEqual(id, f.previous.id); assert.equal(next.baseSha, f.latest) - await f.rt.projectDelivery.preparePoolRole(f.task, { id: 'failed-route-prepare', attemptId: id, generation: 2 }, f.definition) + await prepareRole(f.rt, f.task, { id: 'failed-route-prepare', attemptId: id, generation: 2 }, f.definition) assert.equal(git(next.workspaces.developer.path, ['rev-parse', 'HEAD']), f.latest) }) diff --git a/core/test/project-harness.test.js b/core/test/project-harness.test.js index f864ef605..9f8c3265a 100644 --- a/core/test/project-harness.test.js +++ b/core/test/project-harness.test.js @@ -28,7 +28,7 @@ lines.on('line',line=>{const m=JSON.parse(line); });`) const session = { id: 's', projectId: 'p', taskId: 't', attemptId: 'a', role: 'developer', definition: { role: 'developer', instructions: 'Stable developer convention', skills: [] }, execution: { harness: 'codex' }, context: { text: 'Frozen criteria and role memory' }, resourceLease: { id: 'l', resourceId: 'r', generation: 1 } } const workspace = { id: 'w', projectId: 'p', attemptId: 'a', role: 'developer', path: cwd } - const rt = { home: root, state: { tasks: { t: { id: 't', projectId: 'p' } }, projectResourceLeases: { l: { id: 'l', resourceId: 'r', projectId: 'p', attemptId: 'a', generation: 1, status: 'reserved' } } }, + const rt = { home: root, state: { tasks: { t: { id: 't', projectId: 'p' } }, projectResourceLeases: { l: { id: 'l', resourceId: 'r', projectId: 'p', attemptId: 'a', role: 'developer', generation: 1, status: 'reserved' } } }, config: { projectResources: { r: { type: 'local', root, capacity: 1, runtimeDir, harnesses: ['codex'], allowedProjectOwnerIds: ['owner'], profiles: { owner: { codex: { home: join(root, 'profile'), configDir: join(root, 'profile/codex'), model: 'test-model', envRefs: { OPENAI_API_KEY: 'FLYBOT_TEST_HARNESS_SECRET' } } } } } } }, save: async () => {}, warn() {}, readProject: () => ({ id: 'p', ownerUserId: 'owner' }), requireProjectAccess(actor) { if (actor.memberId !== 'owner') throw new Error('Access denied') }, projectDelivery: { ownsWorkspace: (projectId, path) => projectId === 'p' && path === cwd ? workspace : null, sessionFor: () => session, @@ -562,3 +562,35 @@ await assert.rejects(harness.chiefTurn({agent:{id:'public-p',projectId:'p',lastR assert.equal(rt.projectResources.occupied('r'),0); assert.equal(Object.values(rt.state.projectResourceLeases)[0].chiefRunId,'inspection'); }); + +test('pre-worker SDK admission failure releases its reservation without deleting the Task context', async t => { + for (const kind of ['profile', 'model', 'binding']) { + const { rt, harness, cwd } = await fixture(t) + if (kind === 'profile') delete rt.config.projectResources.r.profiles + if (kind === 'model') delete rt.config.projectResources.r.profiles.owner.codex.model + if (kind === 'binding') { + const { taskSession } = await import('../src/host/project-task-sessions.js') + const session = rt.projectDelivery.sessionFor(), logical = taskSession(rt.state, session) + logical.bindings.push({ version: 'task-role-binding/v2', key: 'foreign', workerId: 'missing', sessionId: 'retained-thread' }) + } + await assert.rejects(harness.spawn({ executionProjectId: 'p', executionWorkDir: cwd, prompt: 'Admission must not leak' })) + assert.equal(rt.projectResources.occupied('r'), 0, kind) + if (kind === 'binding') assert.equal(Object.values(rt.state.projectTaskSessions)[0].bindings[0].sessionId, 'retained-thread') + } +}) + + +test('registered SDK failure before spawn rolls back capacity with a verifiable no-start proof', async t => { + const { rt, harness, cwd } = await fixture(t) + const session = rt.projectDelivery.sessionFor() + session.context.archive = { sources: [] }; session.context.archiveSha256 = 'a'.repeat(64) + rt.projectResources.uploadFile = async () => { throw new Error('Injected pre-spawn archive failure') } + await assert.rejects(harness.spawn({ executionProjectId: 'p', executionWorkDir: cwd, prompt: 'Fail before external spawn' }), /pre-spawn archive/) + const row = Object.values(rt.state.projectHarnessWorkers)[0] + assert.equal(row.status, 'released'); assert.equal(row.spawnPhase, 'aborted') + assert.equal(row.releaseProof.kind, 'sdk-worker-not-started') + assert.equal(rt.projectResources.occupied('r'), 0) + const { confirmedWorkerExit } = await import('../src/host/project-task-sessions.js') + assert.equal(confirmedWorkerExit(rt.state, row), true) + assert.equal(rt.state.projectAttempts, undefined, 'No Task history is rewritten by resource cleanup') +}) diff --git a/core/test/project-resources.test.js b/core/test/project-resources.test.js index 8fe3e03ec..fc2d0c41b 100644 --- a/core/test/project-resources.test.js +++ b/core/test/project-resources.test.js @@ -161,3 +161,73 @@ test('supervisor inspection cannot release a binding changed while the manager y await assert.rejects(releasing, /binding changed/) assert.equal(stored.status, 'running') }) + +test('retired unbound reservations are reclaimed but live preparation and unknown spawn remain held', async () => { + for (const kind of ['returned', 'superseded', 'cancelled', 'active', 'scope', 'worker']) { + const { rt, resources } = fixture() + const lease = await resources.reserve({ projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'orphan', harness: 'flybot', resourceIds: ['local'] }) + rt.state.projectAttempts = { a: { id: 'a', projectId: 'p1', taskId: 't', status: kind === 'returned' || kind === 'cancelled' ? kind : 'running', ...(kind === 'superseded' ? { supersededBy: 'b' } : {}) } } + rt.state.tasks = { t: { id: 't', projectId: 'p1', deliveryAttemptId: 'a' } } + if (kind === 'scope') rt.state.projectResourceLeases[lease.id].processScope = { kind: 'unmanaged-process', hostPid: process.pid } + if (kind === 'worker') rt.state.projectHarnessWorkers = { w: { id: 'w', leaseId: lease.id, generation: lease.generation, status: 'prepared' } } + await resources.reconcile() + assert.equal(resources.occupied('local'), ['returned', 'superseded', 'cancelled'].includes(kind) ? 0 : 1, kind) + if (['returned', 'superseded', 'cancelled'].includes(kind)) await assert.rejects(resources.markStarted(lease.id, lease.generation, { kind: 'sdk', handle: 'late' }), /released/) + } +}) + +test('claim replacement reclaims unbound preparation and failed reclamation save keeps capacity', async () => { + const { rt, resources } = fixture() + rt.state.projectAttempts = { a: { id: 'a', projectId: 'p1', taskId: 't', status: 'running' } } + rt.state.tasks = { t: { id: 't', projectId: 'p1', deliveryAttemptId: 'a', collaboration: { generation: 2, claim: { id: 'claim', generation: 2, status: 'active' } } } } + const lease = await resources.reserve({ projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'owned', harness: 'flybot', resourceIds: ['local'], reservationOwner: { taskId: 't', claimId: 'claim', generation: 2 } }) + await resources.reconcile(); assert.equal(resources.occupied('local'), 1) + rt.state.tasks.t.collaboration.claim.status = 'failed' + const save = rt.save; rt.save = async () => { throw new Error('save failed') } + await assert.rejects(resources.reconcile(), /save failed/) + assert.equal(resources.occupied('local'), 1) + rt.save = save; await resources.reconcile(); assert.equal(resources.occupied('local'), 0) + assert.equal(rt.state.projectResourceLeases[lease.id].reclamation.owner.claimId, 'claim') +}) + +test('a registered or spawning SDK reservation is not inferred idle from a missing controller', async () => { + const { rt, resources } = fixture() + const lease = await resources.reserve({ projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'phase', harness: 'flybot', resourceIds: ['local'] }) + rt.state.projectAttempts = { a: { id: 'a', projectId: 'p1', taskId: 't', status: 'returned' } } + rt.state.projectHarnessWorkers = { sdk: { id: 'sdk', projectId: 'p1', attemptId: 'a', role: 'developer', resourceId: 'local', leaseId: lease.id, generation: lease.generation, status: 'prepared', spawnPhase: 'starting' } } + const row = rt.state.projectResourceLeases[lease.id]; row.worker = { kind: 'sdk', handle: 'sdk' }; row.status = 'running' + await resources.reconcile(); assert.equal(resources.occupied('local'), 1) + rt.state.projectHarnessWorkers.sdk.spawnPhase = 'registered' + await resources.reconcile(); assert.equal(resources.occupied('local'), 0) + await assert.rejects(resources.fenceSpawn(lease.id, lease.generation, 'sdk'), /released|retired/) +}) + +test('asynchronous preparation remains owned until its actual completion, even after claim retirement', async () => { + const { rt, resources } = fixture() + rt.state.projectAttempts = { a: { id: 'a', projectId: 'p1', status: 'running' } } + const lease = await resources.reserve({ projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'preparing', harness: 'flybot', resourceIds: ['local'] }) + const token = await resources.beginPreparation(lease.id, lease.generation) + rt.state.projectAttempts.a.status = 'returned' + await resources.reconcile(); assert.equal(resources.occupied('local'), 1) + assert.equal(await resources.abortReservation(lease.id, lease.generation, 'Attempt ended during preparation'), false) + await resources.endPreparation(token) + await resources.reconcile(); assert.equal(resources.occupied('local'), 0) + await assert.rejects(resources.beginPreparation(lease.id, lease.generation), /released/) +}) + +test('failed reservation/batch admission cannot return a capability or consume capacity', async () => { + const { rt, resources } = fixture() + rt.config.projectResources.local.capacity = 2 + const request = { projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'save-failed', harness: 'flybot', resourceIds: ['local'] } + let writes = 0 + rt.save = async () => { if (++writes === 1) throw new Error('reservation persistence failed') } + await assert.rejects(resources.reserve(request), /persistence failed/) + assert.equal(resources.occupied('local'), 0) + const old = Object.values(rt.state.projectResourceLeases)[0] + await assert.rejects(resources.markStarted(old.id, old.generation, { kind: 'sdk', handle: 'late' }), /released/) + writes = 0; rt.save = async () => { if (++writes === 2) throw new Error('batch persistence failed') } + await assert.rejects(resources.reserveRoles('p1', 'batch', { roles: ['developer', 'tester'].map(role => ({ role, execution: { harness: 'flybot', resourceIds: ['local'] } })) }), /batch persistence failed/) + assert.equal(resources.occupied('local'), 0) + const next = await resources.reserve({ ...request, sessionKey: 'legitimate-tester', role: 'tester' }) + assert.equal(next.status, 'reserved') +}) diff --git a/docs/backend-operations.md b/docs/backend-operations.md index 580630d81..cdb48e9ae 100644 --- a/docs/backend-operations.md +++ b/docs/backend-operations.md @@ -116,3 +116,6 @@ JSON/SQLite回退保留新代码格式守卫、ID和已完成Effects，**不是 #2330：`chief.state.health` 区分 enabled 与 pending/healthy/degraded/overdue，展示最近提醒、启动、完成、成功巡检和有效裁决应用时间。`chief.configure` 的 `inspection.lifecycle` 可配置队列/执行期限、恢复延迟/次数和逾期阈值。超时不等于退出；`waiting-exit` 必须等待可信 SDK/native 或原有 systemd 退出证据。未知执行继续隔离，不能通过删租约或重建会话强行恢复。 + + +#2336：资源 reservation 记录 Task/claim/代次归属。已证明未启动的失效预留由原对账回收；registered/starting/spawned 是不同阶段，starting 或退出未知继续占用。reclamation 和 releaseProof 保留回收依据；不要通过清 bindings、删工作树或强清 lease 恢复容量。 diff --git a/docs/history/2026-11-04-earlier-delivery-roadmap.md b/docs/history/2026-11-04-earlier-delivery-roadmap.md new file mode 100644 index 000000000..f9a0e17c8 --- /dev/null +++ b/docs/history/2026-11-04-earlier-delivery-roadmap.md @@ -0,0 +1,139 @@ +# Earlier delivery roadmap records + +Preserved verbatim from ROADMAP.md on 2026-11-04. Links within each original +record refer to the repository root. Current contracts remain in PROJECT.md and +subsystem documentation; this archive does not grant new work or acceptance. + +S6/#55 integrates current service stages with original native frontend work and +adds repeatable isolated fault/replay acceptance. Actual PR feedback→original Task→ +Codex Dev→Tester has run; final candidate/Review/Chief evidence and compatible UI +rollout are recorded separately. Do not replace #41 or claim its whole visual/ +dual-mode matrix from this backend service acceptance. + +S5/#54 adds the trusted-private compatibility profile and execution-boundary query, +without per-Bot OS infrastructure or live identity/sandbox migration. Current-role, +safe/watch, private-data and concurrency regression suites guard existing rights. +Exact-head full checks and dedicated-account metadata probes are separate from +S6/#55 integrated GitHub/SDK/recovery/UI acceptance, which remains required. + +S4/#53 implements scoped GitHub feedback→existing Task inputs with persistent +revisions, bounded source sweeps and native schedule configuration. Full exact-head +verification, compatible checkpoint rollout and actual GitHub→Bot handoff evidence +must precede delivery claims. S5/#54 trust/capability boundary and S6/#55 integrated +fault/recovery acceptance follow. S1–S3/#60 are deployed; original frontend Task40/ +PR48 now advances its new sealed candidate through native Tester/Reviewer roles. + +Immediate #60 foreground repair completes approved extra-merge host sealing before +S2/S3 batch rollout and current frontend Attempt resumption. Keep the existing +index/MERGE_HEAD, shared-ref restriction, Task/PR and independent candidate gates. +The first batch is deployed; live source admission found a historical-Chief +scope regression. PR62 includes its real-state reproduction and focused guard +repair before resuming the original frontend Attempt. No full frontend acceptance yet. + +Service #49 implementation order: S1/#50 (PR56, deployed), S2/#51 (PR58, verified, +awaiting compatible batch rollout), S3/#52 (PR61,4374 checks and real Codex receipt/pool acceptance verified), then source subscriptions/#53, trusted execution boundary/#54 and +integrated fault/recovery acceptance/#55. #57/PR59 compatibility repair is deployed; +original Chief resumed actual frontend Dev without changing Task40/PR48 ownership. +Each stage keeps memory, architecture and regression evidence with its own PR. + +Immediate compatibility repair #57 removes the verified prior-invocation workspace +and Session resumption blocker before continuing service #51–#55. Preserve the +current frontend Task/Attempt/PR and unknown-exit protections. + + +--- + +The owner-approved plan is [fork #1](https://github.com/zy95-12/FlyBot/issues/1), recorded +in [the fork decision](decisions/2026-10-31-project-first-fork.md). Historical upstream +stages below do not restrict the fork's approved Project and frontend refactor. + +| Milestone | Deliverable | Status / issue | +| --- | --- | --- | +| M0 | Isolated DSH/tool/cwd/lifecycle and unchanged-platform HTTP checks | Merged PR #9; [#2](https://github.com/zy95-12/FlyBot/issues/2) | +| M1 | Project domain and direct Task/Run/Workspace ownership | PR #10 verified locally, awaiting review; [#3](https://github.com/zy95-12/FlyBot/issues/3) | +| M2 | Shared membership, platform identity adapter and backend authorization | PR #11 verified locally, awaiting review; [#4](https://github.com/zy95-12/FlyBot/issues/4) | +| M3 | Worktree, code/tests/PR/review evidence and human acceptance | PR #13 locally verified; integrated real-provider acceptance passed in M6; [#5](https://github.com/zy95-12/FlyBot/issues/5) | +| M4 | Public project Bots and private user/project agents | PR #14 verified locally, awaiting review; [#6](https://github.com/zy95-12/FlyBot/issues/6) | +| M5 | Rebuild task navigation, Project board, details and resource view | PR #15 locally verified, awaiting review; [#7](https://github.com/zy95-12/FlyBot/issues/7) | +| M6 | Repeatable two-user end-to-end acceptance | PR #16: real-model/GitHub/browser acceptance passed; awaiting review; [#8](https://github.com/zy95-12/FlyBot/issues/8) | + +The owner authorized sequential M1–M6 implementation, one PR per milestone. Each PR +updates long-term memory, architecture and tests. Merge status is tracked separately. + +Reuse platform services unchanged first; new project logic/adapters belong in FlyBot. +M0's deterministic model and existing shared-write denials do not imply M2/M3 completion. +GPU pools, multi-host scheduling, auto-merge and exact crash resume remain deferred. +The 30–43 person-day estimate is provisional pending interfaces/frontend design. + +## Authorized M7 sequence + +| Phase | Deliverable | State | +| --- | --- | --- | +| [M7.1 #21](https://github.com/zy95-12/FlyBot/issues/21) | Durable Chief/role memory and temporary Session snapshots | PR #26; full verify and actual DSH context checks passed | +| [M7.2 #22](https://github.com/zy95-12/FlyBot/issues/22) | Execution resource registry, candidate pools and leases | PR #27; full verify + actual local/Docker probe; integration #30 | +| [M7.3 #23](https://github.com/zy95-12/FlyBot/issues/23) | Native/Codex/Claude Agent SDK adapters | Draft PR #28; full verify passed; real authorization acceptance pending | +| [M7.4 #24](https://github.com/zy95-12/FlyBot/issues/24) | Owner configuration, probes and authorized resource terminal | Draft PR #29; full verify + actual PTY checks; browser integration #30 | +| [M7.5 #25](https://github.com/zy95-12/FlyBot/issues/25) | Cross-resource code/evidence transfer and real browser acceptance | Draft PR #30; code/resource/browser checks implemented, authenticated mixed-Harness acceptance pending | + +User explicitly authorized sequential implementation and a separate PR for every issue, +including long-term memory, architecture and tests. Merge/deployment is separate. + +## Automatic development service sequence + +[#49](https://github.com/zy95-12/FlyBot/issues/49) tracks S1 #50 service/TaskView → +S2 #51 durable Effects/outbox → S3 #52 input/role receipts → S4 #53 external +Task subscriptions. S5 #54 trusted deployment/authority follows S1 and applies +throughout; S6 #55 integrated recovery acceptance follows all five. One PR per +implementation issue; no extra tracking-only PR. S1 PR56 has a deployed initial facade (673c741b); long-history follow-up +b9cceb02 passed full4333 verification, deployed at a released checkpoint and +passed three-account scoped HTTP/pagination checks, preserving original UI bytes. +S2/#51 effect/outbox code and fault regressions are implemented; full verification +and source-bound rollout follow before S3 inputs/read receipts. +Independent repository review is not claimed. Original #34–#41 residuals remain. + + +--- + +### Backend convergence #2184 — 2026-11-03 + +SDK Task-role binding hardening: pinned shared definition snapshots, latest observed +partial-thread recovery, verified legacy-key adoption, durable dispatch intent, +explicit incompatible recovery, current rights/resource checks and truthful adapter +capabilities. Native Task-role context restoration, parent waits, transactional Effects +and external command receipts remain dependent acceptance; no live-provider or deploy +claim from controlled process tests. README/API rewrite remains last (#2189). + + +### Collaboration contract convergence — 2026-11-03 (#2175) + +Expose durable Task handoffs through one scoped contract projection, with distinct +pending/accepted/processed/revoked/invalidated states, source/target context references, +actual claim/worker binding, initiative fences and predecessor/successor history. +Idempotent trusted offer/accept ports reuse pools; unresolved old writers cannot be +bypassed by Chief rerouting to another resource. Consultation is a separate private +protocol over retained transcripts, without Task creation or initiative transfer. +Chief prefers an active personal agent for user questions: safe answers require user +publication, watch answers use existing inbox rights; only committed Effects may wake +an authorized native watch. External clients have isolated mailboxes, not fabricated +provider sessions. SDK process/HTTP tests and real-provider acceptance remain distinct. +Native restoration/context bounds, transaction events and parent waits stay on their +original dependent issues; no control-plane, deployment or frontend changes. + + +### Context/memory convergence — 2026-11-03 (#2177) + +Unify model/Harness context budgeting and bounded source tools across persistent +Chief and temporary roles. Unknown capacity stays explicit; full rules/current +constraints remain required, retrieval yields before overflow, and native/SDK input +is checked before delivery. Working summaries use original public role-memory +records with exact source/evidence/uncertainty, separate resident Chief topics and +rollback of uncommitted staging. Facts have source/classification/version/archive; +role-authored experience cannot declare facts or overwrite human control. Private +notes retain history under bounded views. Codex tool history has an explicit cap; +SDK compaction remains runtime-managed, without claiming aggregate usage is current +window pressure. No new memory platform, executor or timer. Native Task-role physical +restore2184 and normalized SQL storage2182 remain separate acceptance; real SDK +verification must name exact runtime source hashes, not borrow deterministic tests. + +#2177 follow-up: validate two sequential native Chief jobs against fresh per-turn facts while retaining the same resident Session; dependent Task admission runtime acceptance remains #2179. + diff --git a/docs/openapi/flybot.openapi.json b/docs/openapi/flybot.openapi.json index f728c1b74..c5d99a39a 100644 --- a/docs/openapi/flybot.openapi.json +++ b/docs/openapi/flybot.openapi.json @@ -3358,6 +3358,129 @@ "oldest" ] }, + "ResourceLease": { + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 200, + "description": "Opaque server-issued ID; never infer authority from it." + }, + "projectId": { + "type": "string", + "minLength": 1, + "maxLength": 200, + "description": "Opaque server-issued ID; never infer authority from it." + }, + "attemptId": { + "type": "string", + "minLength": 1, + "maxLength": 200, + "description": "Opaque server-issued ID; never infer authority from it." + }, + "role": { + "type": "string", + "pattern": "^[a-z][a-z0-9-]{0,31}$" + }, + "sessionKey": { + "type": "string", + "minLength": 1, + "maxLength": 200, + "description": "Opaque server-issued ID; never infer authority from it." + }, + "resourceId": { + "type": "string", + "minLength": 1, + "maxLength": 200, + "description": "Opaque server-issued ID; never infer authority from it." + }, + "generation": { + "type": "integer", + "minimum": 1 + }, + "status": { + "type": "string", + "enum": [ + "reserved", + "running", + "stopping", + "unreachable", + "released" + ] + }, + "reservationOwner": { + "type": "object", + "properties": { + "taskId": { + "type": "string", + "minLength": 1, + "maxLength": 200, + "description": "Opaque server-issued ID; never infer authority from it." + }, + "claimId": { + "type": "string", + "minLength": 1, + "maxLength": 200, + "description": "Opaque server-issued ID; never infer authority from it." + }, + "generation": { + "type": "integer", + "minimum": 1 + }, + "sessionId": { + "type": "string", + "minLength": 1, + "maxLength": 200, + "description": "Opaque server-issued ID; never infer authority from it." + } + }, + "additionalProperties": true + }, + "reclamation": { + "type": "object", + "properties": { + "reason": { + "type": "string", + "maxLength": 400 + }, + "at": { + "type": "integer", + "minimum": 0 + }, + "owner": { + "anyOf": [ + { + "$ref": "#/components/schemas/JsonValue" + }, + { + "type": "null" + } + ] + }, + "effect": { + "$ref": "#/components/schemas/JsonValue" + } + }, + "additionalProperties": true + }, + "releaseProof": { + "$ref": "#/components/schemas/JsonValue" + }, + "releasedAt": { + "type": "integer", + "minimum": 0 + } + }, + "additionalProperties": true, + "required": [ + "id", + "projectId", + "resourceId", + "generation", + "status" + ] + }, "Resource": { "type": "object", "properties": { @@ -19399,7 +19522,7 @@ "resourceLeases": { "type": "array", "items": { - "$ref": "#/components/schemas/JsonValue" + "$ref": "#/components/schemas/ResourceLease" } }, "chief": { @@ -29646,7 +29769,7 @@ "notes": "Read-only transport/initiative history; it is not role claim acceptance.", "implementation": { "path": "core/src/host/project-pools.js", - "sha256": "17cce40f78b034db3b069a87019b0a28a72a43a8449c69bc937bcc6e440061bd" + "sha256": "31eb0837b7fcb63781b5c9a824141d01bb17181707be70d8fb457b2e1cd96cd2" } }, { @@ -29724,7 +29847,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-pools.js", - "sha256": "17cce40f78b034db3b069a87019b0a28a72a43a8449c69bc937bcc6e440061bd" + "sha256": "31eb0837b7fcb63781b5c9a824141d01bb17181707be70d8fb457b2e1cd96cd2" } }, { @@ -30206,7 +30329,7 @@ "notes": "Although a query, the downstream integration checkpoint gate requires manager authority and confirmed release.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" } }, { @@ -30284,7 +30407,7 @@ "notes": "UTF-8 byte offsets, not character offsets; nextOffset preserves character boundaries. Rechecks rights after source I/O. No caller path.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" } }, { @@ -32333,7 +32456,7 @@ "notes": "New Project Task contracts require actual Chief adjudication; third-party clients cannot forge Chief credentials.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" } }, { @@ -32441,7 +32564,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" } }, { @@ -32548,7 +32671,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" } }, { @@ -32642,7 +32765,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" } }, { @@ -32736,7 +32859,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" } }, { @@ -32845,7 +32968,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" } }, { @@ -33897,7 +34020,7 @@ "notes": "payload.requestId identifies the original SDK permission question; outer requestId is the independent command receipt ID. A role string never grants execution identity.", "implementation": { "path": "core/src/host/project-harness.js", - "sha256": "ac15df1ec8a90cd422703abd92a168c8f9e4091f1236c8b152eefe2f29a9beee" + "sha256": "0fee68a81e90d88054b9e56fc5c2031d2bdbb5520e827d18412e1900d14dc4a1" } }, { @@ -39217,7 +39340,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-resources.js", - "sha256": "64b2abadc7900ca005b1b90217e8f512e867932c92e4033904417878fa3112ce" + "sha256": "c987f3dd0ab59491898e1427e210a1ccb89123088760275f8a449b6fe4f18375" } }, { @@ -42809,7 +42932,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-pools.js", - "sha256": "17cce40f78b034db3b069a87019b0a28a72a43a8449c69bc937bcc6e440061bd" + "sha256": "31eb0837b7fcb63781b5c9a824141d01bb17181707be70d8fb457b2e1cd96cd2" }, "aliasOf": "development.query.handoff.get" }, @@ -42885,7 +43008,7 @@ "notes": "Read-only transport/initiative history; it is not role claim acceptance.", "implementation": { "path": "core/src/host/project-pools.js", - "sha256": "17cce40f78b034db3b069a87019b0a28a72a43a8449c69bc937bcc6e440061bd" + "sha256": "31eb0837b7fcb63781b5c9a824141d01bb17181707be70d8fb457b2e1cd96cd2" }, "aliasOf": "development.query.handoff.list" }, @@ -43262,7 +43385,7 @@ "notes": "New Project Task contracts require actual Chief adjudication; third-party clients cannot forge Chief credentials.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" }, "aliasOf": "development.command.task.accept" }, @@ -43355,7 +43478,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" }, "aliasOf": "development.command.task.reject" }, @@ -43447,7 +43570,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" }, "aliasOf": "development.command.attempt.stop" }, @@ -43539,7 +43662,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" }, "aliasOf": "development.command.attempt.cleanup" }, @@ -43918,7 +44041,7 @@ "notes": "All current authorization, Project/Task bindings and domain fences are rechecked server-side.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" }, "aliasOf": "development.command.attempt.publication-retry" }, @@ -43994,7 +44117,7 @@ "notes": "UTF-8 byte offsets, not character offsets; nextOffset preserves character boundaries. Rechecks rights after source I/O. No caller path.", "implementation": { "path": "core/src/host/project-delivery.js", - "sha256": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8" + "sha256": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1" }, "aliasOf": "development.query.attempt.log" }, @@ -44845,7 +44968,7 @@ "notes": "payload.requestId identifies the original SDK permission question; outer requestId is the independent command receipt ID. A role string never grants execution identity.", "implementation": { "path": "core/src/host/project-harness.js", - "sha256": "ac15df1ec8a90cd422703abd92a168c8f9e4091f1236c8b152eefe2f29a9beee" + "sha256": "0fee68a81e90d88054b9e56fc5c2031d2bdbb5520e827d18412e1900d14dc4a1" }, "aliasOf": "development.command.execution.input" }, @@ -45317,7 +45440,7 @@ "core/src/host/groups.js": "d7f0175c8ad67522aca392d7e47f9d2773c02f525bd6f33d564cbec784217bbf", "core/src/host/harness.js": "49e4b3ad56827d47cf8c5af99600026437c37a5e3b7e01bad2000983f6541af3", "core/src/host/home-lock.js": "c4ad69a0a3a2726888b69ffc46a624644379960834f385d83b93f88b2f05fa04", - "core/src/host/hooks.js": "f5416b0a1006c919b99eeea2636c4bd22e63c937ff64e0fb26ac9cfbcb134170", + "core/src/host/hooks.js": "3a4b4e24e1ddb760ea499397a03d019ef2efa5dd01324dc286338d8465ddcd26", "core/src/host/images.js": "18579c262f130ffba10b51cf8f9562730f1ebfa3eff6321f698d72b537fae9d4", "core/src/host/journal.js": "5050048eba15af5520664b7f40dbca1aa457f08c3bbf5ed2798fc99ee9d64906", "core/src/host/meetings.js": "eb68d1cc18ed4370dc8271bd9620966aabb9d8c01001e19fbfc16124852bc063", @@ -45353,7 +45476,7 @@ "core/src/host/project-credentials.js": "18fe61057fb920664fc5077d81eee0957f7eaf64821dd10d2d2a3cd463f5fdea", "core/src/host/project-delegations.js": "d4d86192551994a3419e0d2ca0f64bc4d50a75960970d2b79ed9f97b82940903", "core/src/host/project-delivery-template.js": "daa5cc8185abc1ec84421ae3fcd5574a50573cfd1a7c4cb10941158e29c79a13", - "core/src/host/project-delivery.js": "aa71f5d8a9f2390c5814b4e7ec7f56db8a1f32472beb7f53791b7ce6477299c8", + "core/src/host/project-delivery.js": "87f6aea45fb257987d74cf7c949967eb9f302b33549989f7d58ddb59a6fadef1", "core/src/host/project-development-service.js": "cb1e3b1cda8656f2add087f43be839776d9f6d2e93965d360bf3c8c2b1462178", "core/src/host/project-domain.js": "6d6a20727905005f7a8de0247a429a10ffbd1436773ee96a5d7615b2fb763e82", "core/src/host/project-effects.js": "5557782333a0b3437dacde2b6924fe12ea54225098571fa8e404d79a432be549", @@ -45368,21 +45491,21 @@ "core/src/host/project-github-read.js": "dbbdbbc0d917bbf20c4fdf7306464cbb682b0191b98c36fddbe70e6837a57943", "core/src/host/project-github.js": "bf6d3891a6fa9c0516cd330d3322a750fc6db8813752f7801d4d9760fca84061", "core/src/host/project-harness-turn.js": "a8ab111b8d5473a27413968e4ea5c2631ea472c992414e9387da245463f7b21a", - "core/src/host/project-harness.js": "ac15df1ec8a90cd422703abd92a168c8f9e4091f1236c8b152eefe2f29a9beee", + "core/src/host/project-harness.js": "0fee68a81e90d88054b9e56fc5c2031d2bdbb5520e827d18412e1900d14dc4a1", "core/src/host/project-history-pages.js": "38958623f0aa98099e570559c55a207285fd0349689ffdcec47b965c40f4e1bc", "core/src/host/project-inputs.js": "dd52c6ddc7b0f2614ecaa3c3cccbea7a39ea914f24e1602946b93a290eff622e", "core/src/host/project-lifecycle.js": "055784e0a1cbf2a9e465e990c60fbd6df4b8844052db43bbcb7cae5de9282936", "core/src/host/project-merge-heads.js": "9e5e39b8aaa191c2175feaafcf6524b8a16dbafdfa573f94222b9f71d8482074", "core/src/host/project-metrics.js": "80f209d76955e5a3058fa66757543059fded1030625e5e155be5254494ce5264", "core/src/host/project-native-sessions.js": "d8b796fce7a03252f7e6f10c2725ff97501ab4ba81492d8bdb8da1807a2db581", - "core/src/host/project-pools.js": "17cce40f78b034db3b069a87019b0a28a72a43a8449c69bc937bcc6e440061bd", + "core/src/host/project-pools.js": "31eb0837b7fcb63781b5c9a824141d01bb17181707be70d8fb457b2e1cd96cd2", "core/src/host/project-private-file.js": "e8b51d8da7bcd72658eb6eb953dbf9d0054ea0ee519b0229785e8ea0aacef6df", "core/src/host/project-process-proof.js": "85f4ed03f5191b842ca899b712c99177c059658d43a66e8ddeedc1c06eafcdea", "core/src/host/project-release-github.js": "47c52d0057dd39f1ee627c619970efad36e0134bc0c7bc218d43187230d51d64", "core/src/host/project-releases.js": "ac588064b036f2273c7b61cf3389c1da720e6f2166c37d269afd9d9095eb6885", "core/src/host/project-repository-policy.js": "6ccc5d682016335c4e780a420a48e3ecee02c17315a76f89bacc72d49dfc5040", "core/src/host/project-resource-workspaces.js": "cc9525aab5ea9ea09d4fb2c557a1048fa034d638c0b80dbebea9cc8eba0ede8b", - "core/src/host/project-resources.js": "64b2abadc7900ca005b1b90217e8f512e867932c92e4033904417878fa3112ce", + "core/src/host/project-resources.js": "c987f3dd0ab59491898e1427e210a1ccb89123088760275f8a449b6fe4f18375", "core/src/host/project-role-admission.js": "ccd5f3ac4a5d1f0010a3e533e6809b20a4f88551fc55e6c775d8289d8f9e100f", "core/src/host/project-role-context.js": "e032b6c4e8b7a902c1dedc114782025d73c630d2419f912e10cbd103c8d7f69c", "core/src/host/project-role-governance.js": "fe9ca70113666b0223f55b9225890609c5a1e7ddcb9b6105a3dc0edc3b2ef005", @@ -45393,7 +45516,7 @@ "core/src/host/project-scheduler.js": "9419010ed18fcfe801cab23562ea2ad53cb4dc500078ff1ec2af4438cc6726a6", "core/src/host/project-service-events.js": "71917d1ab367a9db8bef8c1d1003228255a57528e84a57e76de3a39d08ac28ec", "core/src/host/project-service-sse.js": "df97d6b11156da885a6a47d50a4a77c8c99b6af5eb0066de14312cc512762297", - "core/src/host/project-service-views.js": "e34b36f78f4dcbb7e1f507203ba843959fa76a119fdecdc5b9902c39bbbef3fa", + "core/src/host/project-service-views.js": "176bcbe521625324906e05af3fa7cfe202c703658d5bcfefb7a2af9eeec96568", "core/src/host/project-service.js": "8ba30c1d7c066867eae06a6612d5bbba3bc2278f34fcf7f1c2242876565dba8f", "core/src/host/project-sources.js": "cf4c11c8e853f3091c4d159b850d06fe7e8bf291fdc616d251eb23d071d63403", "core/src/host/project-state-format.js": "4838b344439e585287fdc25ef15cea44d7aff4199c7a380ebbfa8cfcdf24f678", @@ -45403,7 +45526,7 @@ "core/src/host/project-state.js": "5d6736eeedcfc09a3949a33457a4d4daf3b3745bebe2823d7852fd5c48f1b825", "core/src/host/project-task-lifecycle.js": "20940d6eeae321fcb064995990085af1f0e94d0a41fde50dc29a35e1af1a58a3", "core/src/host/project-task-repository.js": "6dcba5646b406acae23d269276d58bc7b2591ca7d5589ed88ad556a0d5b65980", - "core/src/host/project-task-sessions.js": "fa33428ce5d4446d230e28569b8a8b0484810e0ddc0129cbb17c7de20e0b43bf", + "core/src/host/project-task-sessions.js": "140dfbed527988942d321c44b983bc07a14b47011133c7f6b81e08c9f4ced699", "core/src/host/project-team.js": "72c3975a4fed654f3750e82a9388320934be7cf322b05cd59bbac26d6e4ede77", "core/src/host/project-test-diagnostics.js": "fed63d4c8631ba8d5dcf39764829bb4c0fe9fb28d7e95b61ed232a21d48c4957", "core/src/host/prompt.js": "919e8c5b28c27af6b9570a7be22609230ea4395d3fd4b7215083f9bc25978d72", diff --git a/docs/subsystems/backend-service.md b/docs/subsystems/backend-service.md index 1bcf3cfc2..733c3da46 100644 --- a/docs/subsystems/backend-service.md +++ b/docs/subsystems/backend-service.md @@ -129,3 +129,6 @@ transport. Source API changes require deliberate catalog and guide review. Chief invocation history accepts the combined configured maximum: one initial invocation, up to four protocol corrections and up to four inspection recoveries (nine records). Public schema validation covers that legal configuration; correction feedback itself retains its separate five-entry bound. + + +#2336 types resource lease ownership/reclamation metadata in OpenAPI. The existing project-resources surface reports the actual reservation lifecycle; no general-purpose release/force-clear endpoint is introduced. diff --git a/docs/subsystems/core-host.md b/docs/subsystems/core-host.md index b26ad6b6f..6936d329b 100644 --- a/docs/subsystems/core-host.md +++ b/docs/subsystems/core-host.md @@ -987,3 +987,6 @@ Store builds safe per-Project event projections from each queued immutable snaps The [compatibility contract](project-compatibility.md) inventories original entrypoints and persistent IDs. Store records the legacy boundary only after Effects recovery; Task card constructors/drivers respect it. Original personal Chief and group channels stay in use. Associated native groups require both original history ACL and current Project membership, without importing private history. #2292 routes explicit release request/reconciliation/status through the existing authenticated channel and restricted gateway. Read-only hosts admit status only; Task result-ready for a release requires Chief approval plus committed verified provider proof, not a PR or model claim. See [release contract](project-releases.md). + + +#2336 connects Workflow admission abort to existing carrier release and claim-scoped reservation rollback. Failure before a returned workerId no longer skips cleanup; unknown execution remains held. See [resources](project-resources.md). diff --git a/docs/subsystems/project-delivery.md b/docs/subsystems/project-delivery.md index 9e85aa339..c9dd4a0a8 100644 --- a/docs/subsystems/project-delivery.md +++ b/docs/subsystems/project-delivery.md @@ -572,3 +572,11 @@ Publication now selects either credential provider through the existing broker, #2330 makes Chief dispatch preparation asynchronous so its own watchdog stays live. The explicit PR-input regression waits for the actual scoped snapshot before invoking its callback; workspace/commit and authority gates remain unchanged. + + +#2336 stamps reservation ownership at receiving-role preparation and exposes a +host-only abortPoolAdmission port for failures after allocation/context/archive or +binding admission. It scopes cleanup to the original Task/Attempt/claim/generation, +retains workspaces and Session history, and delegates only proven unstarted release +to the existing resource service. Direct resource-preparation regressions now install the exact +current claim expected from the pool, preserving its original capacity-hold assertion. diff --git a/docs/subsystems/project-development-service.md b/docs/subsystems/project-development-service.md index e82ccc996..8bba12a29 100644 --- a/docs/subsystems/project-development-service.md +++ b/docs/subsystems/project-development-service.md @@ -235,3 +235,6 @@ v1 github.connection queries and human-only github.connection.configure commands #2327 projects pending Chief protocol correction as awaiting-chief/chief-correction, and exhausted correction as blocked/chief-recovery. No framework-selected personal-agent escalation or destination is inferred. + + +#2336 execution metadata includes startupPhase, while raw resource views retain reservation owner, reclamation reason/effect and exact release proof. Confirmed no-start evidence uses the shared exit predicate; phase alone grants no execution or acceptance. diff --git a/docs/subsystems/project-harness.md b/docs/subsystems/project-harness.md index 1c31af8bc..bb1dc8972 100644 --- a/docs/subsystems/project-harness.md +++ b/docs/subsystems/project-harness.md @@ -381,3 +381,11 @@ Native input budgeting fences the actual `claim.workerId` against the temporary Chief lease correlation is written on the canonical reservation, since reserve returns a copy. Its save remains inside the pre-dispatch cleanup boundary: a failed correlation save releases only the proven unbound reservation, preserving runtime capacity and the original failure. + + +#2336 wraps SDK profile/model/binding, registration, archive transfer and spawn +admission in one cleanup boundary. Before external spawn a durable resource.spawn +intent fences the exact lease/generation/worker. Registered-only failures roll back +through resource.reclaim; possibly spawned work retains occupancy until real exit. +The shared exit predicate accepts only the exact matching no-start proof, not missing +PID/workerId. This does not clear prior Task-role bindings or enable a fresh Session. diff --git a/docs/subsystems/project-pools.md b/docs/subsystems/project-pools.md index a0cf97c3a..6ad1f4af0 100644 --- a/docs/subsystems/project-pools.md +++ b/docs/subsystems/project-pools.md @@ -187,3 +187,10 @@ Canonical claims retain the admitted requirement revision. Decision admission re #2328 preserves wait checkpoints before handoff/child resume. Claim acceptance waits for original projectQueue resume admission outside the pool lock, rechecks authority/generation and does not spend a claim generation while queued. No extra workflow or default next role is added. + + +#2336 invokes the adapter's host-only admission-abort port even when execute never +returned a workerId. Bound native/SDK carriers use their existing release lifecycle; +unstarted reservations use exact claim-scoped rollback. A failed/unknown exit remains +false and retains occupancy. Claim failure is auditable and may return to Chief after +confirmed cleanup; neither empty workerId nor Task failure alone proves exit. diff --git a/docs/subsystems/project-resources.md b/docs/subsystems/project-resources.md index eec034a1a..795684940 100644 --- a/docs/subsystems/project-resources.md +++ b/docs/subsystems/project-resources.md @@ -140,3 +140,27 @@ Native resource release now requires a matching projectNativeWorkers native-agen Removing a current role resource grant or changing its registered Resource binding revokes ongoing canonical execution at the existing scheduler check. The same SDK/native stop and physical-exit proof govern release; current permissions never justify a silent new Session or an assumed exit. + + +## Reservation ownership and startup rollback — #2336 + +Pool reservations retain Task/claim/generation/Session ownership. The existing +reconciler reclaims retired/superseded or invalid-claim reservations only after +checking no worker, process scope, bound Session or ephemeral Bot exists. Startup +registration and reclamation serialize with binding; late binding/spawn refuses a +released or stale owner. Active preparation and unknown external execution remain +occupied regardless of age or an empty controller list. + +SDK registration uses a durable starting fence before external spawn. A registered +worker that never crossed that fence may receive exact sdk-worker-not-started proof; +starting/spawned workers require the original physical exit protocol. Reclamation +uses committed Effects, preserves original owner/reason and rolls back uncommitted +state on save failure. The dispatcher is woken only after release. No workspace, +draft, candidate, logical Session or prior binding is deleted, and no new timer exists. + + +Preparation itself has a fenced active/settled lifetime. In-flight commands keep +capacity after claim retirement until their actual callback settles. A restarted +active preparation remains unknown; only the existing exact local supervisor proof +can reconcile it automatically (remote preparation is not inferred exited). Initial +reservation/batch save failures cannot leak a startup capability before return. diff --git a/patches/ds-bot/files/src/host/hooks.js b/patches/ds-bot/files/src/host/hooks.js index e101d41e9..adca68bbf 100644 --- a/patches/ds-bot/files/src/host/hooks.js +++ b/patches/ds-bot/files/src/host/hooks.js @@ -1393,6 +1393,13 @@ export function install(rt) { ready({ task, definition }) { try { rt.projectResources.plan(task.projectId, { roles: [definition] }); return true } catch { return false } }, + async abort({ task, claim, reason }) { + const attempt = rt.state.projectAttempts?.[claim.attemptId] + for (const session of Object.values(attempt?.roleSessions ?? {})) if (session.poolClaimId === claim.id && session.botId) { + try { await lifecycle.release(session.botId) } catch { /* Do not assume unknown exit. */ } + } + return rt.projectDelivery.abortPoolAdmission(task, claim, reason) + }, async execute({ task, claim, definition }) { return lifecycle.spawn(await rt.projectDelivery.preparePoolRole(task, claim, definition)) }, async dispatch({ task, claim, workerId }) { const taskInbox = await rt.projectInputs?.prepare(task.id, claim) diff --git a/patches/ds-bot/files/src/host/project-delivery.js b/patches/ds-bot/files/src/host/project-delivery.js index b2f26f008..172cfa53c 100644 --- a/patches/ds-bot/files/src/host/project-delivery.js +++ b/patches/ds-bot/files/src/host/project-delivery.js @@ -900,6 +900,13 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { rt.taskDeliveryState?.(attempt.taskId, attempt.id, 'failed', attempt.failure.message) } } + async function abortPoolAdmission(task, claim, reason) { + const attempt = ownEntry(attempts(), claim.attemptId) + if (!attempt || attempt.taskId !== task.id || attempt.projectId !== task.projectId) return false + const rows = Object.values(state.projectResourceLeases ?? {}).filter(row => row.attemptId === attempt.id && row.reservationOwner?.claimId === claim.id && row.reservationOwner.generation === claim.generation) + for (const lease of rows) await rt.projectResources.abortReservation(lease.id, lease.generation, reason) + return rows.every(row => row.status === 'released') + } async function preparePoolRole(task, claim, definition) { const attempt = ownEntry(attempts(), claim.attemptId) if (!attempt || attempt.taskId !== task.id || attempt.protocol !== 'project-handoff/v1' || attempt.status !== 'running') throw reject('Pool Attempt is unavailable') @@ -915,10 +922,12 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { const policy = await repositoryRolePolicy(project.id, attempt.baseSha, role, policyRole, project) const sessionId = `role-session-${randomUUID()}` const execution = definition.execution - const lease = execution.resourceIds?.length ? await rt.projectResources.reserve({ projectId: project.id, attemptId: attempt.id, role, sessionKey: sessionId, ...execution }) : null + const lease = execution.resourceIds?.length ? await rt.projectResources.reserve({ projectId: project.id, attemptId: attempt.id, role, sessionKey: sessionId, reservationOwner: { taskId: task.id, claimId: claim.id, generation: claim.generation, sessionId }, ...execution }) : null if (lease) attempt.resourceLeases[role] = lease let context, workspace + const preparation = lease ? await rt.projectResources.beginPreparation(lease.id, lease.generation) : null try { + try { const freshWorkspace = !attempt.workspaces[role] await allocate(attempt, project, settings, [role], { [role]: policyRole }) workspace = attempt.workspaces[role] @@ -930,6 +939,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { } if (project.repository) { await prepareWorkspace(attempt, workspace, settings); await prepareIntegrations(attempt, workspace) } context = await buildProjectContext(rt, { retrieval: true, policy, projectId: project.id, taskId: task.id, role, revision: pinned.revision, resourceId: lease?.resourceId, definition, memory: definition.memory, knowledge: ownEntry(state.projectKnowledge ?? {}, project.id) ?? [], requirement: attempt.requirement, feedback: attempt.feedback, projectFacts: { workspaceMaterialization: workspace.materialization ?? null, ...(pinned.change ? { roleDefinitionChange: pinned.change } : {}), integrationInputs: structuredClone(attempt.integrationInputs ?? []), testFailure: await failureContext(attempt), publicationBinding: attempt.publicationBinding ?? null, integrationInstruction: attempt.publicationBinding ? 'Merge the pinned upstreamSha into the published candidate using --no-commit, resolve and stage conflicts before handoff. Host sealing retains both trusted inputs; never fabricate parents or discard backend changes.' : null, workingDraft: !attempt.candidateSha && workspaceFor(attempt, 'developer') ? { kind: 'unverified-workspace-reference', resourceId: workspaceFor(attempt, 'developer').resourceId, path: workspaceFor(attempt, 'developer').resourceId === workspace.resourceId ? workspaceFor(attempt, 'developer').path : null, note: 'Unsealed source, not candidate or verification evidence. Cross-resource drafts require a sealed artifact before verification.' } : null, handoff: task.collaboration?.history.at(-1) ? { purpose: task.collaboration.history.at(-1).purpose, reason: task.collaboration.history.at(-1).reason } : null, candidateSha: attempt.candidateSha ?? null, tests: attempt.evidence.tests ? { sha: attempt.evidence.tests.sha, passed: attempt.evidence.tests.passed, sha256: attempt.evidence.tests.sha256, command: attempt.evidence.tests.command } : null, review: attempt.evidence.review ? { sha: attempt.evidence.review.sha, verdict: attempt.evidence.review.verdict, summary: attempt.evidence.review.summary } : null }, repository: await repositoryContext(project.id, attempt.baseSha, project) }) + } finally { if (preparation) await rt.projectResources.endPreparation(preparation) } } catch (error) { const registered = lease && ownEntry(state.projectResourceLeases ?? {}, lease.id) const unbound = registered && registered.generation === lease.generation && !registered.worker && !registered.processScope && !Object.values(state.projectHarnessWorkers ?? {}).some(row => row.leaseId === lease.id) && !Object.values(attempt.roleSessions ?? {}).some(row => row.id === sessionId) @@ -1112,7 +1122,7 @@ export function createProjectDelivery(rt, { publisher, releaseProvider } = {}) { } } const ownsWorkspace = (projectId, path) => Object.values(state.projectWorkspaces ?? {}).find(workspace => workspace.projectId === projectId && workspace.path === path && DELIVERY_ROLES.includes(workspace.policyRole ?? workspace.role) && ['worktree', 'resource-clone', 'decision-directory'].includes(workspace.kind) && workspace.status === 'allocated' && ongoing.has(ownEntry(attempts(), workspace.attemptId)?.status)) - return { releases, recoveryFacts, admitChiefInputs, reconcileExecutions: () => reconcileExecutionAuthority(rt), exitConfirmed: workerId => executionExitConfirmed(rt, workerId), nativeSessions, endpoint, context, admitChildInputs: (taskId, claim) => serialAttempt(claim.attemptId, () => childInputs.admit(taskId, claim)), preparePoolRole, completePoolRole, chiefAccept, reworkPool, recordExecution, repositoryContext, repositoryRolePolicy, bindSession, releaseSession, sessionFor, recover, runReported, ownsWorkspace, gate: input => serialAttempt(current(input.taskId)?.id ?? input.taskId, () => gate(input)) } + return { releases, recoveryFacts, admitChiefInputs, reconcileExecutions: () => reconcileExecutionAuthority(rt), exitConfirmed: workerId => executionExitConfirmed(rt, workerId), nativeSessions, endpoint, context, admitChildInputs: (taskId, claim) => serialAttempt(claim.attemptId, () => childInputs.admit(taskId, claim)), preparePoolRole, abortPoolAdmission, completePoolRole, chiefAccept, reworkPool, recordExecution, repositoryContext, repositoryRolePolicy, bindSession, releaseSession, sessionFor, recover, runReported, ownsWorkspace, gate: input => serialAttempt(current(input.taskId)?.id ?? input.taskId, () => gate(input)) } } export function install(rt) { diff --git a/patches/ds-bot/files/src/host/project-harness.js b/patches/ds-bot/files/src/host/project-harness.js index 631f7b549..f3302f749 100644 --- a/patches/ds-bot/files/src/host/project-harness.js +++ b/patches/ds-bot/files/src/host/project-harness.js @@ -72,6 +72,27 @@ export function createProjectHarness(rt) { return JSON.parse(text) } async function spawn(spec) { + const workspace = spec.preparedChief?.workspace ?? rt.projectDelivery.ownsWorkspace(spec.executionProjectId, spec.executionWorkDir) + const session = spec.preparedChief?.session ?? (workspace && rt.projectDelivery.sessionFor(workspace)) + let preparation + try { + if (session?.resourceLease) preparation = await rt.projectResources.beginPreparation(session.resourceLease.id, session.resourceLease.generation) + const id = await spawnPrepared(spec) + if (preparation) { await rt.projectResources.endPreparation(preparation); preparation = null } + return id + } + catch (error) { + if (preparation) { const token = preparation; preparation = null; try { await rt.projectResources.endPreparation(token) } catch { /* Keep unconfirmed preparation capacity; preserve the admission failure. */ } } + const lease = session?.resourceLease + if (lease) { + const rows = Object.values(workers()).filter(row => row.leaseId === lease.id && row.generation === lease.generation) + for (const row of rows) if (active.has(row.id)) { try { await release(row.id) } catch { /* Unknown external exit retains capacity. */ } } + try { await rt.projectResources.abortReservation(lease.id, lease.generation, `SDK admission failed: ${error.code ?? 'bot/harness'}`) } catch { /* Preserve the original failure and any unresolved reservation. */ } + } + throw error + } + } + async function spawnPrepared(spec) { const chief = spec.preparedChief if (chief && !rt.projectAgents.isDelegationActive(chief.agentId, chief.delegationId)) throw refuse('Chief delegation is no longer active') if (!spec.executionWorkDir && !chief) return null @@ -103,7 +124,7 @@ export function createProjectHarness(rt) { const row = { id, ...(conversation?.roomId ? { roomId: conversation.roomId } : {}), model: session.execution.model ?? selected.selected.model ?? null, chiefRunId: chief?.runId ?? null, chiefInvocation: chief?.runId ? rt.state.projectExecutions?.[chief.runId]?.invocation ?? null : null, visibility: conversation?.visibility ?? 'project', ...(conversation?.visibility === 'private' ? { memberId: conversation.memberId } : {}), projectId: session.projectId, taskId: session.taskId, attemptId: session.attemptId, role: session.role, policyRole: session.policyRole ?? session.definition?.policyRole ?? session.role, sessionKey: session.id, resourceId: session.resourceLease.resourceId, leaseId: session.resourceLease.id, generation: session.resourceLease.generation, harness: session.execution.harness, sdkVersion: null, expectedSdkVersion: VERSIONS[session.execution.harness], principalId: selected.principalId, - cwd: workspace.path, scratchDirectory, status: 'prepared', executionStartedAt: null, createdAt: Date.now(), eventCount: 0 } + cwd: workspace.path, scratchDirectory, spawnPhase: 'registered', status: 'prepared', executionStartedAt: null, createdAt: Date.now(), eventCount: 0 } if (logical) Object.assign(row, { logicalSessionId: logical.id, bindingKey, bindingVersion: 'task-role-binding/v2', definitionKey, ...(previous ? { backendSessionId: previous.sessionId, continuedFrom: previous.workerId, ...(previous.key !== bindingKey ? { migratedBindingKey: previous.key } : {}) } : { rotationReason: 'First Task-role execution' }) }) else if (chief) Object.assign(row, { logicalSessionId: chief.agentId, bindingKey, ...(chief.resumeSessionId ? { backendSessionId: chief.resumeSessionId, continuedFrom: chief.previousWorkerId } : {}) }) setEntry(workers(), id, row) @@ -121,8 +142,10 @@ export function createProjectHarness(rt) { row.contextArchive = { sha256: session.context.archiveSha256, file: source } } row.eventLog = join(directory, `${evidenceName}-sdk.jsonl`) + await rt.projectResources.fenceSpawn(row.leaseId, row.generation, id) const process = rt.projectResources.startProcess(row.projectId, row.resourceId, { argv: ['node', workerPath], cwd: workspace.path }) const control = { process, chief, contextBudget, resumeSessionId: chief?.resumeSessionId ?? previous?.sessionId, continuedPolicyHash: previous?.policySha256, archivePath, logical, bindingKey, env: selected.env, profile: { ...selected.selected, ...(session.execution.model ? { model: session.execution.model } : {}) }, session, prompt: spec.prompt, outputs: [], closed: false, ready: false } + row.spawnPhase = 'spawned' active.set(id, control) let readyResolve, readyReject control.handshake = new Promise((resolve, reject) => { readyResolve = resolve; readyReject = reject }) diff --git a/patches/ds-bot/files/src/host/project-pools.js b/patches/ds-bot/files/src/host/project-pools.js index 2d20659b9..4e7b56012 100644 --- a/patches/ds-bot/files/src/host/project-pools.js +++ b/patches/ds-bot/files/src/host/project-pools.js @@ -415,6 +415,7 @@ export function createProjectPools(rt) { if (!failedClaim) continue failedClaim.exitConfirmed = false if (failedClaim.workerId) { try { failedClaim.exitConfirmed = await adapter.release(failedClaim.workerId) } catch { failedClaim.exitConfirmed = false } } + else if (adapter.abort) { try { failedClaim.exitConfirmed = await adapter.abort({ task, claim: failedClaim, reason }) } catch { /* Retain unconfirmed preparation occupancy. */ } } await serial(async () => { const c = task.collaboration if (c.claim?.id !== token.id) return diff --git a/patches/ds-bot/files/src/host/project-resources.js b/patches/ds-bot/files/src/host/project-resources.js index 3541227b8..52255a12a 100644 --- a/patches/ds-bot/files/src/host/project-resources.js +++ b/patches/ds-bot/files/src/host/project-resources.js @@ -18,6 +18,8 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} const { state, config } = rt const leases = () => (state.projectResourceLeases ??= {}) const resources = () => config.projectResources ?? {} + const preparing = new Map() + let effectStore let recovered = false let tail = Promise.resolve() const serial = fn => { const result = tail.then(fn, fn); tail = result.then(() => {}, () => {}); return result } @@ -80,7 +82,110 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} } return pending } + const workerRecords = lease => [...Object.values(state.projectHarnessWorkers ?? {}), ...Object.values(state.projectNativeWorkers ?? {})].filter(row => row.leaseId === lease.id) + function unbound(lease) { + return !preparing.has(lease.id) && lease.preparation?.status !== 'active' && !lease.worker && !lease.processScope && !workerRecords(lease).length && !Object.values(state.projectAttempts?.[lease.attemptId]?.roleSessions ?? {}).some(row => row.resourceLease?.id === lease.id && row.botId) && !Object.values(state.bots ?? {}).some(row => row.resourceLease?.id === lease.id || row.ephemeral && row.attemptId === lease.attemptId && row.role === lease.role) + } + function retiredReservation(lease) { + const attempt = state.projectAttempts?.[lease.attemptId], owner = lease.reservationOwner + if (!attempt || attempt.projectId !== lease.projectId) return false + if (['returned', 'cancelled', 'accepted', 'failed'].includes(attempt.status) || attempt.supersededBy) return true + if (!owner) return false + const task = state.tasks?.[owner.taskId], claim = task?.collaboration?.claim + return !task || task.projectId !== lease.projectId || task.deliveryAttemptId !== lease.attemptId || !claim || claim.id !== owner.claimId || claim.generation !== owner.generation || task.collaboration.generation !== owner.generation || claim.status !== 'active' + } + async function beginPreparation(id, generation) { + const lease = requireLease(id, generation) + if (lease.status === 'released' || retiredReservation(lease) || preparing.has(id) || lease.preparation?.status === 'active') throw failure('Resource preparation is released, stale or already active') + const token = { leaseId: id, generation, id: randomUUID() }, previous = lease.preparation + preparing.set(id, token.id) + try { + lease.preparation = { ...token, status: 'active', processScope: await captureProcessScope(), at: Date.now() } + if (rt.projectEffects) { + effectHandlers() + const intent = rt.projectEffects.stage({ kind: 'resource.preparation', key: `${id}:${token.id}`, projectId: lease.projectId, payload: token }) + try { await rt.projectEffects.persist(intent) } catch (error) { rt.projectEffects.discardUncommitted(intent); throw error } + } else await rt.save() + if (lease.status === 'released' || retiredReservation(lease)) throw failure('Resource preparation owner changed before commands') + return token + } catch (error) { + preparing.delete(id) + if (previous === undefined) delete lease.preparation; else lease.preparation = previous + throw error + } + } + async function endPreparation(token) { + const lease = requireLease(token.leaseId, token.generation) + if (preparing.get(lease.id) !== token.id || lease.preparation?.id !== token.id) throw failure('Preparation completion is stale') + lease.preparation.status = 'settled'; lease.preparation.finishedAt = Date.now() + try { await rt.save() } catch (error) { lease.preparation.status = 'active'; throw error } + preparing.delete(lease.id) + } + async function abortReservation(id, generation, reason) { + const lease = requireLease(id, generation) + if (lease.status === 'released') return true + if (preparing.has(id) || lease.preparation?.status === 'active') return false + const records = workerRecords(lease) + let changedSession, beforeSession + const before = structuredClone(lease), worker = records.length === 1 && records[0], beforeWorker = worker && structuredClone(worker) + if (!unbound(lease)) { + const row = records.length === 1 && records[0] + // Only the new bind-before-spawn protocol proves that registration never crossed spawn. + if (!row || row.spawnPhase !== 'registered' || row.status !== 'prepared' || row.executionStartedAt != null || lease.worker?.kind !== 'sdk' || lease.worker.handle !== row.id || row.generation !== generation) return false + const proof = { kind: 'sdk-worker-not-started', workerId: row.id, generation, at: Date.now() } + row.spawnPhase = 'aborted'; row.status = 'released'; row.releasedAt = proof.at; row.releaseProof = proof + lease.status = 'released'; lease.releasedAt = proof.at; lease.releaseProof = proof + const attempt = state.projectAttempts?.[row.attemptId], session = attempt?.roleSessions?.[row.role] + if (session?.botId === row.id) { changedSession = session; beforeSession = structuredClone(session); session.status = 'released'; session.releasedAt = proof.at } + } else { + lease.status = 'released'; lease.releasedAt = Date.now(); lease.releaseProof = { kind: 'unbound-reservation-release', at: lease.releasedAt } + } + lease.reclamation = { reason: String(reason).slice(0, 400), at: Date.now(), owner: structuredClone(lease.reservationOwner ?? null) } + let intent + try { + if (rt.projectEffects) { + effectHandlers() + intent = rt.projectEffects.stage({ kind: 'resource.reclaim', key: `${id}:${generation}`, projectId: lease.projectId, payload: { leaseId: id, generation } }) + lease.reclamation.effect = intent; await rt.projectEffects.persist(intent) + } else await rt.save() + } catch (error) { + if (!intent || rt.projectEffects.discardUncommitted(intent)) { + for (const key of Object.keys(lease)) delete lease[key] + Object.assign(lease, before) + if (worker) { for (const key of Object.keys(worker)) delete worker[key]; Object.assign(worker, beforeWorker) } + if (changedSession) { for (const key of Object.keys(changedSession)) delete changedSession[key]; Object.assign(changedSession, beforeSession) } + } + throw error + } + rt.projectScheduler?.wake() + return true + } + async function fenceSpawn(id, generation, workerId) { + const lease = requireLease(id, generation), row = state.projectHarnessWorkers?.[workerId] + if (lease.status !== 'running' || !row || row.spawnPhase !== 'registered' || lease.worker?.handle !== workerId || row.leaseId !== id || row.generation !== generation || retiredReservation(lease)) throw failure('Resource startup reservation is released, stale or retired') + row.spawnPhase = 'starting' + try { + if (rt.projectEffects) { + effectHandlers() + const token = rt.projectEffects.stage({ kind: 'resource.spawn', key: `${id}:${generation}:${workerId}`, projectId: lease.projectId, payload: { leaseId: id, generation, workerId } }) + row.spawnEffect = token + try { await rt.projectEffects.persist(token) } catch (error) { if (rt.projectEffects.discardUncommitted(token)) row.spawnPhase = 'registered'; throw error } + } else await rt.save() + } catch (error) { if (row.spawnPhase === 'starting') row.spawnPhase = 'registered'; throw error } // The caller has not crossed external spawn when the fence itself throws. + requireBinding(lease.projectId, lease) + if (lease.status !== 'running' || row.spawnPhase !== 'starting' || retiredReservation(lease)) throw failure('Resource startup changed before external spawn') + } + function effectHandlers() { + if (!rt.projectEffects || rt.projectEffects === effectStore) return + rt.projectEffects?.register('resource.spawn', { run: row => ({ workerId: row.payload.workerId, status: 'spawn-fenced' }) }, { replace: true }) + rt.projectEffects.register('resource.preparation', { run: row => ({ leaseId: row.payload.leaseId, status: 'preparation-fenced' }) }, { replace: true }) + rt.projectEffects.register('resource.reclaim', { run: row => ({ leaseId: row.payload.leaseId, status: 'reclaimed-before-spawn' }) }, { replace: true }) + effectStore = rt.projectEffects + } + effectHandlers() async function reserve(request) { + const owner = request.reservationOwner + if (owner && (typeof owner.taskId !== 'string' || typeof owner.claimId !== 'string' || !Number.isSafeInteger(owner.generation) || owner.generation < 1 || owner.sessionId !== undefined && owner.sessionId !== request.sessionKey)) throw failure('Invalid reservation ownership') const existing = Object.values(leases()).find(row => row.sessionKey === request.sessionKey) if (existing) { if (existing.projectId !== request.projectId || existing.attemptId !== request.attemptId || existing.role !== request.role || existing.harness !== request.harness || existing.status === 'released') throw failure('Execution resource Session key is already bound or released') @@ -89,19 +194,28 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} const selected = select(request) const lease = { id: `resource-lease-${randomUUID()}`, generation: 1, projectId: request.projectId, attemptId: request.attemptId, role: request.role, sessionKey: request.sessionKey, purpose: request.purpose ?? 'execution', resourceId: selected.id, harness: request.harness, principalId: rt.readProject(request.projectId).ownerUserId, runtimeUser: selected.item.type === 'docker' ? selected.item.user : `uid:${process.getuid?.() ?? 'unknown'}`, - bindingHash: bindingHash(selected.item), status: 'reserved', createdAt: Date.now() } + ...(request.reservationOwner ? { reservationOwner: structuredClone(request.reservationOwner) } : {}), bindingHash: bindingHash(selected.item), status: 'reserved', createdAt: Date.now() } setEntry(leases(), lease.id, lease) - await rt.save() + try { await rt.save() } catch (error) { + // No caller received this reservation, so no external startup was authorized. + lease.status = 'released'; lease.releasedAt = Date.now(); lease.releaseProof = { kind: 'unbound-reservation-release', at: lease.releasedAt }; lease.reclamation = { reason: 'Initial reservation save failed before admission returned', at: lease.releasedAt } + throw error + } return structuredClone(lease) } async function reserveRoles(projectId, attemptId, definitions) { plan(projectId, definitions) const reserved = {} - for (const role of definitions?.roles ?? []) { - if (role.role === 'chief' || !role.execution?.resourceIds?.length) continue - reserved[role.role] = await reserve({ projectId, attemptId, role: role.role, sessionKey: `${attemptId}:${role.role}`, ...role.execution }) + try { + for (const role of definitions?.roles ?? []) { + if (role.role === 'chief' || !role.execution?.resourceIds?.length) continue + reserved[role.role] = await reserve({ projectId, attemptId, role: role.role, sessionKey: `${attemptId}:${role.role}`, ...role.execution }) + } + return reserved + } catch (error) { + for (const lease of Object.values(reserved)) { try { await abortReservation(lease.id, lease.generation, 'Batch admission failed before returning reservations') } catch { /* Reconciliation retains any uncommitted cleanup. */ } } + throw error } - return reserved } function requireLease(id, generation) { const lease = ownEntry(leases(), id) @@ -111,8 +225,10 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} async function markStarted(id, generation, worker) { const lease = requireLease(id, generation) if (lease.status === 'released') throw failure('Execution resource lease was released') + if (retiredReservation(lease)) throw failure('Execution reservation owner is no longer current') if (lease.worker && JSON.stringify(lease.worker) !== JSON.stringify(worker)) throw failure('Execution resource worker is already bound') const scope = setting(lease.projectId, lease.resourceId).type === 'local' ? await captureProcessScope() : null + if (lease.status === 'released' || retiredReservation(lease)) throw failure('Execution reservation changed before binding') lease.worker = structuredClone(worker); lease.status = 'running' if (scope) lease.processScope = scope await rt.save() @@ -126,7 +242,7 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} const supervisor = worker?.kind === 'native' && supervisorProofMatches(scope, confirmation?.proof) && await priorExit(scope) if (requireLease(id, generation) !== lease || lease.worker !== worker || lease.processScope !== scope) throw failure('Execution resource binding changed during exit inspection') if (lease.status === 'released') return - const confirmed = supervisor || !worker || worker.kind === 'native' && !ownEntry(state.bots ?? {}, worker.botId) && native?.leaseId === lease.id && nativeExitConfirmed(state, native) || worker.kind !== 'native' && typeof worker.handle === 'string' && worker.handle.length > 0 && confirmation?.handle === worker.handle && confirmation?.exited === true + const confirmed = supervisor || !worker && unbound(lease) || worker.kind === 'native' && !ownEntry(state.bots ?? {}, worker.botId) && native?.leaseId === lease.id && nativeExitConfirmed(state, native) || worker.kind !== 'native' && typeof worker.handle === 'string' && worker.handle.length > 0 && confirmation?.handle === worker.handle && confirmation?.exited === true lease.status = confirmed ? 'released' : 'stopping' if (confirmed) { lease.releasedAt = Date.now(); lease.releaseProof ??= confirmation?.proof ?? native?.releaseProof ?? { kind: worker ? 'host-confirmed-worker-exit' : 'unbound-reservation-release', at: lease.releasedAt } } await rt.save() @@ -166,7 +282,16 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} if (lease.processScope?.hostPid === process.pid) { if (lease.worker?.kind === 'native' && rt.projectDelivery?.nativeSessions?.currentHost(lease.worker.botId) || lease.worker?.kind === 'sdk' && rt.projectHarness?.hasController(lease.worker.handle) || lease.worker?.kind === 'native-turn' && rt.projectAgents?.hasExchange(lease.worker.handle)) continue } - if (!lease.worker) { if (lease.status === 'unreachable') await release(lease.id, lease.generation); continue } + if (lease.preparation?.status === 'active') { + if (preparing.has(lease.id)) continue + if (setting(lease.projectId, lease.resourceId).type !== 'local' || !await priorExit(lease.preparation.processScope)) continue + lease.preparation.status = 'settled'; lease.preparation.recoveryProof = { kind: 'systemd-prior-invocation-exited', previousInvocationId: lease.preparation.processScope.invocationId, at: Date.now() }; await rt.save() + } + if (workerRecords(lease).some(row => row.spawnPhase === 'registered') && retiredReservation(lease)) { await serial(() => abortReservation(lease.id, lease.generation, 'Registered worker lost its startup owner before spawn')); continue } + if (!lease.worker) { + if (unbound(lease) && (lease.status === 'unreachable' || retiredReservation(lease))) await serial(() => unbound(lease) && active(lease) && (lease.status === 'unreachable' || retiredReservation(lease)) ? abortReservation(lease.id, lease.generation, 'No current reservation owner remains before binding') : undefined) + continue + } if (lease.worker.kind === 'native' && !ownEntry(state.bots ?? {}, lease.worker.botId)) await release(lease.id, lease.generation) else if (lease.worker.kind !== 'native' && lease.status !== 'unreachable') { lease.status = 'unreachable'; changed = true } } @@ -221,7 +346,7 @@ export function createProjectResources(rt, { priorExit = priorScopeExited } = {} return structuredClone(row) } return { setting, requireBinding, uploadFile, downloadFile, list, checkPool, plan, command, startProcess, probe, occupied, reconcile, recover, - reserveRoles: (...args) => serial(() => reserveRoles(...args)), + beginPreparation: (...args) => serial(() => beginPreparation(...args)), endPreparation: (...args) => serial(() => endPreparation(...args)), abortReservation: (...args) => serial(() => abortReservation(...args)), fenceSpawn: (...args) => serial(() => fenceSpawn(...args)), reserveRoles: (...args) => serial(() => reserveRoles(...args)), reserve: request => serial(() => reserve(request)), markStarted: (...args) => serial(() => markStarted(...args)), release: (...args) => serial(() => release(...args)) } } export function install(rt) { rt.projectResources = createProjectResources(rt) } diff --git a/patches/ds-bot/files/src/host/project-service-views.js b/patches/ds-bot/files/src/host/project-service-views.js index 2ce5d1b27..50350dcc9 100644 --- a/patches/ds-bot/files/src/host/project-service-views.js +++ b/patches/ds-bot/files/src/host/project-service-views.js @@ -83,7 +83,7 @@ export function executionMetadataView(state, row) { const lease = row.leaseId && state.projectResourceLeases?.[row.leaseId] const exit = native ? nativeExitConfirmed(state, row) : confirmedWorkerExit(state, row) return { ...pick(row, ['id', 'role', 'resourceId', 'generation', 'sessionKey', 'logicalSessionId', 'createdAt', 'executionStartedAt', 'releasedAt']), - harness: native ? 'flybot' : row.harness ?? null, status: row.status, + harness: native ? 'flybot' : row.harness ?? null, status: row.status, startupPhase: native ? null : row.spawnPhase ?? null, policyRole: session?.policyRole ?? row.policyRole ?? row.role ?? null, definitionRevision: session?.definitionRevision ?? null, activationEffectId: native ? row.activationEffect?.id ?? null : null, diff --git a/patches/ds-bot/files/src/host/project-task-sessions.js b/patches/ds-bot/files/src/host/project-task-sessions.js index 5de63cc19..4d5211f20 100644 --- a/patches/ds-bot/files/src/host/project-task-sessions.js +++ b/patches/ds-bot/files/src/host/project-task-sessions.js @@ -115,6 +115,10 @@ export function confirmedWorkerExit(state, worker) { lease.worker?.kind === 'sdk' && lease.worker.handle === worker.id && lease.releaseProof?.kind === proof.kind && lease.releaseProof.workerId === worker.id && lease.releaseProof.generation === worker.generation && lease.releaseProof.at === proof.at) } const proof = worker.releaseProof + if (proof?.kind === 'sdk-worker-not-started') { + const lease = ownEntry(state.projectResourceLeases ?? {}, worker.leaseId) + return worker.spawnPhase === 'aborted' && worker.executionStartedAt == null && proof.workerId === worker.id && proof.generation === worker.generation && lease?.status === 'released' && lease.generation === worker.generation && lease.worker?.handle === worker.id && lease.projectId === worker.projectId && lease.attemptId === worker.attemptId && lease.resourceId === worker.resourceId && lease.role === worker.role && lease.releaseProof?.kind === proof.kind && lease.releaseProof.workerId === worker.id && lease.releaseProof.at === proof.at + } if (proof?.kind === 'sdk-stop-ack-and-process-exit') return proof.generation === worker.generation if (proof?.kind !== 'systemd-prior-invocation-exited' || !/^[a-f0-9]{32}$/.test(proof.previousInvocationId ?? '')) return false const lease = ownEntry(state.projectResourceLeases ?? {}, worker.leaseId) diff --git a/patches/ds-bot/files/test/project-delivery.test.js b/patches/ds-bot/files/test/project-delivery.test.js index 2ecae88b5..130d9fed6 100644 --- a/patches/ds-bot/files/test/project-delivery.test.js +++ b/patches/ds-bot/files/test/project-delivery.test.js @@ -62,6 +62,16 @@ async function setup(t, testCommand = [process.execPath, '--test', 'code.test.mj return { rt, home, task, start, attempt, gate, repository, published, publisher, workflow, directory } } +// Direct preparation probes emulate the pool's current claim before using a resource. +function prepareRole(rt, task, claim, definition) { + if (definition.execution?.resourceIds?.length && task.collaboration?.claim?.id !== claim.id) { + const c = task.collaboration + c.generation = claim.generation; c.attemptId = claim.attemptId; c.botId = definition.id; c.status = 'claimed' + c.claim = { ...claim, taskId: task.id, projectId: task.projectId, botId: definition.id, role: definition.role, policyRole: definition.policyRole ?? definition.role, status: 'active', createdAt: Date.now() } + } + return rt.projectDelivery.preparePoolRole(task, claim, definition) +} + async function implement(fixture) { writeFileSync(join(fixture.attempt().workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await fixture.gate('developer') @@ -621,10 +631,10 @@ test('a transferred unsealed implementation is a labelled draft reference, not a const attempt = f.attempt(), task = f.rt.state.tasks[f.task.id] const defs = f.rt.projectAgents.deliveryDefinitions('p1').roles const dev = defs.find(row => row.role === 'developer'), tester = defs.find(row => row.role === 'tester') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'draft-dev', attemptId: attempt.id, generation: 1 }, dev) + await prepareRole(f.rt, task, { id: 'draft-dev', attemptId: attempt.id, generation: 1 }, dev) const path = attempt.workspaces.developer.path writeFileSync(join(path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'draft-test', attemptId: attempt.id, generation: 2 }, tester) + await prepareRole(f.rt, task, { id: 'draft-test', attemptId: attempt.id, generation: 2 }, tester) const facts = JSON.parse(attempt.roleSessions.tester.context.text.split('\n')[1].split('\n')[0]).projectFacts assert.equal(facts.candidateSha, null) assert.equal(facts.workingDraft.kind, 'unverified-workspace-reference') @@ -638,10 +648,10 @@ test('restart preserves a tested pool checkpoint waiting for Chief only when eve const f = await setup(t, undefined, { pool: true }); f.rt.projectPools.connect({ validate: () => ({ ok: true }) }); await f.start('checkpoint-' + variant) const a = f.attempt(), task = f.rt.state.tasks[f.task.id] const roles = f.rt.projectAgents.deliveryDefinitions('p1').roles - await f.rt.projectDelivery.preparePoolRole(task, { id: 'dev', attemptId: a.id, generation: 1 }, roles.find(r => r.role === 'developer')) + await prepareRole(f.rt, task, { id: 'dev', attemptId: a.id, generation: 1 }, roles.find(r => r.role === 'developer')) writeFileSync(join(a.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: a.id }, role: 'developer', outputs: {} }) - await f.rt.projectDelivery.preparePoolRole(task, { id: 'test', attemptId: a.id, generation: 2 }, roles.find(r => r.role === 'tester')) + await prepareRole(f.rt, task, { id: 'test', attemptId: a.id, generation: 2 }, roles.find(r => r.role === 'tester')) await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: a.id }, role: 'tester', outputs: {} }) task.collaboration.status = 'blocked' task.collaboration.claim = { botId: 'p1:chief', status: variant === 'active-claim' ? 'active' : 'waiting-human', workerId: 'chief-worker', chiefRunId: 'chief-run' } @@ -668,7 +678,7 @@ test('published pool rework retains its PR binding and seals a descendant of the await f.start('published-rework') const task = f.rt.state.tasks[f.task.id], first = f.attempt() const developer = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'first-dev', attemptId: first.id, generation: 1 }, developer) + await prepareRole(f.rt, task, { id: 'first-dev', attemptId: first.id, generation: 1 }, developer) writeFileSync(join(first.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: first.id }, role: 'developer', outputs: {} }) const original = first.candidateSha @@ -682,7 +692,7 @@ test('published pool rework retains its PR binding and seals a descendant of the const id = await f.rt.projectDelivery.reworkPool({ task, claim: { id: 'chief-return', attemptId: first.id, botId: 'p1:chief' }, reason: 'Maintain the existing PR after its base advanced' }) const next = f.rt.state.projectAttempts[id] assert.deepEqual(next.publicationBinding, { number: 48, repository: 'example/demo', branch: first.branch, previousSha: original, upstreamSha: first.baseSha, upstreamRef: 'main' }) - await f.rt.projectDelivery.preparePoolRole(task, { id: 'next-dev', attemptId: id, generation: 2 }, developer) + await prepareRole(f.rt, task, { id: 'next-dev', attemptId: id, generation: 2 }, developer) writeFileSync(join(next.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => Number(a)+Number(b); // intermediate\n') git(next.workspaces.developer.path, ['-c', 'user.name=Fixture', '-c', 'user.email=fixture@localhost', 'commit', '-am', 'Private intermediate work']) writeFileSync(join(next.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => Number(a)+Number(b);\n') @@ -716,7 +726,7 @@ test('published rework pins the actual target and seals both published and resol f.rt.projectPools.connect({ validate: () => ({ ok: true }) }); await f.start('upstream-rework') const task = f.rt.state.tasks[f.task.id], first = f.attempt() const developer = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'dev-before', attemptId: first.id, generation: 1 }, developer) + await prepareRole(f.rt, task, { id: 'dev-before', attemptId: first.id, generation: 1 }, developer) writeFileSync(join(first.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: first.id }, role: 'developer', outputs: {} }) const seed = first.candidateSha @@ -737,7 +747,7 @@ test('published rework pins the actual target and seals both published and resol const id = await f.rt.projectDelivery.reworkPool({ task, claim: { id: 'upstream-return', attemptId: first.id, botId: 'p1:chief' }, reason: 'Integrate current backend without losing published history' }) const next = f.rt.state.projectAttempts[id] assert.equal(next.publicationBinding.upstreamSha, upstream) - await f.rt.projectDelivery.preparePoolRole(task, { id: 'dev-after', attemptId: id, generation: 2 }, developer) + await prepareRole(f.rt, task, { id: 'dev-after', attemptId: id, generation: 2 }, developer) const path = next.workspaces.developer.path await assert.rejects(f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: id }, role: 'developer', outputs: {} }), /merge the pinned upstream/) assert.equal(git(path, ['rev-parse', 'HEAD']), seed) @@ -755,7 +765,7 @@ test('published rework pins the actual target and seals both published and resol assert.equal(first.candidateSha, seed) assert.equal(next.evidence.upstreamIntegration.sha, upstream) const tester = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'tester') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'test-after', attemptId: id, generation: 3 }, tester) + await prepareRole(f.rt, task, { id: 'test-after', attemptId: id, generation: 3 }, tester) await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: id }, role: 'tester', outputs: {} }) assert.equal(next.evidence.tests.passed, true) }) @@ -795,12 +805,12 @@ test('context admission checkpoints require host rejection and no external execu test('failed context admission preserves an existing unsealed developer workspace',async t=>{ const f=await setup(t,undefined,{pool:true}); f.rt.projectPools.connect({validate:()=>({ok:true})}); await f.start('context-retry') const a=f.attempt(),task=f.rt.state.tasks[f.task.id],dev=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r=>r.role==='developer') - await f.rt.projectDelivery.preparePoolRole(task,{id:'first',attemptId:a.id,generation:1},dev) + await prepareRole(f.rt, task,{id:'first',attemptId:a.id,generation:1},dev) const path=a.workspaces.developer.path writeFileSync(join(path,'code.mjs'),'export const preserved = true;\n') a.seedSha=a.baseSha a.feedback='x'.repeat(49000) // Required current Task input cannot be summarized away. - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task,{id:'second',attemptId:a.id,generation:2},dev),/exceeds/) + await assert.rejects(prepareRole(f.rt, task,{id:'second',attemptId:a.id,generation:2},dev),/exceeds/) assert.equal(readFileSync(join(path,'code.mjs'),'utf8'),'export const preserved = true;\n') assert.equal(a.preparationFailures.at(-1).releasedBeforeExternalStart,false) assert.equal(a.roleSessions.developer.poolClaimId,'first') @@ -814,7 +824,7 @@ test('context failure releases an actual unbound reservation before registering await f.start('unbound-context-failure') const a=f.attempt(),task=f.rt.state.tasks[f.task.id],dev=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r=>r.role==='developer') dev.instructions='x'.repeat(49000) // Required persona cannot be summarized away. - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task,{id:'rejected',attemptId:a.id,generation:1},dev),/exceeds/) + await assert.rejects(prepareRole(f.rt, task,{id:'rejected',attemptId:a.id,generation:1},dev),/exceeds/) const lease=Object.values(f.rt.state.projectResourceLeases).find(x=>x.attemptId===a.id) assert.equal(lease.status,'released') assert.equal(lease.releaseProof.kind,'unbound-reservation-release') @@ -831,7 +841,7 @@ test('a preparation-only successor retains its inherited sealed seed across anot await f.start('retain-seed') const first = f.attempt(), task = f.rt.state.tasks[f.task.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, {id: 'original-dev', attemptId: first.id, generation: 1}, dev) + await prepareRole(f.rt, task, {id: 'original-dev', attemptId: first.id, generation: 1}, dev) writeFileSync(join(first.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({task, claim: {attemptId: first.id}, role: 'developer', outputs: {implementation: 'Fixed'}}) const seed = first.candidateSha @@ -842,7 +852,7 @@ test('a preparation-only successor retains its inherited sealed seed across anot const finalId = await f.rt.projectDelivery.reworkPool({task, claim: {attemptId: nextId, id: 'preparation-retry', botId: 'chief'}, reason: 'Context preparation repaired', force: true}) const final = f.rt.state.projectAttempts[finalId] assert.equal(final.seedSha, seed) - await f.rt.projectDelivery.preparePoolRole(task, {id: 'recovered-dev', attemptId: finalId, generation: 3}, dev) + await prepareRole(f.rt, task, {id: 'recovered-dev', attemptId: finalId, generation: 3}, dev) assert.match(readFileSync(join(final.workspaces.developer.path, 'code.mjs'), 'utf8'), /a\+b/) assert.equal(next.status, 'interrupted', 'Original failed preparation history remains') }) @@ -854,25 +864,25 @@ test('a rework receiver gets hash-verified bounded failed-test facts instead of await f.start('failure-feedback') const a = f.attempt(), task = f.rt.state.tasks[f.task.id], roles = f.rt.projectAgents.deliveryDefinitions('p1').roles const dev = roles.find(row => row.role === 'developer'), tester = roles.find(row => row.role === 'tester') - await f.rt.projectDelivery.preparePoolRole(task, {id: 'dev', attemptId: a.id, generation: 1}, dev) + await prepareRole(f.rt, task, {id: 'dev', attemptId: a.id, generation: 1}, dev) writeFileSync(join(a.workspaces.developer.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') await f.rt.projectDelivery.completePoolRole({task, claim: {attemptId: a.id}, role: 'developer', outputs: {}}) - await f.rt.projectDelivery.preparePoolRole(task, {id: 'test', attemptId: a.id, generation: 2}, tester) + await prepareRole(f.rt, task, {id: 'test', attemptId: a.id, generation: 2}, tester) await assert.rejects(f.rt.projectDelivery.completePoolRole({task, claim: {attemptId: a.id}, role: 'tester', outputs: {tests: 'Run host tests'}}), /tests failed/) const id = await f.rt.projectDelivery.reworkPool({task, claim: {attemptId: a.id, id: 'failed-test-return', botId: 'chief'}, reason: 'Fix actual failure', force: true}) const next = f.rt.state.projectAttempts[id] - await f.rt.projectDelivery.preparePoolRole(task, {id: 'repair', attemptId: id, generation: 3}, dev) + await prepareRole(f.rt, task, {id: 'repair', attemptId: id, generation: 3}, dev) const context = next.roleSessions.developer.context.text assert.ok(context.includes('popup layer regression')) assert.ok(context.includes(a.evidence.tests.sha256)) assert.ok(context.length < 48000) const deepId = await f.rt.projectDelivery.reworkPool({task, claim: {attemptId: id, id: 'preparation-only-return', botId: 'chief'}, reason: 'Preserve failure across preparation retry', force: true}) - await f.rt.projectDelivery.preparePoolRole(task, {id: 'deep-repair', attemptId: deepId, generation: 4}, dev) + await prepareRole(f.rt, task, {id: 'deep-repair', attemptId: deepId, generation: 4}, dev) assert.ok(f.rt.state.projectAttempts[deepId].roleSessions.developer.context.text.includes('popup layer regression')) writeFileSync(a.evidence.tests.log, 'tampered failure') - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, {id: 'stale-log', attemptId: deepId, generation: 5}, dev), /log changed/) + await assert.rejects(prepareRole(f.rt, task, {id: 'stale-log', attemptId: deepId, generation: 5}, dev), /log changed/) a.projectId = 'another-project' - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, {id: 'foreign-log', attemptId: deepId, generation: 6}, dev), /another Task or Project/) + await assert.rejects(prepareRole(f.rt, task, {id: 'foreign-log', attemptId: deepId, generation: 6}, dev), /another Task or Project/) }) test('workspace allocation failure releases an actual reservation before any external execution', async t => { @@ -888,7 +898,7 @@ test('workspace allocation failure releases an actual reservation before any ext // A real filesystem obstacle after reservation and before git worktree creation. writeFileSync(join(f.directory, 'delivery'), 'Not a directory') const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'failed-allocation', attemptId: a.id, generation: 1 }, dev), /ENOTDIR|EEXIST/) + await assert.rejects(prepareRole(f.rt, task, { id: 'failed-allocation', attemptId: a.id, generation: 1 }, dev), /ENOTDIR|EEXIST/) const lease = Object.values(f.rt.state.projectResourceLeases).find(row => row.attemptId === a.id) assert.equal(lease.status, 'released') assert.equal(lease.releaseProof.kind, 'unbound-reservation-release') @@ -907,7 +917,7 @@ test('every pool receiver is host-prepared in its own checkout before registerin const a = f.attempt(), task = f.rt.state.tasks[f.task.id] for (const [index, role] of ['developer', 'tester', 'reviewer'].entries()) { const definition = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === role) - const node = await f.rt.projectDelivery.preparePoolRole(task, { id: `prepare-${role}`, attemptId: a.id, generation: index + 1 }, definition) + const node = await prepareRole(f.rt, task, { id: `prepare-${role}`, attemptId: a.id, generation: index + 1 }, definition) assert.equal(readFileSync(join(node.executionWorkDir, 'node_modules/ready'), 'utf8'), node.executionWorkDir) assert.equal(a.workspacePreparations.at(-1).workspaceId, a.workspaces[role].id) assert.equal(a.roleSessions[role].status, 'prepared') @@ -917,7 +927,7 @@ test('every pool receiver is host-prepared in its own checkout before registerin settings.workspacePrepareCommand = [process.execPath, '-e', 'process.exit(37)'] const previous = structuredClone(a.roleSessions.reviewer) const reviewer = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'reviewer') - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'prepare-failed', attemptId: a.id, generation: 9 }, reviewer), error => error.code === 37) + await assert.rejects(prepareRole(f.rt, task, { id: 'prepare-failed', attemptId: a.id, generation: 9 }, reviewer), error => error.code === 37) assert.deepEqual(a.roleSessions.reviewer, previous) assert.equal(a.workspacePreparations.length, 3) assert.equal(a.preparationFailures.at(-1).role, 'reviewer') @@ -949,14 +959,14 @@ test('bound-resource preparation executes before SDK registration and failure re await f.start('resource-preparation') const a = f.attempt(), task = f.rt.state.tasks[f.task.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'prepared-resource', attemptId: a.id, generation: 1 }, dev) + await prepareRole(f.rt, task, { id: 'prepared-resource', attemptId: a.id, generation: 1 }, dev) assert.equal(a.workspacePreparations.at(-1).resourceId, 'one') const original = structuredClone(a.roleSessions.developer) const first = a.resourceLeases.developer await f.rt.projectResources.release(first.id, first.generation) const preparationCommand = settings.workspacePrepareCommand settings.workspacePrepareCommand = [process.execPath, '-e', 'process.exit(37)'] - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'resource-command-failed', attemptId: a.id, generation: 2 }, dev), error => error.code === 37) + await assert.rejects(prepareRole(f.rt, task, { id: 'resource-command-failed', attemptId: a.id, generation: 2 }, dev), error => error.code === 37) assert.equal(f.rt.state.projectResourceLeases[a.resourceLeases.developer.id].status, 'released') assert.equal(a.preparationFailures.at(-1).releasedBeforeExternalStart, true) settings.workspacePrepareCommand = preparationCommand @@ -967,7 +977,8 @@ test('bound-resource preparation executes before SDK registration and failure re await command(...args) throw new Error('Connection lost after external registration') } - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'unknown-preparation', attemptId: a.id, generation: 2 }, dev), /Connection lost/) + const unknownClaim = { id: 'unknown-preparation', attemptId: a.id, generation: 2 } + await assert.rejects(prepareRole(f.rt, task, unknownClaim, dev), /Connection lost/) const held = f.rt.state.projectResourceLeases[a.resourceLeases.developer.id] assert.equal(held.status, 'running') assert.equal(held.worker.handle, 'unknown-worker') @@ -982,7 +993,7 @@ test('unstarted bound delivery refreshes a verified descendant target without re const a=f.attempt(),task=f.rt.state.tasks[f.task.id],dev=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r=>r.role==='developer'),base=a.baseSha a.seedSha=base;a.publicationBinding={number:48,repository:'example/demo',branch:'original-delivery',previousSha:base,upstreamSha:base,upstreamRef:'main'} const originalInstructions=dev.instructions;dev.instructions='x'.repeat(49000) - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task,{id:'before',attemptId:a.id,generation:1},dev),/exceeds/) + await assert.rejects(prepareRole(f.rt, task,{id:'before',attemptId:a.id,generation:1},dev),/exceeds/) dev.instructions=originalInstructions const path=a.workspaces.developer.path;writeFileSync(join(path,'code.mjs'),'export const preservedDraft = true;\n') writeFileSync(join(f.repository,'backend.mjs'),'export const backend = 2;\n');git(f.repository,['add','backend.mjs']);git(f.repository,['commit','-m','Advance trusted target']) @@ -1005,7 +1016,7 @@ test('unstarted bound delivery refreshes a verified descendant target without re if(variant==='missing')f.publisher.read=async()=>({sha:base,state:'open',baseRef:'main',baseSha:'1'.repeat(40)}) if(variant==='moved-head')f.publisher.read=async()=>({sha:target,state:'open',baseRef:'main',baseSha:target}) if(variant==='moved-ref')f.publisher.read=async()=>({sha:base,state:'open',baseRef:'other',baseSha:target}) - const prepare=()=>f.rt.projectDelivery.preparePoolRole(task,{id:'after',attemptId:a.id,generation:2},dev) + const prepare=()=>prepareRole(f.rt, task,{id:'after',attemptId:a.id,generation:2},dev) if(variant==='safe'){ await prepare();assert.equal(a.publicationBinding.upstreamSha,target);assert.equal(a.publicationBindingHistory.length,1) assert.deepEqual(a.publicationBindingHistory[0].previous,old);assert.deepEqual(a.publicationBindingHistory[0].current,a.publicationBinding) @@ -1097,7 +1108,7 @@ test('SDK Task rework preserves an unsealed same-base draft and old cleanup cann await f.rt.projectEndpoint('project-agent-definition',{projectId:'p1',role:'developer',execution:{harness:'codex',resourceIds:['one']}},alice) await f.start('retained-draft') const previous=f.attempt(),task=f.rt.state.tasks[f.task.id],dev=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r=>r.role==='developer') - await f.rt.projectDelivery.preparePoolRole(task,{id:'first',attemptId:previous.id,generation:1},dev) + await prepareRole(f.rt, task,{id:'first',attemptId:previous.id,generation:1},dev) const cwd=previous.workspaces.developer.path writeFileSync(join(cwd,'code.mjs'),'export const preservedDraft = true;\n') const lease=previous.resourceLeases.developer @@ -1105,7 +1116,7 @@ test('SDK Task rework preserves an unsealed same-base draft and old cleanup cann previous.status='interrupted' const id=await f.rt.projectDelivery.reworkPool({task,claim:{id:'recover',attemptId:previous.id,botId:'p1:chief'},reason:'Resume confirmed idle draft',force:true}) const next=f.rt.state.projectAttempts[id] - await f.rt.projectDelivery.preparePoolRole(task,{id:'second',attemptId:id,generation:2},dev) + await prepareRole(f.rt, task,{id:'second',attemptId:id,generation:2},dev) assert.equal(next.workspaces.developer.path,cwd) assert.equal(next.workspaces.developer.retainedDraft,true) assert.equal(readFileSync(join(cwd,'code.mjs'),'utf8'),'export const preservedDraft = true;\n') @@ -1125,14 +1136,14 @@ test('a returned SDK draft reuses its Task workspace after exact supervisor exit await f.start('supervised-draft') const previous = f.attempt(), task = f.rt.state.tasks[f.task.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'old', attemptId: previous.id, generation: 1 }, dev) + await prepareRole(f.rt, task, { id: 'old', attemptId: previous.id, generation: 1 }, dev) const workspace = previous.workspaces.developer, lease = f.rt.state.projectResourceLeases[previous.resourceLeases.developer.id] writeFileSync(join(workspace.path, 'code.mjs'), 'export const add = (a,b) => a+b; // retained draft\n') const proof = { kind: 'systemd-prior-invocation-exited', previousInvocationId: 'a'.repeat(32), at: Date.now() } Object.assign(lease, { status: 'released', worker: { kind: 'sdk', handle: 'old-worker' }, processScope: { kind: 'systemd-control-group', killMode: 'control-group', invocationId: proof.previousInvocationId }, releaseProof: proof }) f.rt.state.projectHarnessWorkers = { 'old-worker': { id: 'old-worker', projectId: 'p1', taskId: task.id, attemptId: previous.id, role: 'developer', leaseId: lease.id, resourceId: 'one', generation: lease.generation, status: 'released', releaseProof: proof } } const id = await f.rt.projectDelivery.reworkPool({ task, claim: { id: 'return', attemptId: previous.id, botId: dev.id }, reason: 'Resume the retained draft after supervisor restart', force: true }) - await f.rt.projectDelivery.preparePoolRole(task, { id: 'next', attemptId: id, generation: 2 }, dev) + await prepareRole(f.rt, task, { id: 'next', attemptId: id, generation: 2 }, dev) const current = f.rt.state.projectAttempts[id].workspaces.developer assert.equal(current.path, workspace.path) assert.equal(current.reusesWorkspaceId, workspace.id) @@ -1164,7 +1175,7 @@ async function additionalIntegrationFixture(t, { conflict = false, merge = true attempt.seedSha=seed attempt.publicationBinding={number:7,repository:'example/demo',branch:'published-ui',previousSha:seed,upstreamSha:base,upstreamRef:'main'} const developer=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row=>row.role==='developer') - await f.rt.projectDelivery.preparePoolRole(task,{id:'integration-claim',attemptId:attempt.id,generation:1},developer) + await prepareRole(f.rt, task,{id:'integration-claim',attemptId:attempt.id,generation:1},developer) const path=attempt.workspaces.developer.path if(merge){if(conflict)assert.throws(()=>git(path,['merge','--no-commit','--no-ff',source]));else git(path,['merge','--no-commit','--no-ff',source])} return {...f,base,source,ref,seed,path,task,attempt,developer} @@ -1204,7 +1215,7 @@ test('manager explicitly pins integration input through service snapshot without assert.equal(replay.result.admission.id,approved.result.admission.id) assert.equal(f.attempt.integrationAdmissions.length,1) assert.equal(f.attempt.candidateSha,undefined) - const node=await f.rt.projectDelivery.preparePoolRole(f.task,{id:'resumed-integration',attemptId:f.attempt.id,generation:99},f.developer) + const node=await prepareRole(f.rt, f.task,{id:'resumed-integration',attemptId:f.attempt.id,generation:99},f.developer) assert.match(node.prompt,/do not git commit/) const text=f.attempt.roleSessions.developer.context.text assert.ok(text.includes(f.source)) @@ -1307,7 +1318,7 @@ test('an approved operator object reaches a resource clone through the existing const query=await f.rt.projectEndpoint('project-development',{version:'project-development/v1',kind:'query',name:'attempt.integration-state',projectId:'p1',payload:{attemptId:attempt.id}},alice) await f.rt.projectEndpoint('project-development',{version:'project-development/v1',kind:'command',name:'attempt.integration-bind',projectId:'p1',requestId:'remote-source',payload:{attemptId:attempt.id,inputs:[{ref,sha}],generation:query.result.generation,expectedWorkspace:null}},alice) const developer=f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row=>row.role==='developer') - await f.rt.projectDelivery.preparePoolRole(task,{id:'remote-claim',attemptId:attempt.id,generation:9},developer) + await prepareRole(f.rt, task,{id:'remote-claim',attemptId:attempt.id,generation:9},developer) const workspace=attempt.workspaces.developer assert.equal(workspace.kind,'resource-clone') assert.equal(git(workspace.path,['cat-file','-t',sha]),'commit') @@ -1551,7 +1562,7 @@ test('approved custom role capabilities preserve logical identities while sealin const attempt = f.attempt(), task = f.rt.state.tasks[f.task.id] const prepare = async (policyRole, generation) => { const role = names[policyRole], definition = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === role) - await f.rt.projectDelivery.preparePoolRole(task, { id: `custom-${role}`, attemptId: attempt.id, generation }, definition) + await prepareRole(f.rt, task, { id: `custom-${role}`, attemptId: attempt.id, generation }, definition) return role } const dev = await prepare('developer', 1) @@ -1560,7 +1571,7 @@ test('approved custom role capabilities preserve logical identities while sealin const backup = { ...structuredClone(f.rt.state.projectAgentDefinitions.p1.roles.find(row => row.role === 'implementer')), id: 'p1:backup', role: 'backup' } f.rt.state.projectAgentDefinitions.p1.roles.push(backup); ++f.rt.state.projectAgentDefinitions.p1.revision f.rt.config.projectRepositories['example/demo'].requiredRolePolicySources = { backup: ['MISSING.md'] } - await assert.rejects(f.rt.projectDelivery.preparePoolRole(task, { id: 'bad-backup', attemptId: attempt.id, generation: 2 }, f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'backup')), /Required role policy source is missing/) + await assert.rejects(prepareRole(f.rt, task, { id: 'bad-backup', attemptId: attempt.id, generation: 2 }, f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'backup')), /Required role policy source is missing/) assert.equal(attempt.roleAssignments.developer, 'implementer', 'Failed preparation cannot replace the actual artifact producer') assert.match(attempt.roleSessions[dev].policy.text, /Never invent passing tests/) writeFileSync(join(workspace.path, 'code.mjs'), 'export const add = (a,b) => a+b;\n') @@ -1593,12 +1604,12 @@ test('Task execution selects an associated repository and retains it after the P f.rt.state.projects.p1.repository = 'unprovisioned/changed-default' const attempt = f.rt.state.projectAttempts[started.attempt.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r => r.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(f.task, { id: 'repo-dev', attemptId: attempt.id, generation: 1 }, dev) + await prepareRole(f.rt, f.task, { id: 'repo-dev', attemptId: attempt.id, generation: 1 }, dev) assert.equal(attempt.repositoryBinding.repository, 'example/second') assert.equal(attempt.workspaces.developer.repository, second) const association = Object.values(f.rt.state.projectRepositories).find(row => row.repository === 'example/second') delete f.rt.state.projectRepositories[association.id] - await assert.rejects(f.rt.projectDelivery.preparePoolRole(f.task, { id: 'revoked-repo', attemptId: attempt.id, generation: 2 }, dev), /not currently associated/) + await assert.rejects(prepareRole(f.rt, f.task, { id: 'revoked-repo', attemptId: attempt.id, generation: 2 }, dev), /not currently associated/) assert.equal(attempt.requirement.repository, 'example/second') }) @@ -1611,7 +1622,7 @@ test('a repository-free decision Task prepares a scoped workspace and records ve assert.equal(started.attempt.status, 'running') const attempt = f.rt.state.projectAttempts[started.attempt.id] const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(r => r.role === 'developer') - const spec = await f.rt.projectDelivery.preparePoolRole(f.task, { id: 'decision-dev', attemptId: attempt.id, generation: 1 }, dev) + const spec = await prepareRole(f.rt, f.task, { id: 'decision-dev', attemptId: attempt.id, generation: 1 }, dev) assert.equal(attempt.baseSha, undefined) assert.equal(attempt.workspaces.developer.kind, 'decision-directory') assert.match(spec.prompt, /no repository/i) @@ -1659,7 +1670,7 @@ test('Release Task closes only after explicit durable authorization and current await f.start('release-task') const a = f.attempt() const dev = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'release-dev', attemptId: a.id, generation: 1 }, dev) + await prepareRole(f.rt, task, { id: 'release-dev', attemptId: a.id, generation: 1 }, dev) await f.rt.projectDelivery.completePoolRole({ task, claim: { attemptId: a.id }, role: 'developer', outputs: { releasePlan: 'Publish the explicitly authorized base commit' } }) const envelope = { version: 'project-development/v1', kind: 'command', projectId: 'p1', name: 'release.request', requestId: 'publish-release', payload: { attemptId: a.id, tag: 'v1.0.0', sha: a.baseSha, mode: 'publish' } } await assert.rejects(f.rt.projectEndpoint('project-development', envelope, bob), /denied/i) @@ -1712,14 +1723,15 @@ async function baselineResourceFixture(t, kind = 'local') { await f.rt.projectEndpoint('project-agent-definition', { projectId: 'p1', role: 'developer', execution: { harness: 'codex', resourceIds: ['baseline'] } }, alice) await f.start('baseline-recovery') const previous = f.attempt(), task = f.rt.state.tasks[f.task.id], definition = f.rt.projectAgents.deliveryDefinitions('p1').roles.find(row => row.role === 'developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'first-baseline', attemptId: previous.id, generation: 1 }, definition) + await prepareRole(f.rt, task, { id: 'first-baseline', attemptId: previous.id, generation: 1 }, definition) const workspace = previous.workspaces.developer, lease = previous.resourceLeases.developer await f.rt.projectResources.release(lease.id, lease.generation); previous.status = 'interrupted' + task.collaboration.claim.status = 'failed'; task.collaboration.status = 'blocked' writeFileSync(join(f.repository, 'host-repair.txt'), 'Updated trusted baseline\n'); git(f.repository, ['add', 'host-repair.txt']); git(f.repository, ['commit', '-m', 'Advance trusted base']) const latest = git(f.repository, ['rev-parse', 'HEAD']) const recover = async () => { const id = await f.rt.projectDelivery.reworkPool({ task, claim: { id: 'baseline-rework', attemptId: previous.id, botId: 'p1:chief' }, reason: 'Use repaired baseline', force: true }) - return { next: f.rt.state.projectAttempts[id], prepare: () => f.rt.projectDelivery.preparePoolRole(task, { id: 'next-baseline', attemptId: id, generation: 2 }, definition) } + return { next: f.rt.state.projectAttempts[id], prepare: () => prepareRole(f.rt, task, { id: 'next-baseline', attemptId: id, generation: 2 }, definition) } } return { ...f, previous, task, definition, workspace, latest, recover } } @@ -1795,7 +1807,7 @@ test('actual Chief route admits a pinned inspect input and Dev receives the PR o assert.equal(run.status, 'done', run.failure) assert.equal(attempt.integrationInputs[0].purpose, 'inspect') assert.equal(task.collaboration.botId, 'p1:developer') - await f.rt.projectDelivery.preparePoolRole(task, { id: 'inspect-after-chief', attemptId: attempt.id, generation: task.collaboration.generation }, f.definition) + await prepareRole(f.rt, task, { id: 'inspect-after-chief', attemptId: attempt.id, generation: task.collaboration.generation }, f.definition) assert.equal(git(attempt.workspaces.developer.path, ['cat-file', '-t', sha]), 'commit') assert.equal(git(attempt.workspaces.developer.path, ['rev-parse', 'HEAD']), attempt.baseSha) assert.equal(attempt.candidateSha, undefined) @@ -1808,6 +1820,6 @@ test('a newly created unsealed successor after failure pins current host HEAD ev const id = await f.rt.projectDelivery.reworkPool({ task: f.task, claim: { id: 'failed-route', attemptId: f.previous.id, botId: 'p1:chief' }, reason: 'Host repair is complete; route back', force: false }) const next = f.rt.state.projectAttempts[id] assert.notEqual(id, f.previous.id); assert.equal(next.baseSha, f.latest) - await f.rt.projectDelivery.preparePoolRole(f.task, { id: 'failed-route-prepare', attemptId: id, generation: 2 }, f.definition) + await prepareRole(f.rt, f.task, { id: 'failed-route-prepare', attemptId: id, generation: 2 }, f.definition) assert.equal(git(next.workspaces.developer.path, ['rev-parse', 'HEAD']), f.latest) }) diff --git a/patches/ds-bot/files/test/project-harness.test.js b/patches/ds-bot/files/test/project-harness.test.js index f864ef605..9f8c3265a 100644 --- a/patches/ds-bot/files/test/project-harness.test.js +++ b/patches/ds-bot/files/test/project-harness.test.js @@ -28,7 +28,7 @@ lines.on('line',line=>{const m=JSON.parse(line); });`) const session = { id: 's', projectId: 'p', taskId: 't', attemptId: 'a', role: 'developer', definition: { role: 'developer', instructions: 'Stable developer convention', skills: [] }, execution: { harness: 'codex' }, context: { text: 'Frozen criteria and role memory' }, resourceLease: { id: 'l', resourceId: 'r', generation: 1 } } const workspace = { id: 'w', projectId: 'p', attemptId: 'a', role: 'developer', path: cwd } - const rt = { home: root, state: { tasks: { t: { id: 't', projectId: 'p' } }, projectResourceLeases: { l: { id: 'l', resourceId: 'r', projectId: 'p', attemptId: 'a', generation: 1, status: 'reserved' } } }, + const rt = { home: root, state: { tasks: { t: { id: 't', projectId: 'p' } }, projectResourceLeases: { l: { id: 'l', resourceId: 'r', projectId: 'p', attemptId: 'a', role: 'developer', generation: 1, status: 'reserved' } } }, config: { projectResources: { r: { type: 'local', root, capacity: 1, runtimeDir, harnesses: ['codex'], allowedProjectOwnerIds: ['owner'], profiles: { owner: { codex: { home: join(root, 'profile'), configDir: join(root, 'profile/codex'), model: 'test-model', envRefs: { OPENAI_API_KEY: 'FLYBOT_TEST_HARNESS_SECRET' } } } } } } }, save: async () => {}, warn() {}, readProject: () => ({ id: 'p', ownerUserId: 'owner' }), requireProjectAccess(actor) { if (actor.memberId !== 'owner') throw new Error('Access denied') }, projectDelivery: { ownsWorkspace: (projectId, path) => projectId === 'p' && path === cwd ? workspace : null, sessionFor: () => session, @@ -562,3 +562,35 @@ await assert.rejects(harness.chiefTurn({agent:{id:'public-p',projectId:'p',lastR assert.equal(rt.projectResources.occupied('r'),0); assert.equal(Object.values(rt.state.projectResourceLeases)[0].chiefRunId,'inspection'); }); + +test('pre-worker SDK admission failure releases its reservation without deleting the Task context', async t => { + for (const kind of ['profile', 'model', 'binding']) { + const { rt, harness, cwd } = await fixture(t) + if (kind === 'profile') delete rt.config.projectResources.r.profiles + if (kind === 'model') delete rt.config.projectResources.r.profiles.owner.codex.model + if (kind === 'binding') { + const { taskSession } = await import('../src/host/project-task-sessions.js') + const session = rt.projectDelivery.sessionFor(), logical = taskSession(rt.state, session) + logical.bindings.push({ version: 'task-role-binding/v2', key: 'foreign', workerId: 'missing', sessionId: 'retained-thread' }) + } + await assert.rejects(harness.spawn({ executionProjectId: 'p', executionWorkDir: cwd, prompt: 'Admission must not leak' })) + assert.equal(rt.projectResources.occupied('r'), 0, kind) + if (kind === 'binding') assert.equal(Object.values(rt.state.projectTaskSessions)[0].bindings[0].sessionId, 'retained-thread') + } +}) + + +test('registered SDK failure before spawn rolls back capacity with a verifiable no-start proof', async t => { + const { rt, harness, cwd } = await fixture(t) + const session = rt.projectDelivery.sessionFor() + session.context.archive = { sources: [] }; session.context.archiveSha256 = 'a'.repeat(64) + rt.projectResources.uploadFile = async () => { throw new Error('Injected pre-spawn archive failure') } + await assert.rejects(harness.spawn({ executionProjectId: 'p', executionWorkDir: cwd, prompt: 'Fail before external spawn' }), /pre-spawn archive/) + const row = Object.values(rt.state.projectHarnessWorkers)[0] + assert.equal(row.status, 'released'); assert.equal(row.spawnPhase, 'aborted') + assert.equal(row.releaseProof.kind, 'sdk-worker-not-started') + assert.equal(rt.projectResources.occupied('r'), 0) + const { confirmedWorkerExit } = await import('../src/host/project-task-sessions.js') + assert.equal(confirmedWorkerExit(rt.state, row), true) + assert.equal(rt.state.projectAttempts, undefined, 'No Task history is rewritten by resource cleanup') +}) diff --git a/patches/ds-bot/files/test/project-resources.test.js b/patches/ds-bot/files/test/project-resources.test.js index 8fe3e03ec..fc2d0c41b 100644 --- a/patches/ds-bot/files/test/project-resources.test.js +++ b/patches/ds-bot/files/test/project-resources.test.js @@ -161,3 +161,73 @@ test('supervisor inspection cannot release a binding changed while the manager y await assert.rejects(releasing, /binding changed/) assert.equal(stored.status, 'running') }) + +test('retired unbound reservations are reclaimed but live preparation and unknown spawn remain held', async () => { + for (const kind of ['returned', 'superseded', 'cancelled', 'active', 'scope', 'worker']) { + const { rt, resources } = fixture() + const lease = await resources.reserve({ projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'orphan', harness: 'flybot', resourceIds: ['local'] }) + rt.state.projectAttempts = { a: { id: 'a', projectId: 'p1', taskId: 't', status: kind === 'returned' || kind === 'cancelled' ? kind : 'running', ...(kind === 'superseded' ? { supersededBy: 'b' } : {}) } } + rt.state.tasks = { t: { id: 't', projectId: 'p1', deliveryAttemptId: 'a' } } + if (kind === 'scope') rt.state.projectResourceLeases[lease.id].processScope = { kind: 'unmanaged-process', hostPid: process.pid } + if (kind === 'worker') rt.state.projectHarnessWorkers = { w: { id: 'w', leaseId: lease.id, generation: lease.generation, status: 'prepared' } } + await resources.reconcile() + assert.equal(resources.occupied('local'), ['returned', 'superseded', 'cancelled'].includes(kind) ? 0 : 1, kind) + if (['returned', 'superseded', 'cancelled'].includes(kind)) await assert.rejects(resources.markStarted(lease.id, lease.generation, { kind: 'sdk', handle: 'late' }), /released/) + } +}) + +test('claim replacement reclaims unbound preparation and failed reclamation save keeps capacity', async () => { + const { rt, resources } = fixture() + rt.state.projectAttempts = { a: { id: 'a', projectId: 'p1', taskId: 't', status: 'running' } } + rt.state.tasks = { t: { id: 't', projectId: 'p1', deliveryAttemptId: 'a', collaboration: { generation: 2, claim: { id: 'claim', generation: 2, status: 'active' } } } } + const lease = await resources.reserve({ projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'owned', harness: 'flybot', resourceIds: ['local'], reservationOwner: { taskId: 't', claimId: 'claim', generation: 2 } }) + await resources.reconcile(); assert.equal(resources.occupied('local'), 1) + rt.state.tasks.t.collaboration.claim.status = 'failed' + const save = rt.save; rt.save = async () => { throw new Error('save failed') } + await assert.rejects(resources.reconcile(), /save failed/) + assert.equal(resources.occupied('local'), 1) + rt.save = save; await resources.reconcile(); assert.equal(resources.occupied('local'), 0) + assert.equal(rt.state.projectResourceLeases[lease.id].reclamation.owner.claimId, 'claim') +}) + +test('a registered or spawning SDK reservation is not inferred idle from a missing controller', async () => { + const { rt, resources } = fixture() + const lease = await resources.reserve({ projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'phase', harness: 'flybot', resourceIds: ['local'] }) + rt.state.projectAttempts = { a: { id: 'a', projectId: 'p1', taskId: 't', status: 'returned' } } + rt.state.projectHarnessWorkers = { sdk: { id: 'sdk', projectId: 'p1', attemptId: 'a', role: 'developer', resourceId: 'local', leaseId: lease.id, generation: lease.generation, status: 'prepared', spawnPhase: 'starting' } } + const row = rt.state.projectResourceLeases[lease.id]; row.worker = { kind: 'sdk', handle: 'sdk' }; row.status = 'running' + await resources.reconcile(); assert.equal(resources.occupied('local'), 1) + rt.state.projectHarnessWorkers.sdk.spawnPhase = 'registered' + await resources.reconcile(); assert.equal(resources.occupied('local'), 0) + await assert.rejects(resources.fenceSpawn(lease.id, lease.generation, 'sdk'), /released|retired/) +}) + +test('asynchronous preparation remains owned until its actual completion, even after claim retirement', async () => { + const { rt, resources } = fixture() + rt.state.projectAttempts = { a: { id: 'a', projectId: 'p1', status: 'running' } } + const lease = await resources.reserve({ projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'preparing', harness: 'flybot', resourceIds: ['local'] }) + const token = await resources.beginPreparation(lease.id, lease.generation) + rt.state.projectAttempts.a.status = 'returned' + await resources.reconcile(); assert.equal(resources.occupied('local'), 1) + assert.equal(await resources.abortReservation(lease.id, lease.generation, 'Attempt ended during preparation'), false) + await resources.endPreparation(token) + await resources.reconcile(); assert.equal(resources.occupied('local'), 0) + await assert.rejects(resources.beginPreparation(lease.id, lease.generation), /released/) +}) + +test('failed reservation/batch admission cannot return a capability or consume capacity', async () => { + const { rt, resources } = fixture() + rt.config.projectResources.local.capacity = 2 + const request = { projectId: 'p1', attemptId: 'a', role: 'developer', sessionKey: 'save-failed', harness: 'flybot', resourceIds: ['local'] } + let writes = 0 + rt.save = async () => { if (++writes === 1) throw new Error('reservation persistence failed') } + await assert.rejects(resources.reserve(request), /persistence failed/) + assert.equal(resources.occupied('local'), 0) + const old = Object.values(rt.state.projectResourceLeases)[0] + await assert.rejects(resources.markStarted(old.id, old.generation, { kind: 'sdk', handle: 'late' }), /released/) + writes = 0; rt.save = async () => { if (++writes === 2) throw new Error('batch persistence failed') } + await assert.rejects(resources.reserveRoles('p1', 'batch', { roles: ['developer', 'tester'].map(role => ({ role, execution: { harness: 'flybot', resourceIds: ['local'] } })) }), /batch persistence failed/) + assert.equal(resources.occupied('local'), 0) + const next = await resources.reserve({ ...request, sessionKey: 'legitimate-tester', role: 'tester' }) + assert.equal(next.status, 'reserved') +}) diff --git a/scripts/api-contracts.mjs b/scripts/api-contracts.mjs index 4f0b6ce60..dff526230 100644 --- a/scripts/api-contracts.mjs +++ b/scripts/api-contracts.mjs @@ -63,6 +63,7 @@ export const schemas = { Event: record({ projectId: id, cursor: str(600), revision: int(), entityType: str(), entityId: id, type: str(), at: int(), data: json }, ['projectId', 'cursor']), Snapshot: result({ project: ref('Project'), revision: int(), cursor: str(600), entities: array(json), next: nullable(id) }, ['project', 'cursor']), EventPage: result({ events: array(ref('Event')), next: nullable(str()), cursor: str(), hasMore: bool }, ['events']), + ResourceLease: record({ id, projectId: id, attemptId: id, role, sessionKey: id, resourceId: id, generation: int(1), status: choice('reserved', 'running', 'stopping', 'unreachable', 'released'), reservationOwner: record({ taskId: id, claimId: id, generation: int(1), sessionId: id }), reclamation: record({ reason: str(400), at: int(), owner: nullable(json), effect: json }), releaseProof: json, releasedAt: int() }, ['id', 'projectId', 'resourceId', 'generation', 'status']), Resource: record({ id, type: choice('local', 'docker'), harnesses: array(harness), status: str(), capacity: int(), occupied: int(), health: json }), HealthCheck: record({ status: choice('checking', 'ready', 'unreachable', 'unconfigured', 'authRequired', 'environmentError'), reason: str(), at: int(), harness, resourceId: id, sdkVersion: nullable(str()), expectedSdkVersion: nullable(str()) }, ['status']), Terminal: record({ id, terminalId: id, expiresAt: int(), resourceId: id, harness, seq: int(), output: array(json), closed: bool, released: bool, reason: str() }), @@ -320,7 +321,7 @@ native.get('project-role').permission = 'manage' native.get('project-role').notes = 'The shared Project access layer requires Owner manage permission. The underlying legacy role policy also supports self-demotion, but that does not bypass this public shared gateway gate. Owner cannot be assigned, removed or demoted.' native.get('project-board').response = result({ tasks: array(json), done: array(json), decisions: result({ dir: nullable(str()), files: array(json), reason: str() }, ['files']), meetings: result({ decided: array(json), pending: array(json), reason: str() }, ['decided', 'pending']), waiting: array(json), inflight: array(json), blocked: array(json), counts: map(nullable(int())), metrics: nullable(json) }) native.get('workbench-state').response = result({ demoMode: nullable(str()), self: result({ memberId: id, displayName: str() }), projects: array(ref('Project')), models: json, repositories: array(repository) }) -native.get('project-resources').response = result({ projectId: id, quota: result({ maxConcurrent: int(1, 32) }), activeSlots: int(), capacity: nullable(ref('CapacityState')), repository: json, queue: array(record({ id, taskId: id, status: str(), reason: str() }, ['id', 'taskId', 'status'])), taskExecution: array(json), metrics: json, workers: array(json), workspaces: array(json), harnessWorkers: array(json), executionResources: array(ref('Resource')), resourceLeases: array(json), chief: queries.get('chief.state').response, taskViews: array(ref('TaskView')), host: json, platformInstance: nullable(json) }, ['projectId', 'quota', 'activeSlots', 'repository', 'queue', 'taskExecution', 'metrics']) +native.get('project-resources').response = result({ projectId: id, quota: result({ maxConcurrent: int(1, 32) }), activeSlots: int(), capacity: nullable(ref('CapacityState')), repository: json, queue: array(record({ id, taskId: id, status: str(), reason: str() }, ['id', 'taskId', 'status'])), taskExecution: array(json), metrics: json, workers: array(json), workspaces: array(json), harnessWorkers: array(json), executionResources: array(ref('Resource')), resourceLeases: array(ref('ResourceLease')), chief: queries.get('chief.state').response, taskViews: array(ref('TaskView')), host: json, platformInstance: nullable(json) }, ['projectId', 'quota', 'activeSlots', 'repository', 'queue', 'taskExecution', 'metrics']) queries.get('compatibility').response = result({ version: str(), cutoverId: nullable(id), committed: bool, baseline: nullable(result({ count: int(), sha256: hash })), totals: result({ contract: int(), legacy: int(), held: int(), active: int() }), tasks: array(record({ id, projectId: id, parentId: nullable(id), status: str(), mode: choice('contract', 'legacy', 'held'), attemptId: nullable(id), parentPresent: bool }, ['id', 'projectId', 'status', 'mode'])), page: result({ offset: int(), limit: int(1, 100), total: int(), nextOffset: nullable(int()) }) }) const sourceRecord = record({ id, version: lit('project-source/v1'), projectId: id, repository, kind: str() }, ['id', 'projectId']) const subscription = record({ id, version: lit('project-source/v1'), projectId: id, taskId: id, sourceId: id, threadId: id, createdBy: id, enabled: bool, mode: choice('active', 'context'), includeHistory: bool, createdAt: int() }, ['id', 'projectId', 'taskId', 'threadId', 'enabled'])