Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
Goal:
Restore a checkpoint without replaying or resuming runtime runs that occurred
after that checkpoint, while retaining those runs and events for audit.

Scope:
- Add durable, backwards-compatible run supersession fields to JSONL, SQLite,
and PostgreSQL runtime records.
- Mark post-checkpoint active runs inside SQLite and PostgreSQL durable restore
transactions.
- Add an explicit active-run query contract and use it for session history,
latest-run/resume state, result fallback, and ACP session load.
- Exclude superseded runs from executor claim paths and persist the reconciled
current turn in session metadata.
- Add focused persistence, restore, query, HTTP, ACP, and WebUI regressions.
- Record the recurring projection-reconciliation failure in `postmortem/`.

Out of scope:
- Deleting runtime runs, events, snapshots, or interactions.
- Atomic runtime-projection reconciliation for legacy JSONL checkpoint restore.
- Production deployment, o6n mutation, GitOps updates, or live smoke tests.
- Changing developer-console audit or direct run-id/event endpoints to hide
superseded records.
- Redesigning checkpoint snapshot payloads or restore authorization.

Context:
- ADRs:
- `docs/adr/0005-checkpoint-restore-uses-truncate-rollback.md`
- `docs/adr/0010-synchronize-checkpoint-restore-with-active-turns.md`
- `docs/adr/0068-local-sqlite-transactional-durable-fencing.md`
- `docs/adr/0075-checkpoint-restore-active-run-timeline.md`
- Relevant files:
- `src/coding_agent/stores/runtime_store.py`
- `src/coding_agent/stores/durable_local.py`
- `src/coding_agent/stores/durable_pg.py`
- `src/coding_agent/runs/query.py`
- `src/coding_agent/server/http_server.py`
- `src/coding_agent/acp/server.py`
- `webui/app/src/App.test.tsx`
- Existing behavior:
- Checkpoint capture runs under exclusive runtime maintenance admission.
- Durable restore already reconciles tape, session, topics, and checkpoints in
one SQLite or PostgreSQL transaction.
- `list_agent_runs()` is an audit/control query and must remain unfiltered.
- A test command that selects zero tests is a failure, not verification.
- Postmortem:
- `PM-0026` matches `runtime_store.py`; run its focused adapter/lifecycle
release checks and verify no new blank exception-to-run-error path.

Target tests:
- `uv run pytest tests/coding_agent/test_sqlite_runtime_store.py tests/coding_agent/test_pg_runtime_store.py tests/coding_agent/test_sqlite_local_durable_fencing.py tests/coding_agent/test_pg_durable_fencing.py tests/coding_agent/test_runtime_query_service.py -v`
- `uv run pytest tests/ui/test_http_server.py tests/acp/test_server.py -k "checkpoint or runtime_run or session_load or result" -v`
- `uv run pytest tests/ui/test_session_manager_runtime.py -k "restore_durable_state" -v`
- `uv run pytest tests/coding_agent/test_adapter_types.py tests/coding_agent/test_pipeline_adapter.py tests/coding_agent/test_run_lifecycle.py -v`
- `pnpm --dir webui/app --ignore-workspace run test -- App.test.tsx`
- `rg -n "str\\(exc\\)" src/coding_agent/stores/runtime_store.py src/coding_agent/stores/durable_local.py src/coding_agent/stores/durable_pg.py`

Loop policy:
- Engineer writes focused failing tests first, confirms the intended failures,
implements the smallest correct change, and runs the target tests.
- Reviewer reviews only the resulting diff and affected tests.
- Reviewer reports only P1/P2 findings.
- Engineer fixes only accepted P1/P2 findings and reruns the same target tests.
- Verifier reruns the exact target tests and reports pass/fail only.

Stop conditions:
- At most one review/fix/retest cycle.
- Escalate architectural redirection or scope expansion to the human.
- Ignore non-blocking optimization suggestions.
- Do not push, open a PR, deploy, or mutate o6n without explicit authorization.
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# ADR-0075: Reconcile the active run timeline on checkpoint restore

**Status**: Proposed
**Date**: 2026-08-05

## Context

Checkpoint restore replaces the stable tape and session configuration with the
selected snapshot, but runtime run records and their events remain durable. The
session history, resume metadata, result fallback, and ACP session load currently
list every durable run. A run created after the restored checkpoint can therefore
reappear as an apparently current assistant response even though its tape entries
were rolled back.

Runtime records also serve audit, worker recovery, and developer diagnostics.
Deleting those records would repair the visible timeline at the cost of losing
evidence. Filtering by the checkpoint timestamp forever is also incorrect because
new runs created after a successful restore belong to the new active timeline.

Checkpoint capture is admitted through the session's exclusive runtime
maintenance boundary. A run cannot legitimately overlap capture, so the
checkpoint creation time is the durable boundary available to existing snapshots.

## Decision

Preserve every runtime run and event for audit, and add nullable run fields that
record when and by which checkpoint a run was superseded. During durable SQLite
and PostgreSQL restore, mark previously active runs for the same session whose
`started_at` is later than the checkpoint's `created_at`. This mutation is part of
the same fenced transaction that restores tape, session state, topics, and the
checkpoint set. Existing supersession markers are not overwritten.

Before the restore transaction persists the session payload, recompute
`current_turn_id` from the latest active run at or before the checkpoint boundary.
Persist that optional value in session metadata so restart hydration remains
aligned with the restored timeline.

Keep `list_agent_runs()` and direct run/event lookup as full audit queries. Add an
explicit active-timeline query in the run query service that excludes superseded
runs. Session history, latest-run selection, resume metadata, result fallback, and
ACP session load use the active query; worker recovery, developer diagnostics, and
direct run/event endpoints keep using the audit query.

New runs created after restore are active because supersession is persisted only
for rows present when restore executes. Existing databases are upgraded
idempotently so the nullable fields do not require an offline migration.

JSONL records round trip the optional fields for compatibility, but JSONL restore
does not gain projection atomicity in this change. JSONL is a legacy/debug format;
the production contract applies to the authoritative SQLite and PostgreSQL
durable stores.

## Alternatives Rejected

- Delete post-checkpoint runs and events: fixes replay but destroys audit and
incident evidence.
- Filter permanently by checkpoint creation time: also hides legitimate runs
created after restore.
- Filter only in WebUI: leaves ACP load, resume, session summary, and result
fallback inconsistent.
- Change `list_agent_runs()` to hide superseded rows everywhere: breaks audit,
startup recovery, worker state, and developer-console consumers.

Executor claim paths are control-plane consumers rather than audit queries. They
exclude superseded requested or expired rows so rolled-back work cannot execute
after restore.

## Acceptance Criteria

- [ ] SQLite and PostgreSQL runtime schemas upgrade existing databases and round
trip supersession fields.
- [ ] SQLite and PostgreSQL durable restore atomically mark only active runs that
started after the restored checkpoint.
- [ ] Active run queries, latest-run selection, session summary, result fallback,
and ACP load ignore superseded runs.
- [ ] Full run listing and direct run/event/display-event lookup retain
superseded records for audit.
- [ ] A run created after restore remains visible on the active timeline.
- [ ] Superseded requested or expired runs cannot be claimed by an executor.
- [ ] Restart hydration retains the reconciled `current_turn_id`.
- [ ] `uv run pytest tests/coding_agent/test_sqlite_runtime_store.py tests/coding_agent/test_pg_runtime_store.py tests/coding_agent/test_sqlite_local_durable_fencing.py tests/coding_agent/test_pg_durable_fencing.py tests/coding_agent/test_runtime_query_service.py -v`
- [ ] `uv run pytest tests/ui/test_http_server.py tests/acp/test_server.py -k "checkpoint or runtime_run or session_load or result" -v`
- [ ] `pnpm --dir webui/app --ignore-workspace run test -- App.test.tsx`

## References

- `docs/adr/0005-checkpoint-restore-uses-truncate-rollback.md`
- `docs/adr/0010-synchronize-checkpoint-restore-with-active-turns.md`
- `docs/adr/0068-local-sqlite-transactional-durable-fencing.md`
- `src/coding_agent/stores/runtime_store.py`
- `src/coding_agent/stores/durable_local.py`
- `src/coding_agent/stores/durable_pg.py`
- `src/coding_agent/runs/query.py`
- `src/coding_agent/server/http_server.py`
- `src/coding_agent/acp/server.py`
- `webui/app/src/lib/api.ts`
28 changes: 28 additions & 0 deletions coding-agent/postmortem/index.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -669,3 +669,31 @@ patterns:
- message
- run
- persistence
- id: PM-0027
title: Reconcile runtime projections on checkpoint restore
path: patterns/PM-0027-reconcile-runtime-projections-on-checkpoint-restore.md
severity: high
confidence: high
subsystems:
- checkpoint
- runtime
- ui
related_files:
- src/coding_agent/stores/durable_local.py
- src/coding_agent/stores/durable_pg.py
- src/coding_agent/stores/runtime_store.py
- src/coding_agent/runs/query.py
- src/coding_agent/server/http_server.py
- src/coding_agent/acp/server.py
- webui/app/src/App.tsx
- webui/app/src/lib/api.ts
related_commits: []
keywords:
- checkpoint
- restore
- runtime
- projection
- superseded
- timeline
- replay
- audit
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
id: PM-0027
title: Reconcile runtime projections on checkpoint restore
status: active
severity: high
confidence: high
subsystems:
- checkpoint
- runtime
- ui
related_commits: []
related_files:
- src/coding_agent/stores/durable_local.py
- src/coding_agent/stores/durable_pg.py
- src/coding_agent/stores/runtime_store.py
- src/coding_agent/runs/query.py
- src/coding_agent/server/http_server.py
- src/coding_agent/acp/server.py
- webui/app/src/App.tsx
- webui/app/src/lib/api.ts
release_checks:
- Verify SQLite and PostgreSQL restore reconcile the active run timeline atomically.
- Verify session history, resume, result fallback, ACP, and WebUI use active runs only.
- Verify direct run and event lookup still exposes superseded records for audit.
- Verify a run created after restore remains active.
- Verify executor claim paths reject superseded requested and expired runs.
- Verify the reconciled current turn survives session reload.
---

# Summary

Restoring tape entries without reconciling durable runtime projections can make
post-checkpoint assistant replies reappear after a successful restore. The tape
is the conversation source of truth, but run records and display events remain
valid audit evidence and must not be deleted.

Persist an explicit supersession marker during the restore transaction. Product
timeline consumers must query active runs, while audit and control consumers keep
the full run set. Do not infer the active timeline only in a frontend, and do not
apply a permanent timestamp filter that would hide new runs after restore.

# Release Review Checklist

- Run focused SQLite and PostgreSQL durable restore tests.
- Run runtime-query, HTTP result/session-history, ACP load, and WebUI restore tests.
- Confirm superseded run events remain addressable by direct run id.
- Confirm post-restore runs are not hidden.
- Confirm superseded queued runs cannot be claimed and executed.
- Confirm restored session persistence reloads the latest active turn id.
11 changes: 6 additions & 5 deletions coding-agent/src/coding_agent/acp/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@ async def replay_display_events(

async def list_runtime_runs(self, session_id: str) -> list[Any]: ...

async def list_active_runtime_runs(self, session_id: str) -> list[Any]: ...

async def list_sessions_async(self) -> list[str]: ...

async def close_session(self, session_id: str) -> None: ...
Expand Down Expand Up @@ -151,9 +153,7 @@ def __init__(
self._model_name = model_name
self._base_url = base_url
self._max_steps = max_steps
self._modes = (
modes if modes is not None else default_acp_modes()
)
self._modes = modes if modes is not None else default_acp_modes()
self._active_prompt_sessions: set[str] = set()
self._cancelled_prompt_sessions: set[str] = set()

Expand Down Expand Up @@ -389,7 +389,7 @@ async def _session_load(self, params: JSONObject) -> None:
session_id,
additional_directories,
)
runs = await self._session_manager.list_runtime_runs(session_id)
runs = await self._session_manager.list_active_runtime_runs(session_id)
for run in _sort_runtime_runs(runs):
run_id = getattr(run, "run_id", None)
if not isinstance(run_id, str) or not run_id:
Expand Down Expand Up @@ -872,7 +872,8 @@ def _session_mode_state(modes: Iterable[AcpMode] | None = None) -> JSONObject:
{
"id": m.id,
"name": m.name,
"description": m.description or f"Provider: {m.provider or 'any'}, Model: {m.model or 'any'}",
"description": m.description
or f"Provider: {m.provider or 'any'}, Model: {m.model or 'any'}",
}
for m in (modes or default_acp_modes())
]
Expand Down
5 changes: 5 additions & 0 deletions coding-agent/src/coding_agent/cli/local_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ async def update_session_mcp_servers(

async def list_runtime_runs(self, session_id: str) -> Any: ...

async def list_active_runtime_runs(self, session_id: str) -> Any: ...

async def list_checkpoints(self, session_id: str) -> Any: ...

async def session_resume_metadata(self, session_id: str) -> dict[str, object]: ...
Expand Down Expand Up @@ -233,6 +235,9 @@ async def update_session_additional_directories(
async def list_runtime_runs(self, session_id: str) -> Any:
return await self._delegate.list_runtime_runs(session_id)

async def list_active_runtime_runs(self, session_id: str) -> Any:
return await self._delegate.list_active_runtime_runs(session_id)

async def list_checkpoints(self, session_id: str) -> Any:
return await self._delegate.list_checkpoints(session_id)

Expand Down
14 changes: 12 additions & 2 deletions coding-agent/src/coding_agent/runs/query.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,18 @@ async def load_runtime_run(self, run_id: str) -> AgentRunRecord:
async def list_runtime_runs(self, session_id: str) -> list[AgentRunRecord]:
return await self._require_store().list_agent_runs(session_id)

async def list_active_runtime_runs(
self,
session_id: str,
) -> list[AgentRunRecord]:
return [
run
for run in await self.list_runtime_runs(session_id)
if run.superseded_at is None
]

async def latest_runtime_run(self, session_id: str) -> AgentRunRecord | None:
runs = await self.list_runtime_runs(session_id)
runs = await self.list_active_runtime_runs(session_id)
if not runs:
return None
return max(runs, key=lambda run: (run.started_at, run.run_id))
Expand Down Expand Up @@ -194,7 +204,7 @@ async def session_resume_metadata(
)
interrupted_runs = [
run
for run in await self.list_runtime_runs(session.id)
for run in await self.list_active_runtime_runs(session.id)
if run.status == "interrupted"
]
if interrupted_runs:
Expand Down
4 changes: 2 additions & 2 deletions coding-agent/src/coding_agent/server/http_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -4195,7 +4195,7 @@ async def list_session_runtime_runs(
del request
_ = await _get_visible_session(session_id, auth_context)
try:
records = await session_manager.list_runtime_runs(session_id)
records = await session_manager.list_active_runtime_runs(session_id)
except RuntimeError as exc:
raise HTTPException(
status_code=404,
Expand Down Expand Up @@ -6491,7 +6491,7 @@ async def _session_result_runtime_run_id(session: Session) -> str | None:
if session.current_turn_id is not None:
return session.current_turn_id
try:
runs = await session_manager.list_runtime_runs(session.id)
runs = await session_manager.list_active_runtime_runs(session.id)
except RuntimeError:
return None
if not runs:
Expand Down
Loading
Loading