Repository navigation

#39 Liveness check-in for processor run (co-processor-live) - #42

Merged

gregoryfoster merged 18 commits into

mainfrom

Oct 9, 2026

Merged

#39 Liveness check-in for processor run (co-processor-live)#42
gregoryfoster merged 18 commits into
mainfrom
39-liveness

gregoryfoster commented Oct 8, 2026 •

edited

Loading

Member

Refs #39. Plan: docs/plans/2026-10-08-39-liveness.md. The table is on CannObserv/status#29 (#24 had closed). The monitor co-processor-live = 01M4H5FH3PTPGEB24XYQJ4NG7A was created disabled, and its id is in deploy/processor.service.

What: processor run checks in ok to Status's co-processor-live every 300 s, only while the consume loop has made progress within 605 s (reclaim_min_idle_ms + read_block_ms). It never sends alert: silence past the 900 s grace is the page.

processor.liveness.Heartbeat: a task beside Consumer.run.
- Each check-in runs in a daemon thread (not the default executor) and is awaited for at most 10 s.
- A tick whose check-in is still running is skipped. One attempt per tick, and nothing raises into the loop.
- It logs on transitions only.
Consumer.last_progress is stamped after every step (backoff included) and every message (reclaim walks included). Consumer.backing_off goes in the variables.
StatusSettings is shared by Settings (live_monitor_id, live_interval_s, live_checkin_timeout_s < interval) and DriftSettings. The drift unit still needs no broker settings.
The starting record gains liveness: on <id>, off, or off: missing the status-checkin-key credential, plus an ERROR. A missing key never stops the service.
deploy/processor.service loads status-checkin-key, with the \n fallback.
Docs: DEPLOYMENT § Liveness (missing, the records, the grace, deliberate stops, rollback past Liveness check-in for processor run (co-status monitor co-processor-live) #39), the containment host check over both keys, spec §2/§3, AGENTS.md.

Done-when tests:

test_a_hung_status_never_delays_a_command: Status takes the connection and never answers, and a command is processed while the check-in hangs.
test_a_wedged_loop_goes_silent_while_the_event_loop_stays_up.
test_the_child_cannot_read_the_real_secrets with the Status key, on disk and as the unit's credential. It skips where exedev can't read it; DEPLOYMENT § Containment's key loop is the live evidence.

Review: CR 1–5 are applied, one commit each (see the plan's "Amended in review"). The real one is CR 1: the deploy's smoke run now carries the unit's SetCredential= fallback, so a missing Status key can't fail a deploy.

After merge:

scripts/deploy.sh (after 2026-10-09T21:35Z needs no watcher#326 heads-up).
The first ok on status#29.
Status enables the monitor.
The rehearsal, a PATCH to 60/0 then back, with no restart in its window. The operator confirms both notices.

🤖 Generated with Claude Code

gregoryfoster and others added 10 commits

October 8, 2026 17:14


          #39 docs: the plan — a liveness check-in to co-processor-live

edbef8a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 test: the heartbeat checks in only while the consume loop progresses

b1a1e14

A hung Status never delays a command; a wedged loop goes silent while the
event loop stays up; empty reads, backoffs and each reclaimed message are
progress.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 feat: processor.liveness, and the consumer's last_progress

5df4854

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 test: the live monitor's settings, shared with the drift check's

b815775

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 feat: StatusSettings, and Settings' live_monitor_id, interval and…

9e3ec8b

… timeout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 test: processor run checks in to its live monitor, or says why not

9d7a1a9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 feat: processor run starts the heartbeat beside the consume loop

f458563

The starting record names it: on <id>, off, or off for a missing key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 test: the unit loads the Status key; the child can't read it

4de4797

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 feat: processor.service loads status-checkin-key, with the empty …

d81bb42

…fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 docs: § Liveness, the second credential, spec §2/§3, AGENTS.md

3b74b6b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gregoryfoster mentioned this pull request

co-processor-live: Processor's liveness monitor (follows #24) CannObserv/status#29

Closed

gregoryfoster commented Oct 9, 2026

Member Author

The table moved to CannObserv/status#29: status#24 had closed on 2026-10-07, before the table was posted there. The monitor id will come from #29.

gregoryfoster mentioned this pull request

Burn-down: open issues from #8 on CannObserv/status#16

Closed

30 tasks done

gregoryfoster and others added 2 commits

October 9, 2026 20:29


          #39 test: processor.service names co-processor-live

40e2495

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 feat: co-processor-live's id in processor.service (status#29)

f3ba617

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gregoryfoster mentioned this pull request

Liveness check-in for processor run (co-status monitor co-processor-live) #39

Closed

gregoryfoster marked this pull request as ready for review

October 9, 2026 21:01

gregoryfoster and others added 6 commits

October 9, 2026 21:09


          #39 fix: CR 1 — the smoke run carries the unit's SetCredential= fallback

07eb86d

Without it a missing status-checkin.key failed the smoke's transient unit
(exit 243, probed) and switched the deploy back, where the service itself
starts with liveness off. systemd-run unescapes the \n (probed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 fix: CR 2 — a tick that raises is logged, and the heartbeat lives on

2e45512

A bug outside tick's own handlers ended the task silently: the monitor went
missing with no record why, and processor run's stop re-raised it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 test: CR 5 — the heartbeat tests wait out the tasks they cancel

2ef49e7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 docs: CR 4 — spec §7 names the liveness tests

cc61f64

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 docs: CR 3 — the plan's amendments in implementation and review

7f42db5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


          #39 docs: CR 1 — DEPLOYMENT's smoke run names the credential fallbacks

fe0de17

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gregoryfoster merged commit fe0de17 into main

4 of 6 checks passed

gregoryfoster commented Oct 9, 2026

Member Author

Shipped: FF-merged to main at fe0de17 (2026-10-09T21:59:14Z), and deployed with scripts/deploy.sh (CI run 37996667935 passed; live fe0de17d4c96). Issue #39 stays open for the live checks.

gregoryfoster deleted the 39-liveness branch

October 9, 2026 22:03

This was referenced Oct 10, 2026

Index processor on co-index's shared SocratiCode store (init-socraticode, STORE=external) #5

Closed

CI pulls redis:7.0.15 from Docker Hub unauthenticated; since #34 a Hub outage or rate limit blocks deploys #43

Closed

Burn-down: open issues after v1 go-live (closed 2026-10-23; #54 continues alone) #24

Closed

gregoryfoster mentioned this pull request

Root disk at 79% with nothing pruning or watching it #82

Closed

gregoryfoster mentioned this pull request

#82 Adopt the cohort disk GC: config, links, pytest retention, runbook #95

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet