Repository navigation
Liveness check-in for processor run (co-status monitor co-processor-live) #39
Description
Suggested by CannObserv/status#24, where the operator decided on 2026-10-07: yes to a liveness monitor, created once this issue settles its design. Raised in this repo's #35 hand-off as out of scope there.
Why
Since 2026-10-06T21:35Z (CannObserv/watcher#326), Processor's output decides Watcher's production results. A quiet Processor shows up only in Watcher's own health, and nothing alerts the operator directly. co-processor-drift (status#24) doesn't cover it: its missing means the drift check isn't reporting, not that processor run has stopped consuming.
What co-status provides
A monitor co-processor-live on the existing co-processor tenant, with the same key as the drift check (/etc/processor/status-checkin.key). It uses the same contract: POST http://status:9000/api/v1/monitors/{id}/checkin, with {"status": "ok"|"alert", "variables": {...}} and X-API-Key. Alerts go to the same Mailgun + Slack pair. Silence past interval + grace goes missing, then recovered on the next check-in.
co-status creates the monitor once the table below is filled in, and posts its id here.
To decide here
| Field | Question |
|---|---|
interval_seconds |
N: how often processor run checks in |
grace_seconds |
How much silence is normal: restarts, deploys (scripts/deploy.sh restarts the unit), slow batches |
renotify_seconds |
Daily, as the drift monitor? |
title_template / body_template |
Only if alert is ever sent; they render variables |
| variables | What each check-in carries (opaque to co-status; every check-in should carry all the template uses) |
Design questions
- What does a check-in prove? If it fires only after work was done, an empty queue reads as an outage, a false alarm. A check-in from the consume loop on a timer, whether or not there was work, proves the loop is alive. Which does "live" mean for Processor?
okonly, oralerttoo? E.g. consuming but failing every item. Eachalertcheck-in sends its own report, with no dedupe on co-status's side, so analertevery N minutes is one notification every N minutes.- A co-status outage must not affect Processor (A drift signal when live lags origin/main (status#12's drift check) #35's trap 4). The drift check stays out of
processor run's path. A liveness check-in can't, so it needs a short timeout, must never block or slow consumption, and never retries in a loop. A failed check-in logs and moves on. - Deploys: a restart longer than the grace pages. Size the grace for it, or accept the page.
Done when
processor runchecks in on its cadence, and a stubbed slow or failing co-status provably doesn't stall consumption (tests).- The table above is posted on Tenant + drift monitor for Processor (co-processor / co-processor-drift) status#24, so co-status can create the monitor.
- Live: the monitor goes
pending→ok. Stoppingprocessor.servicepast the grace sends "has stopped reporting", and starting it sends "has recovered", confirmed by the operator.
Activity
gregoryfoster commentedon Oct 8, 2026
Hand-off from the burn-down epic #24, 2026-10-08. This is next. #40 shipped in #41 (test-only, no deploy).
Why now. Processor has decided production results since 2026-10-06T21:35:24Z. Since the last check-in it has decided three real changes on its own: the WSLCB item at 2026-10-07T19:29Z (
7d020b93…) and again at 22:32Z (abddd5dc…), and01KTCNXKXN7Z94G495YEGM6JX3at 19:22Z (85da1fe7…). If Processor goes quiet now, nothing pages the operator directly. The operator said yes to this monitor on status#24.Soak policy (operator, 2026-10-07): Watcher's soak runs until 2026-10-09T21:35Z. This item deploys (a restart), and its live Done-when stops
processor.serviceon purpose. Both are allowed during the soak, each announced on watcher#326 beforehand. That's a cross-repo post, drafted here and approved by the operator each time. Doing the deliberate stop after 21:35Z on the 9th needs no heads-up.What #35 left you (checked against
mainatf8f5534):src/processor/checkin.py:read_key(directory)andpost_checkin(base_url, monitor_id, key, status, variables, *, timeout). It refuses a malformed key before any request (CR 1), and raisesCheckinFailedwithout quoting headers.src/processor/settings.py:70–90: a separate settings class, so the drift unit never needs the broker credential. It hasstatus_url(defaulthttp://status:9000),drift_monitor_id(a ULID pattern) andcredentials_directory.deploy/processor-drift.service:39–40:LoadCredential=status-checkin-key:/etc/processor/status-checkin.key, plus aSetCredential=fallback.deploy/processor.serviceloads onlygcs-writer-key, so the main unit needs the Status key added.Consumer.run(consumer.py:118):start, thenstepin a loop, backing off on any exception.stepreclaims when due, thenread(count=1, block_ms). One extraction can hold a step for up to 120 s.Traps found while drafting:
post_checkinis blockingrequests. Called from the async loop, a slow Status stalls consumption for up to its timeout, every interval. Run it off the loop (asyncio.to_thread, or its own task with a boundedasyncio.timeout). Add a test where a stubbed Status hangs and a command is still processed on time (the issue's Done-when).tests/test_checkin.py::test_a_stall_is_cut_offis the existing stub to build on.stepis wedged (a hung read, or a reclaim walk that never ends), a free-running heartbeat keeps reportingok. Recommendation: the consume loop recordslast_step_at(monotonic) after each completedstep, success or backoff, and the heartbeat checks inokonly if that's fresh: newer than the interval plus the longest legitimate step (the extraction timeout plus the read block). Stale means stay silent and let Status gomissing. Don't sendalert: Status sends a report for everyalertcheck-in with no dedupe (design question 2), and silence already pages.status-checkin-keynext togcs-writer-key. Containment's allowlist leaves/run/credentialsout, andtests/test_child.py(around line 263) pins the GCS key's denial. Add the Status key to that test, so the denial is proven rather than assumed.Settingsand the drift check's class are separate on purpose. Addlive_monitor_id(and an interval) whereprocessor runreads them, without making the drift unit need broker settings. An emptylive_monitor_idmeans liveness is off (dev, CI), stated in thestartingrecord.last_step_atadvances on backoff. Size the grace above the worst case a deploy produces (give the number in the table), so a normal deploy never pages.The status#24 table (the issue's), filled in on status#24, as a cross-repo comment drafted and approved. Status then creates
co-processor-liveand posts its id.interval_seconds,grace_seconds,renotify_seconds(the drift monitor's is daily), variables (e.g.build,last_step_age_s; no templates if noalertis ever sent).Done when (the issue's, plus):
status-checkin-key(test).pending→ok, and a deliberate stop past the grace sends "has stopped reporting", then "has recovered", confirmed by the operator. That's a stop during the soak unless after 2026-10-09T21:35Z, so a heads-up first.Ship: issue → plan → the status#24 table (approval) → TDD → PR → FF merge → a heads-up on watcher#326 (approval) →
scripts/deploy.sh→ Status's id in the unit → live checks. The closing comment gives each box with its evidence, the known limits, and the follow-ups (or "none").Kickoff prompt:
#39 test: the heartbeat checks in only while the consume loop progresses
#39 feat: processor.liveness, and the consumer's last_progress
#39 test: the live monitor's settings, shared with the drift check's
#39 feat: StatusSettings, and Settings' live_monitor_id, interval and…
#39 test: processor run checks in to its live monitor, or says why not
#39 feat: processor run starts the heartbeat beside the consume loop
#39 test: the unit loads the Status key; the child can't read it
#39 feat: processor.service loads status-checkin-key, with the empty …
#39 docs: § Liveness, the second credential, spec §2/§3, AGENTS.md
#39 test: processor.service names co-processor-live
#39 feat: co-processor-live's id in processor.service (status#29)
gregoryfoster commentedon Oct 9, 2026
Monitor created by Status, disabled:
co-processor-live=01M4H5FH3PTPGEB24XYQJ4NG7A(CannObserv/status#29, issuecomment-6088692433, 2026-10-09T20:26:53Z; 300/900/86400, the same channels asco-processor-drift; check-ins on a disabled monitor are recorded, and the sweep doesn't watch it). The id is indeploy/processor.serviceon PR #42 (40e2495 test, f3ba617 feat).Agreed with Status (status#29 is the record):
ok(server time, build) on status#29. Status's operator then enables the monitor.processor.serviceduring that window. The operator confirms both notices.This replaces the plan's deliberate stop: the rehearsal never stops
processor.service.#39 fix: CR 1 — the smoke run carries the unit's SetCredential= fallback
#39 fix: CR 2 — a tick that raises is logged, and the heartbeat lives on
#39 test: CR 5 — the heartbeat tests wait out the tasks they cancel
#39 docs: CR 4 — spec §7 names the liveness tests
#39 docs: CR 3 — the plan's amendments in implementation and review
#39 docs: CR 1 — DEPLOYMENT's smoke run names the credential fallbacks
gregoryfoster commentedon Oct 9, 2026
Shipped and live. PR #42 FF-merged at
fe0de17(2026-10-09T21:59:14Z), with one CR round (CR 1–5).scripts/deploy.shputfe0de17d4c96live: CI run 37996667935 passed after re-runs through Docker Hub's outage.starting22:03:07.493863Z:liveness: on 01M4H5FH3PTPGEB24XYQJ4NG7A.202,last_progress_age_s0,backing_offfalse./run/credentials/processor.service/status-checkin-key(PermissionError), while the service uid can read it (the control). Same forgcs-writer-key.Left:
gregoryfoster commentedon Oct 9, 2026
Done. Shipped in #42 (FF to
fe0de17, 2026-10-09T21:59:14Z, one CR round, CR 1–5). Live asfe0de17d4c96since 22:03:05Z. Monitorco-processor-live=01M4H5FH3PTPGEB24XYQJ4NG7A(CannObserv/status#29).processor runchecks in on its cadencestarting22:03:07.493863Z withliveness: on 01M4H5FH3PTPGEB24XYQJ4NG7A. First check-in 22:03:07.539540Z,202. Then every 300 s: Status recorded 22:08, 22:13:08 and 22:18:11, against our ticks at :08:07, :13:07 and :18:07tests/test_consumer.py::test_a_hung_status_never_delays_a_command: Status takes the connection and never answers, and a command completes while the check-in hangs.tests/test_liveness.py: refusals, an unreachable Status, a hung one cut off at the timeout, and the next tick skipped while one is in flighttests/test_consumer.py::test_a_wedged_loop_goes_silent_while_the_event_loop_stays_upstatus-checkin-keytests/test_child.py::test_the_child_cannot_read_the_real_secretsgained both paths. It skips whereexedevcan't read them, which is the case onco-processor. Live, after the deploy (DEPLOYMENT § Containment): the service uid reads/run/credentials/processor.service/status-checkin-key(the control), and the contained child getsPermissionError. Same forgcs-writer-keyokpending→ok, then "has stopped reporting" and "has recovered", confirmed by the operatorpending→okat the first check-in. The rehearsal was run by Status's operator onco-processor. It was a PATCH to 60/0 rather than a stop ofprocessor.service, agreed with Status and recorded in the plan's amendments. "Has stopped reporting" went out at 22:14:44Z, and the PATCH back to 300/900 followed within 5 s. Our 22:18:07 tick sent "has recovered" at 22:18:11Z. Both were delivered, and both arrived by email and Slack, confirmed by the operator. Our side: same MainPID throughout, no failed, skipped or stale tick. Timeline and dispatch ids: status#29missingmeans, the records, the grace, deliberate stops and rollback past #39. Also the second credential in §/etc/processor/, the containment loop over both keys, and the smoke run's credential fallbacks. Spec §2 (Liveness row, tailnet, grants), §3 (the heartbeat) and §7 (its tests); AGENTS.md's infrastructure rowKnown limits:
backing_off. The broker's own probe is the alarm for that.CO_PROCESSOR_LIVE_INTERVAL_Smust move together with the monitor'sinterval_seconds.main's run gated the deploy. It resolved itself, so nothing is filed.Follow-ups: none.