Skip to content

feat(runstore): stream run progress over Server-Sent Events - #925

Open
HugoXK wants to merge 12 commits into
ThinkFlowLab:mainfrom
HugoXK:agent/922-runstore-sse
Open

HugoXK wants to merge 12 commits into
ThinkFlowLab:mainfrom
HugoXK:agent/922-runstore-sse

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #922

What changed

Adds GET /api/runs/<run_id>/events to the existing read-only run-store
server. It is a Server-Sent Events stream:

  • an initial status event from status.json;
  • one round event per new rounds.jsonl line, with the round index as the
    event id, so a reconnect can send Last-Event-ID and resume after the last
    round it saw;
  • a final status event when the run reaches done, failed or cancelled,
    then the stream closes;
  • retry: 1000 and periodic : keep-alive comments;
  • the same loopback-only, read-only, loopback-CORS boundary as the JSON
    endpoints, and one _run_id_in helper shared by both routes for the run-id
    traversal check.

The stream polls rounds.jsonl (default every EVENT_POLL_SECONDS = 0.5) and
does not add a write path, an auth change or a background thread to the run
itself. serve_http(..., event_poll=...) is a wall-clock knob, used by the
tests to keep them fast. The keep-alive interval is EVENT_KEEPALIVE_SECONDS.

The root CNAME that came in from the fork's main history is removed
(0d21fef): it is not part of this feature, and a root CNAME would repoint
the project's GitHub Pages domain.

Verification at the tip

Tip SHA: bf4b25b

Ran:

  • pytest -q tests/test_run_events.py tests/test_run_panel.py → 21 passed
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 70 changelog fragment(s) OK
  • pytest -q (full suite) on a trial merge of this tip into main
    1aaf1bd0: 20 failed, 10 errors, 30 node ids; that set is the same one, line
    for line, that main 1aaf1bd0 fails on this machine (docker/sandbox and
    environment failures in test_sandbox_container, test_genesis_example,
    test_era_srbench, test_apply_noop, test_skill_library_*). This PR adds
    none of them.

Not run:

  • a browser EventSource client — the offline tests read the stream with
    urllib; the protocol is covered at the transport level.
  • mkdocs build --strict to completion on this machine: no route to
    fonts.google.com, which the Material social plugin fetches while building
    index.md. With social disabled the docs build completes cleanly
    (exit 0), and CI's build job runs the real one.

Non-blocking items from the review

In the reviewer's order:

  1. A stalled run streams forever — deferred, no code change needed. The
    guard already exists: RunDir.status() rewrites a running run whose pid is
    gone as failed, and the stream closes on that terminal state (probed at
    this tip: closes immediately with state: failed). A test now pins it.
    Ending the stream for a live but wedged run needs a max-idle policy, which
    is separate work with its own design question.
  2. Every tick re-parses all of rounds.jsonl — deferred. Tracking a byte
    offset changes the read model (torn lines, file replacement) and deserves
    its own PR and tests rather than riding this review.
  3. Status and the rounds that caused it are emitted status-then-rounds —
    deferred
    with (2): changing the order changes what a reconnecting client
    sees, and is easiest to judge together with the incremental reader.
  4. No Last-Event-ID test for a negative or garbage value — fixed. A
    garbage value replays every round; pinned by a test.
  5. Keep-alive interval hard-coded — fixed. It is
    EVENT_KEEPALIVE_SECONDS, with a test that exercises the keep-alive comment.
  6. _events duplicates the run-id traversal check — fixed. One
    _run_id_in helper is used by both routes.

@HugoXK HugoXK added area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) area:runstore Run store, ledger, checkpoint, resume and replay enhancement New feature or request labels Nov 10, 2026

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CHANGES REQUESTED

Tip SHA: 900f69a
Ran (on a trial merge of this tip into main 1aaf1bd):

  • python3 .github/scripts/risk_gate.py --repo ThinkFlowLab/agentdescent --pr 925 → T3 (path agentdescent/runstore.py)
  • pytest -q tests/test_run_events.py tests/test_run_panel.py → 18 passed
  • pytest -q (full suite) → no failures through ~81% before my 600 s local timeout; CI test (3.9–3.13) still running at review time
    Not run: a browser EventSource client; the full suite to completion locally.

Blocking

  1. Unrelated CNAME file at the repo root (agentdescent.hugoxk.online). It isn't part of this feature and isn't in the commit 900f69a; it comes in from the fork's history (the Merge branch 'Birfy:main' into main merge on the fork's main). Merged into this repo, a root CNAME can repoint the project's GitHub Pages to a personal domain. Please drop it from the PR. The cleanest way is to branch from current ThinkFlowLab/agentdescent main (not the fork's main) and cherry-pick 900f69a.
  2. PR body still says Tip SHA: PENDING and pytest -q → PENDING. Please fill in the real tip and full-suite result once (1) is fixed.

Non-blocking (correctness/design)

  • A stalled run streams forever. If the run process dies without writing a terminal status, state stays running and the handler thread loops until the client disconnects. A cheap guard would be to emit a terminal status (or close) when status.json's pid is no longer alive, the same way status / list_runs detect a stale run, or to cap the stream with a max idle time so the client reconnects.
  • Each poll re-parses the whole rounds.jsonl. That's fine for tens of rounds, but tracking the byte offset (or at least only parsing lines past the last-seen count) would keep long runs O(new lines) per tick.
  • Status change and the rounds that caused it are emitted in status-then-rounds order. On the terminal tick, the final rounds are emitted after the status was read but before return, which is right. On a non-terminal state change, though, a client sees status before the rounds that preceded it. Consider reading rounds first, then status, so event order matches what happened.
  • Last-Event-ID with a negative or garbage value silently falls back to replay-all. That's reasonable, but a one-line test would pin it.
  • Keep-alive interval (15 s) is hard-coded while the poll interval is configurable; making it a module constant next to EVENT_POLL_SECONDS would let tests cover the keep-alive branch, which no test exercises today.
  • _events duplicates the run-id traversal check in do_GET; hoisting it into one helper would keep the two endpoints from drifting.

Good

The read-only, loopback-only CORS boundary matches the JSON endpoints. X-Accel-Buffering: no, retry: and the round-index ids are the right SSE details, and the tests drive a real server on an ephemeral port with a small event_poll. The docs change in docs/plugins.md merges cleanly onto main as a small paragraph, so it's fine as is.

Risk tier T3: this PR needs Danyang's explicit merge click even after a PASS; it won't be auto-merged.

@github-actions github-actions Bot added the risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller label Nov 10, 2026
CNAME (agentdescent.hugoxk.online) is the fork's Pages file, not part of this
feature; it entered the branch from the fork's main history. A root CNAME in
the project repository would repoint its GitHub Pages to a personal domain.
The keep-alive interval was a literal while the poll interval was configurable;
make it EVENT_KEEPALIVE_SECONDS so a test can exercise the branch. Both routes
that take a run id now use one _run_id_in helper, so the SSE route and the JSON
route cannot drift apart.

Tests: a stalled run closes on its terminal status, a garbage Last-Event-ID
replays every round, and a quiet stream sends keep-alive comments.
@HugoXK

HugoXK commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Please re-review.

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS

Tip SHA: bf4b25b
Ran (on a trial merge of this tip into main 1aaf1bd):

  • python3 .github/scripts/risk_gate.py --repo ThinkFlowLab/agentdescent --pr 925 → T3 (path agentdescent/runstore.py)
  • git diff --stat main <trial merge> → only the 4 intended files (runstore.py, changelog.d/922-runstore-sse.md, docs/plugins.md +5/-1, tests/test_run_events.py); root CNAME is gone
  • pytest -q tests/test_run_events.py tests/test_run_panel.py tests/test_runstore_cli.py → all passed
  • CI test (3.9–3.13), build, risk/reviewed-tip report checks → SUCCESS at this tip
    Not run: a browser EventSource client; the full suite locally (CI owns it).

Both blocking items from the 900f69a review are fixed: the fork's CNAME is dropped and the PR body carries the real tip and suite result. The stalled-run case is now pinned by a test (dead pid → terminal failed status closes the stream), and keep-alive, garbage Last-Event-ID, and a shared _run_id_in guard are covered.

Non-blocking: each poll still re-parses all of rounds.jsonl, and status is emitted before the rounds of the same tick on a non-terminal change; fine for now, worth a follow-up if long runs get panels.

T3 (risk:T3): needs Danyang's merge click; not merging from the agent.

@HugoXK HugoXK added the awaiting-approval Green T2/T3 PR: waits for a PASS at its tip and Danyang's merge click label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) area:runstore Run store, ledger, checkpoint, resume and replay awaiting-approval Green T2/T3 PR: waits for a PASS at its tip and Danyang's merge click enhancement New feature or request risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(runstore): stream run progress over SSE

2 participants