Skip to content

Show live agent browser view from the feature card (#70) - #265

Open
zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-70-barbara
Open

zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-70-barbara

Conversation

@zwanski-start

Copy link
Copy Markdown

Closes #70

Context

Most of what #70 asks for is already on master (commit 9f87f7a, "embed browser windows"). Agents always run playwright-cli headless, and BrowserViewService streams periodic screenshots of each agent's session over the project WebSocket. These appear on the Browsers page and in the debug panel's Browsers tab. The issue also asked for the browser to be reachable from the feature card, and that part was missing. This PR adds it and fixes a subscription bug I found in the existing streaming code.

What changed

  • Live browser in the feature modal (FeatureBrowserPreview.tsx, FeatureModal.tsx, layout/Modals.tsx). Clicking the card of a feature an active agent is working on now shows a Live Browser section with that agent's latest screenshot. A batched feature is matched through the agent that owns the batch. The section also has an All browsers shortcut to the Browsers page. Features with no active agent show nothing new.
  • Subscription fix (useWebSocket.ts):
    • subscribeBrowserView/unsubscribeBrowserView are now reference-counted. Before, closing the modal while the Browsers page was mounted would have unsubscribed both.
    • The subscription is re-sent in onopen. Before, a subscribe made while the socket was still connecting (e.g. a page reload on the Browsers view) or lost on a reconnect was silently dropped, so no screenshots appeared.
  • The modal's status now says IN PROGRESS for in-progress features; it used to say PENDING.

Verification

  • npm run lint: 0 errors. The 6 warnings were already there, none are in changed files.
  • npm run build passes.
  • Manual check: ran the real backend (isolated HOME) with Vite on :5443, using a demo project with feature fix: auto-activate venv in start_ui.sh #1 in progress. Mocked the project WebSocket with Playwright routeWebSocket to send an agent_update and answer browser_view_subscribe with a browser_screenshot. Results:

Assumptions

  • I treated the existing screenshot streaming as the answer to "display inside the UI" and did not add a real interactive iframe. The app under test runs in the agent's own headless browser session; an iframe would load a separate copy of the app, not show what the agent sees.
  • No backend changes were needed.

🤖 Generated with Claude Code

Clicking a feature card that an agent is working on now shows that
agent's embedded browser view, with a shortcut to the Browsers page,
so browser testing can be watched without desktop windows.

Browser-view subscriptions in useProjectWebSocket are now reference
counted (so the modal and Browsers page don't cancel each other) and
are re-sent when the socket (re)connects; previously a subscribe issued
before the socket opened, or across a reconnect, was silently lost.

The modal also shows IN PROGRESS instead of PENDING for active features.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zwanski-start

Copy link
Copy Markdown
Author

🔍 QA report: ✅ Passed

Tester: Ken (cubefarm agent, in a fresh session) · Round: 1 · Author: Barbara

The feature modal now shows a Live Browser section with the latest screenshot from the agent working on the feature, including features in another feature's batch, plus an 'All browsers' shortcut. The browser-view subscription is now reference-counted and re-sent when the WebSocket opens. Lint and build pass, and with a real backend and a mocked agent WebSocket I confirmed the modal preview, batched-feature matching, that idle features are unchanged, balanced subscribe/unsubscribe messages, and that a reload on the Browsers page now shows the image. The e2e suite could not run because Playwright's Chromium is missing on the QA machine; that is an environment problem, not caused by this PR.

Check Details
✅ Issue #70: browser view reachable from the feature card With a mocked agent_update (Spark on feature #1, batch [1,3]), clicking the 'Login form' card on Kanban opened the modal with a LIVE BROWSER section showing the mocked screenshot, the agent name, its type and a timestamp.
✅ Batched feature matched through the agent that owns the batch Feature #3 is in feature #1's batch. Its modal showed the screenshot from session coding-1 (featureId 1).
✅ Feature with no active agent unchanged Feature #2's modal shows PENDING and no Live Browser section, and opening it sent no subscribe message.
✅ IN PROGRESS status label In-progress features now show a cyan IN PROGRESS label with a spinner instead of PENDING.
✅ 'All browsers' shortcut Closes the modal, switches to the Browsers view (active view 'browsers') and shows the screenshot. Messages sent: exactly one browser_view_unsubscribe then one browser_view_subscribe.
✅ Subscribe/unsubscribe balance on view changes I wrapped WebSocket.send in the page. Each Browsers -> Logs -> Browsers -> Logs switch sent matched unsubscribe/subscribe pairs, with no duplicates.
✅ Subscription re-sent on socket open (reload on Browsers page) After reloading with the Browsers view active, the screenshot rendered. The mock only sends a screenshot after it receives browser_view_subscribe, so the subscribe was sent after the socket opened.
✅ Reference counting with several consumers at once Checked by reading the code. Only the first subscribe and the last unsubscribe reach the server, and the count never goes below zero. Not exercised in the browser because the current UI never mounts two consumers at the same time.
✅ Phone layout (390x844) The modal with the live browser fits the screen, and document scrollWidth equals the viewport width (no horizontal scroll).
✅ Console errors 0 errors. The only warnings are Radix's missing-dialog-description warnings, which come from the existing dialog code.
✅ Backend unchanged / security intact The diff touches only 4 UI files. security.py and the middleware are untouched, and no secrets are logged.
⏭️ UI Playwright e2e suite Playwright's chromium-headless-shell is not installed on the QA machine and npx playwright install chromium timed out after 240s. Port 5173, which the stock config reuses, is held by another process. Environment problem, not caused by this PR.
🧪 Commands run
Command Result
git diff origin/master...HEAD 4 UI files changed, +189/-8
npm --prefix ui run lint 0 errors, 6 warnings (all in files this PR doesn't touch)
npm --prefix ui run build tsc -b && vite build succeeded
ruff check . All checks passed
venv/bin/python test_security.py ALL TESTS PASSED
HOME=<scratch>/home venv/bin/uvicorn server.main:app --host 127.0.0.1 --port 5345 Booted; GET /api/projects 200, GET /api/projects/demo/features 200
npx playwright test -c <scratch config, baseURL :5345> 15 failed: browser executable missing (environment), not test failures
npx playwright install chromium Timed out (download did not finish within 240s)

No browser screenshots were taken in this round.

Posted by cubefarm · merges automatically once GitHub's checks pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Launch browser window within UI

1 participant