Open the live view's soft keyboard from the remote browser's focus by masnwilliams · Pull Request #464 · kernel/kernel-images

masnwilliams · 2026-10-14T23:42:14Z

Stacked on #448 (base hypeship/live-view-touch-input).

Summary

On touch devices, the live view decides whether to raise the soft keyboard from the remote cursor shape, which Chromium also shows over selectable text. That guess cannot tell a text field from a paragraph, picks the wrong keyboard, and misses closes after a submit that navigates to a page with a search box in the same spot. This PR has the browser VM report which editable element has focus, and where the fields on screen are. The live view uses that report to raise the keyboard only for real fields, give it the field's keyboard and AutoFill, and close it when the field loses focus.

Design

In the VM: server/lib/livefocus. It runs only while a live view is subscribed, and holds its own DevTools connection. The browsersurface tracker is used with WithoutLocations, so it enables no Page domain. Each frame of every page and out-of-process iframe gets watcher.js in an isolated world (Page.createIsolatedWorld). The server long-polls that watcher with Runtime.callFunctionOn(awaitPromise).

Each watcher reports its document's focused editable element: input type or textarea or contenteditable, inputmode, autocomplete, enterkeyhint, the page's own autocapitalize/autocorrect/spellcheck, a one-time-code flag, and its rect. It also reports whether its document has focus.
A top frame also reports the editable fields on screen that a tap at their center would reach (checked with elementFromPoint), including fields in same-origin iframes. The server converts these rects to screen pixels using the window bounds (Browser.getWindowForTarget) and page zoom.
The one-time-code flag is set by autocomplete="one-time-code", by one-character boxes, or by name, label or placeholder hints such as code, OTP or verification on short or numeric inputs. Zip, postal and promo codes are excluded.
A navigation ends the pending poll ("Inspected target navigated or closed" or "Execution context was destroyed"). That clears the frame's focus and puts a new watcher in the next document. A new child frame shows up in its parent's report and triggers a frame-tree rescan.
Reports are sent on focusin and focusout, on window focus and blur, on visibility changes, scrolls, resizes and DOM mutations (throttled), and on a 1s tick. Only changes are sent.

Stealth. Nothing runs in the page's main world, and no Runtime, Page or DOM domain is enabled (the real-Chromium test asserts this through a recording proxy). The page cannot see the isolated world's globals, listeners or observers, and page changes to globals or prototypes do not reach it. The watcher never logs, and catches its own errors. If a watcher goes 30s without a poll, it removes its listeners and observer, so a dropped connection leaves nothing running in the page. A CDP client of the customer's own that has Runtime enabled will see one extra isolated-world context per document while a touch live view is open. Playwright and Puppeteer ignore isolated worlds they did not create.

Dismissing the keyboard: POST /live-view/focus/blur?seq=N. Hide, the keyboard's own Done, and an iOS dismissal that leaves the overlay focused (seen as the keyboard leaving the visual viewport) all blur the remote's focused field too, the way Safari's Done does. The blur runs from the focused frame's isolated world, so the page and the live view agree and the next tap is a fresh focus. Each watcher counts focus changes and mouse presses and reports the count as the focused field's seq. The live view sends back the count it last saw, and a blur sent before a later tap is dropped, even a tap on the field that already had focus. Closing the keyboard for the page's own reasons (a blur, a navigation, the clipboard panel), refocusing, or leaving for another app does not blur the remote field.

Delivery: GET /live-view/focus, server-sent events from the kernel-images API. This is outside the OpenAPI spec, like the PTY attach endpoint. The live view page is served at /browser/live?jwt=…, and the browser proxy already serves the kernel-images API at /browser/kernel/* with the same session token, streaming included. So the client derives the URL from its own location, with no neko change. The other route was neko's data channel. neko's legacy protocol, which this client uses, drops custom broadcasts, so that route needs a neko fork change and release, plus a privileged neko session for the API to send through. The stream sends the current state, then each change, then a ping every 15s. The client opens it only on touch devices, while hosting with input enabled and the page visible. Without the proxy route (an older image, or a local run-docker.sh live view) or with ?focusSignal=off, the stream is unavailable and the client keeps the cursor-shape behavior.

Client.

remote-focus.ts holds the stream state, hit-tests taps against the field rects (in stream pixels), and ignores rects from before a client scroll until new ones arrive.
focus-keyboard.ts decides when the keyboard opens and closes, under iOS's rule that focus() must run inside the touch:
- A tap on a field from the last rects, or on the focused field wherever it is, raises the keyboard inside the tap, set up for that field. If no field has focus by the end of a 1s wait, it closes. If the overlay kept its focus with no keyboard on screen, the tap focuses it afresh, because a focus it already has would not raise the keyboard.
- On iOS, any other tap on the page holds focus with inputmode="none", as Make live view usable on touch devices #448 does. A focus report within the wait then raises the keyboard. This covers labels and buttons that focus a field.
- A keyboard raised for a field closes 300ms after the field's blur, unless focus returns. That includes a submit that navigates or reloads, and a search box in the same place on the next page does not keep it open. A chat box that keeps focus after Enter keeps it open.
- When the page moves focus to another field, such as the next box of a code, the open keyboard switches layout.
The cursor-shape guess (TapKeyboard) stays for two cases. One is taps outside the page area, which is browser UI such as the address bar, where no page document has focus. The other is a live view without the stream, or with it down or stale; a keyboard raised then is not tied to a field until a focus report claims it. A keyboard raised for the address bar closes when the page gets keyboard focus back (pageFocused), which happens when the address bar navigates. The Enter-then-cursor check runs only without the stream.
field-keyboard.ts copies the field onto the overlay:
- The layout goes through inputmode (email, tel, url, numeric, decimal, search). type="email" or type="number" would sanitize the overlay's sentinel and break its selection APIs.
- Code fields get autocomplete="one-time-code", for iOS SMS code AutoFill. Email and tel fields get the owner's own details, and name and address fields get contact AutoFill. Credentials never get AutoFill.
- Plain text gets autocorrect, capitals and spellcheck. A page that turns these off is followed. Google's search box sets autocorrect="off", for example. Password, username, code, email, url, tel and number fields get all three off.
- iOS reads these attributes when the overlay gains focus. So a field that needs other attributes refocuses the overlay inside the tap, while a held focus or a page-driven focus change only switches the layout.
soft-keyboard.ts tells whether a soft keyboard is actually up from how much of the layout viewport (innerHeight) the visual viewport leaves visible. The keyboard covers the layout viewport without resizing it. Safari's toolbars resize both, so they are not mistaken for a keyboard, and no learned baseline is needed, through a rotation included. keyboardShown is the one source for the extra keys row, the keyboard control's Hide label and what that control does. On a phone, or once a keyboard has been seen, it is a focused overlay with the keyboard measured up, however the focus came about. Otherwise, as with a tablet's hardware keyboard or inside a frame, it goes by the focus.
On touch devices, focusing the overlay no longer reads the phone clipboard. On iOS that read shows the native Paste callout and holds back the keyboard; the Paste control reads the clipboard inside its own tap.
typed-text.ts: for autocorrect fields, printable keys and Backspace go into the overlay instead of being sent by the Guacamole keyboard, and each change to the overlay's text is diffed against what was sent. iOS autocorrect (insertReplacementText), predictive text, dictation and composing keyboards then reach the remote as backspaces and replacement text, whatever order the events come in. The overlay starts over on a tap, on Enter or another non-text key, and on a paste or shortcut. Other fields keep Make live view usable on touch devices #448's per-key path.

Desktop is unchanged: the stream is never opened, and the overlay's attributes and typing path are as before.

Testing

Go: go test -race ./lib/livefocus ./cmd/api/.... Unit tests cover:
- the merge (focused frame, shown tab, zoom, cross-origin frames, pageFocused), subscriptions and linger, and the SSE stream;
- against a fake browser: a frame that fails for a while is watched once it works, a frame gone from the tree is dropped, and an out-of-process iframe is never watched through its parent's session.
KERNEL_LIVEFOCUS_CHROME_E2E=1 go test ./lib/livefocus runs against headless Chromium with --site-per-process, 5 runs under -race. It covers:
- fields and attributes, including a one-character code box and a same-origin iframe field inside border and padding;
- leaving out button, covered, disabled and off-screen fields;
- focus and blur from real clicks, and focus in a cross-origin iframe (no rect), with no further createIsolatedWorld calls while it is shown;
- clearing focus on navigation;
- the page's main world unchanged, and Runtime.enable, Page.enable, DOM.enable, addScriptToEvaluateOnNewDocument and addBinding never sent.
Client: bun test tests (229 tests). The simulated iOS overlay also covers the first keyboard after loading the live view (including Safari's toolbars appearing after the load, and a keyboard that comes up after the focus), no clipboard read on a touch focus, a keyboard iOS shows for a held focus, rotation with the keyboard up or going, and a hardware keyboard. A simulated iOS overlay covers tap, Hide, and a re-tap of the same field (with and without the remote blur landing), a dismissal that keeps the overlay focused, the keyboard's own Done versus leaving the page, a focus iOS shows no keyboard for, and the address bar with Hide and a re-tap. New tests cover the focus keyboard state machine, field-to-keyboard mapping, stream state and hit-testing, the autocorrect diff, and the video component's focus, refocus and tap routing.
On a 390x844 headful Kernel browser, with the API and live view swapped in:
- The stream through the browser proxy reported focus about 33ms after a CDP click, with the click itself about 16ms of that.
- Driving the live view from Chromium with the iPhone 13 profile (iOS user agent, touch), every check passed:
  - each field type got its keyboard attributes and typed text reached the remote;
  - an insertReplacementText edit replaced the word on the remote;
  - a chat box stayed open after Enter;
  - a search on html.duckduckgo.com closed the keyboard though the results page has a search box;
  - tapping paragraph text or a button opened no keyboard;
  - an address-bar URL plus Enter opened and then closed the keyboard;
  - Hide blurred the remote field through the proxy, and a re-tap focused it again and typing reached it.
- Desktop typing (Shift, Backspace) was unchanged and opened no stream.

Not verified: a physical iPhone or Android device (the real keyboard, AutoFill and autocorrect behavior, and the inputmode switch on a held focus). The CI workflows only run for PRs against main, so they did not run on this stacked PR; the tests above were run locally.

Known gaps

A frame whose watcher cannot start, such as one showing an error page, is retried with backoff (up to 5s) for as long as it is in its session's frame tree.
Fields inside cross-origin iframes and shadow DOM report focus but have no pre-sent rects. On iOS those taps use the held focus, so the keyboard keeps the default attributes (no autocorrect, code AutoFill offered).
Rect conversion assumes an unscaled screen (device scale factor 1) and no pinch zoom inside the remote page.
Browser UI other than the address bar (find bar, prompt dialogs) uses the cursor-shape guess.

Note

Medium Risk
Adds always-on-while-subscribed CDP instrumentation and new HTTP routes that blur remote fields; behavior is complex on iOS but gated to touch/proxy sessions with seq-checked blurs and cursor fallback.

Overview
Touch live views no longer rely only on the remote cursor shape to decide when to show the soft keyboard. The VM exposes SSE focus state (GET /live-view/focus) and a remote blur action (POST /live-view/focus/blur?seq=…) via new livefocus CDP watchers in isolated worlds, reporting focused editables, on-screen field rects, and page vs browser UI focus.

The chromium-headful client subscribes on touch devices (browser proxy + JWT), routes taps through focus-keyboard and remote-focus hit-testing, maps fields to overlay attributes (field-keyboard), detects real keyboard visibility (soft-keyboard / keyboardShown), and syncs autocorrect through typed-text. Hiding the keyboard or iOS dismissing it blurs the remote field when appropriate; cursor-shape tap-keyboard remains as fallback when the stream is off or stale.

Extensive Go and Bun tests cover merge logic, CDP connection edge cases, keyboard state machines, and simulated iOS overlay flows.

^{Reviewed by Cursor Bugbot for commit 6966a49. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor