fix(desktop): tighten computer use input guards for text, tabs, menus and AX failures

[waleedlatif1]

Summary

Follow-ups to #9084, #9093 and #9100. A review of those changes found input guards that were looser than they needed to be. Each finding below was confirmed on a macOS 26 VM, against both the previous helper and this one.

Text input

• type_text and input_sequence accepted any element whose focus can be set. That let them type into sliders, and, in web and Electron content, into buttons and lists, where a space or a letter is a command. Typed text now needs a text control: a text field or area (terminals included), a combo box, a date or time field, or anything with an editable text selection. A container target is accepted only while its focused editor holds focus, and keys are sent only while focus is in a text input.
• Targeting a container whose inner editor already had focus moved focus to the container. That editor now keeps focus.

Windows

• A hidden native tab (Finder, Terminal) is a separate off-screen window with the same bounds as the visible tab. Accessibility exposes only the visible tab, so a click aimed at a hidden tab's id was accepted and landed in the visible tab. The selected window could also be a hidden tab. Off-screen windows no longer get a window identity, and the focused window is now matched by its Accessibility identity rather than by bounds.
• A click on a window's own background or title bar was refused as occluded, because a window does not report an AXWindow of itself. One containingWindow helper now handles this for the hit-test, element windows and the editor's window.
• An element click that falls back to a coordinate click now targets the element's own window. Before, it used whichever observed window contained the point, in arbitrary order.

Menus

• The menu bar is now read with the priority entries. On a large window the old helper never reached the menu bar: with a 2,000-node Safari page it observed no menu nodes, so no menu could be opened. Now an open menu's items come first. A closed menu's items are read last, still capped at 80.
• The Return that completes a highlighted menu item is sent only while the item's own menu is still open and the app itself reports the item as highlighted. Before, any pop-up-level window of the app counted as an open menu, and the helper set the highlight itself. Calendar keeps reporting an item as highlighted after its menu closes, so the open-menu check is what stops a stray Return.

Accessibility failures

• An Accessibility failure now counts as dispatched unless Accessibility rejected the request before the app saw it (unsupported attribute or action, illegal argument, not implemented, API disabled). A generic failure can follow an effect. It used to report "Nothing was sent; this snapshot is still valid", which could lead to a duplicate retry.
• "This snapshot is still valid" is now added only to request errors. It used to follow "observe again" on stale-window, occlusion and focus errors, and it repeated activation_required's own wording.

Other

• The desktop service rejects arguments its bundled schema predates with a structured invalid_arguments error marked not_started, instead of a raw ZodError. The web treated a raw ZodError as "outcome unknown, do not retry", although nothing had been queued.
• The computer-use contract copies are synced from simstudioai/mothership#725: one description per shared field, and scroll deltas no longer described as "Defaults to 0" while one is required. This is a description-only change.

Type of Change

• Bug fix

Testing

Run on a macOS 26.6 VM, previous helper vs this one:

Check	Previous helper	This change
tests/live.py	pass	pass
New live.py checks: slider refuses text; window target types into its focused field; title-bar click	fail	pass
Finder window with three tabs: right-click aimed at each window id	all 4 accepted, a hidden tab selected	only the visible tab accepted and selected
TextEdit title-bar click	window_occluded	accepted
Safari with a 2,000-node page: History or Bookmarks open	no menu bar observed	15 and 16 open items observed
Finder New Tab, Calendar New Event via the menus	ran once	ran once

• The paired benchmark against the previous helper is posted below.
• vitest for service.test.ts (32 passed) and ipc.test.ts (45 passed); biome.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing (new tests pass the test-audit authoring gate)
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 16, 2026 5:15am UTC

[cubic-dev-ai]

All reported issues were addressed across 6 files

<sub>Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic</sub>

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no blocking issue remains.

Summary

This PR tightens text input, window targeting, menu actions, and Accessibility error reporting. It also returns structured errors for unsupported desktop arguments and syncs shared contract descriptions.

• Text actions only send characters to a text input.
• Coordinate input follows the window that owns its target.
• Open menus stay visible in snapshots and only finish while open.
• Accessibility errors distinguish rejected requests from input that may have run.
• Older desktop builds return a structured error for new arguments.
• Computer-use descriptions explain shared fields the same way.

_{Reviews (2) · Last reviewed commit: "fix(desktop): keep hidden tabs out of th..." · Reviewed by Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 6 files

<sub>Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic</sub>