Version: simdeck 0.2.0 (82e61b3), Xcode 26.6, iOS 26.5 simulator (iPad Pro 11-inch M5), headless (simctl boot, no video session).
What happens: on a simulator turned to landscape, a selector tap (POST /api/simulators/{udid}/action with {"action":"tap","selector":{...}}) returns ok but lands somewhere else. Raw normalized taps land correctly once the caller rotates the point itself.
Repro:
- Boot an iPad simulator, open any app with a button near a corner (Settings works).
POST .../action {"action":"rotateLeft"}.POST .../action {"action":"tap","selector":{"label":"<that button>"}} → {"ok":true}, and the button is not pressed.
Why: the accessibility tree reports an app's frames in the turned UI, while touches are injected in the unrotated (portrait) screen.
- REST path:
tap_point_from_snapshot (api/accessibility_query.rs:207) normalizes the element centre by the root frame and sends that as the touch point, with no rotation.
- CLI path:
normalize_accessibility_point_for_display (main/tap_target.rs:164) swaps x and y when root and display orientation differ. A swap is a mirror, not a rotation, so it is wrong in both landscape directions.
The correct mapping depends on the direction. With u, v the element centre normalized in the turned UI, and the device turned q quarter turns:
| q |
touch x |
touch y |
| 1 |
1 - v |
u |
| 2 |
1 - u |
1 - v |
| 3 |
v |
1 - u |
So the swap (v, u) should become (1 - v, u) or (v, 1 - u). The direction cannot be read from the tree: the root frame is identical in both landscape directions. It can be read from GET .../accessibility-point, which hit-tests in unrotated screen points and answers with UI-space frames. Hit-test a known element's centre under each candidate turn, and the one that returns that element is the device's turn. Repeated rotateLeft calls also do not accumulate: each REST call creates a new display bridge whose _deviceRotationDegrees starts at 0, so you cannot reach upside-down portrait through the REST API.
SpringBoard alerts (e.g. "Open in …?") on a turned device report their children in unrotated portrait points under a root whose frame is the turned screen, so those must not be rotated.
We work around this on the client side for now (rotate the point ourselves and send a plain normalized tap). Happy to test a fix.
Version: simdeck 0.2.0 (
82e61b3), Xcode 26.6, iOS 26.5 simulator (iPad Pro 11-inch M5), headless (simctl boot, no video session).What happens: on a simulator turned to landscape, a selector tap (
POST /api/simulators/{udid}/actionwith{"action":"tap","selector":{...}}) returnsokbut lands somewhere else. Raw normalized taps land correctly once the caller rotates the point itself.Repro:
POST .../action {"action":"rotateLeft"}.POST .../action {"action":"tap","selector":{"label":"<that button>"}}→{"ok":true}, and the button is not pressed.Why: the accessibility tree reports an app's frames in the turned UI, while touches are injected in the unrotated (portrait) screen.
tap_point_from_snapshot(api/accessibility_query.rs:207) normalizes the element centre by the root frame and sends that as the touch point, with no rotation.normalize_accessibility_point_for_display(main/tap_target.rs:164) swaps x and y when root and display orientation differ. A swap is a mirror, not a rotation, so it is wrong in both landscape directions.The correct mapping depends on the direction. With
u, vthe element centre normalized in the turned UI, and the device turnedqquarter turns:1 - vu1 - u1 - vv1 - uSo the swap
(v, u)should become(1 - v, u)or(v, 1 - u). The direction cannot be read from the tree: the root frame is identical in both landscape directions. It can be read fromGET .../accessibility-point, which hit-tests in unrotated screen points and answers with UI-space frames. Hit-test a known element's centre under each candidate turn, and the one that returns that element is the device's turn. RepeatedrotateLeftcalls also do not accumulate: each REST call creates a new display bridge whose_deviceRotationDegreesstarts at 0, so you cannot reach upside-down portrait through the REST API.SpringBoard alerts (e.g. "Open in …?") on a turned device report their children in unrotated portrait points under a root whose frame is the turned screen, so those must not be rotated.
We work around this on the client side for now (rotate the point ourselves and send a plain normalized tap). Happy to test a fix.