An AI agent that operates a Mac the way a person does. It perceives the screen through the macOS Accessibility tree, decides the next move with Claude, and drives the real, visible cursor along a human-like path, typing with human cadence, instead of teleporting the pointer.
Built in 2026 as a personal project. Fully native Swift, macOS 14+, no Electron, no browser extension.
Surveying computer-use agents in 2026 showed an empty intersection: the real-OS agents teleported the pointer and had no concept of credentials; the tools with real secret vaults were browser-only; and the human-motion libraries had never been wired to an LLM. HumanHands is the glue between those three.
1. Perceive through the Accessibility tree, not screenshots.
The AX tree is local, free and pixel-exact. The agent grounds every action in element geometry, so
screen data never leaves the machine and the loop runs on a Claude subscription through the CLI
(claude setup-token) at zero API cost. Vision is kept only as a concurrent fallback when the tree is
missing or wrong.
2. Learn by compression, not gradients. A hosted model cannot be fine-tuned, and the agent kept stumbling on the same spots. After reading the agent-memory literature (Reflexion, ExpeL, Voyager, Agent Workflow Memory, Skyvern's learn-once-replay), the design became: distil each messy successful run into a clean reusable recipe plus its gotchas, inject that as guidance next time, and once a path proves stable graduate it into a deterministic macro that replays with no model calls and only wakes the model when the screen diverges. A traversed path compressed into a reflex.
3. Keep secrets structurally out of the model. Real credentials live in the Keychain and reach the model only as placeholder tokens that are substituted at the last instant, in the executor. A domain-verified gate refuses to type a secret unless the genuine site is confirmed. User-owned app and site allowlists bound what the agent can touch. TOTP generation, a redaction registry, a tripwire and a kill switch sit in the same ring.
Ring-layered SwiftPM targets; every OS primitive sits behind a protocol so the logic rings test off-device.
| Ring | Target | Role |
|---|---|---|
| 0 | HHCore |
Pure value types, no OS dependencies |
| 0 | HHOSBridge |
OS-touching primitives (CGEvent, AX, ScreenCaptureKit, Keychain) behind protocols |
| 1 | HHExecution |
Humanised executor: WindMouse paths, Fitts' law timing, typing cadence |
| 1 | HHPerception |
Perception geometry, the single coordinate chokepoint |
| 1 | HHSecrets |
Vault, placeholder substitution, TOTP, redaction, tripwire |
| 1 | HHSafety |
Domain-verified site gate |
| 2 | HHSites |
Per-site profiles: allowlist + playbook |
| 2 | HHAgent |
Provider abstraction and the perceive → decide → act loop |
| 3 | HHRuntime |
Shared engine and structured event stream used by both front-ends |
| – | humanhands |
CLI: preflight, capture, run |
| – | HumanHandsApp |
SwiftUI app: a window that runs a goal |
| – | HHTests |
135 pure-logic tests, run off-device |
swift build
swift test # 135 tests, no device access needed
Scripts/build-app.sh # builds and signs HumanHands.app into dist/
# reasoning provider: either a Claude subscription token or an API key
claude setup-token && export ANTHROPIC_AUTH_TOKEN='sk-ant-oat...'
# or
export ANTHROPIC_API_KEY='sk-ant-api...'
.build/debug/humanhands preflight # checks Accessibility and Screen Recording permissions
.build/debug/humanhands run "open Safari and search for the TU/e library opening hours"macOS will ask for Accessibility (and, for the vision fallback, Screen Recording) permission the first time. Nothing runs outside the allowlists you configure.
Working personal tool, not a product. Things that are deliberately not here: cloud relays, telemetry, credential sync, anything that would move screen contents or secrets off the machine.
Author: Pulkit Chawla · linkedin.com/in/pulkitchawla11