diff --git a/.changeset/browser-module.md b/.changeset/browser-module.md new file mode 100644 index 00000000..66d8a0ec --- /dev/null +++ b/.changeset/browser-module.md @@ -0,0 +1,5 @@ +--- +"@cloudflare/computer": minor +--- + +Add `createBrowserModule()` from `@cloudflare/computer/modules/browser`, the `codemode:browser` host module: an agents `Browser` driven from isolate JavaScript with the same `browser.*` functions as `browserTool`. Each run opens one session in its browser scope, runs in one browser scope take turns, and the session's notices go to the run's stderr when it ends. It is experimental. diff --git a/docs/17_isolate_javascript.md b/docs/17_isolate_javascript.md index 1ab042c8..bc4aad93 100644 --- a/docs/17_isolate_javascript.md +++ b/docs/17_isolate_javascript.md @@ -276,7 +276,7 @@ Caller source can import four kinds of module, and all of them are fixed when th | Built in | Always installed | The isolate, backed by the Workspace | `node:fs`, `node:fs/promises` | | Node.js | `nodejs_compat` in `compatibilityFlags`, the default | The isolate, provided by the runtime | `node:path`, `node:crypto` | | Source | `modules: { name: "source" }` | The isolate | a bundled library | -| Host | `modules: { "codemode:name": { fn } }`, a factory, or `createCodemodeModule()` | The Durable Object | `codemode:git`, `codemode:container`, `codemode:tools`, your own | +| Host | `modules: { "codemode:name": { fn } }`, a factory, or `createCodemodeModule()` | The Durable Object | `codemode:git`, `codemode:container`, `codemode:tools`, `codemode:browser`, your own | ```ts import { createArtifactsModule } from "@cloudflare/computer/modules/artifacts"; @@ -675,6 +675,43 @@ createContainerModule({ }); ``` +### `codemode:browser` + +`createBrowserModule()` from `@cloudflare/computer/modules/browser` gives JavaScript the agent's persistent browser: an agents `Browser` (from `agents/browser`, version 0.33.0 or later). The exports are the same `browser.*` functions `browserTool` gives its code, so code imports the module as a namespace and reads the same: + +```js +import * as browser from "codemode:browser"; + +export default async function () { + await browser.cdp({ method: "Page.navigate", params: { url: "https://example.com" }, sessionId: "active" }); + const { result } = await browser.cdp({ + method: "Runtime.evaluate", + params: { expression: "document.title", returnByValue: true }, + sessionId: "active", + }); + return result.value; +} +``` + +The module keeps its browser session in a scoped context. A run opens one browser session on its first call to the module and closes it when the run waits in `ctx.interrupt()` or ends, which saves its active tab, so the next run, or the next `browserTool` call, picks up where it left off. When the run ends, the session's notices, such as "the browser was restarted", are written to its stderr after its output, along with how long it waited for its turn if that was a second or more. Nothing reaches stderr while the code runs, so replay's output check holds. Every function takes one input object. `cdp` is marked `unsafeToRepeat()`, since a command may submit a form; the rest are safe to repeat. The browser keeps its logins and can change things on the sites it visits, so every function refuses to run on a read-only backend, as `codemode:container`'s `exec` does. + +The browser scope says which tab a run drives: runs in one browser scope share an active tab, while cookies and logins are shared by every browser scope. Without a `scope` option, every run uses the `"shared"` browser scope. Pass a string, such as `scope: "research"`, to use another, or a function that picks one from the run's `{ id, metadata }` (see [Run metadata](./09_tool_interface.md#run-metadata) for how a run gets its metadata): + +```ts +createBrowserModule({ + browser: this.browser, + scope: ({ metadata }) => (metadata.user ? `user:${metadata.user}` : "shared"), +}); +``` + +The function may throw instead, when a run without that metadata shouldn't browse. An empty browser scope is rejected. + +Runs in one browser scope take turns, like `browserTool` calls, because they would otherwise see each other's navigation: pi and the AI SDK run a message's tool calls in parallel. A run takes its turn at its first browser call and keeps it until it ends, and runs that never call the module never wait. A run gives up after `lockTimeoutMs` (45 seconds by default) with an error saying why; keep it under the run's timeout and the backend's `maxHostCallMs`. A run that browsed and is waiting for a person keeps its browser scope while it waits: its session is closed, and other runs' first calls in that browser scope fail at once, saying the browser is held by a run that's waiting for a person. A run that took its turn while one ahead of it paused fails the same way. The module remembers which runs browsed in memory, so after the Durable Object restarts, a waiting run holds nothing until it resumes. Once that run resumes, it takes its turn again only at its first new browser call, since replay answers its earlier calls from the journal. + +Don't expose `browserTool` to code through `codemode:tools` alongside this module. A run calling it would wait for its own turn until the host call timed out. The module's description says so, and also tells the model to save screenshots to a file with `node:fs` and open them with `read`, to take about eight per run (each host call returns at most 1 MiB, and a run's calls share 8 MiB each way), and to work on several pages inside one run rather than across runs started together. The backend adds that description to its own, which the `exec` tool shows. An app that writes `exec`'s description itself must include it, from `backend.description`. + +The module is experimental and may change in any release. + ## Isolation and lifecycle Each execution receives a fresh Dynamic Worker with: diff --git a/packages/computer/README.md b/packages/computer/README.md index 56854ab1..806ba270 100644 --- a/packages/computer/README.md +++ b/packages/computer/README.md @@ -34,6 +34,7 @@ removed in a future release; each one names its replacement. | Resuming interrupted isolate JavaScript: `onInterrupt: "restart"` and `"replay"`, `workspace.runtime.recover()`, `unsafeToRepeat()`, and waiting for answers with `ctx.interrupt()`, `pending()` and `respond()` | `@cloudflare/computer`, `@cloudflare/computer/backends/worker-javascript` | Experimental | | `WorkerShellBackend` and the shell command groups | `@cloudflare/computer/backends/worker-shell`, `@cloudflare/computer/shell/*` | Experimental | | The `codemode:container` host module | `@cloudflare/computer/modules/container` | Experimental | +| The `codemode:browser` host module | `@cloudflare/computer/modules/browser` | Experimental | | Codemode modules: `createCodemodeModule()`, with objects and scoped context | `@cloudflare/computer`, `@cloudflare/computer/backends/worker-javascript` | Experimental | | `workspace.push()`, `workspace.pull()`, and the sync protocol between the durable object and `computerd` | `@cloudflare/computer` | Experimental | | The `computerd` daemon and its container image | `ghcr.io/cloudflare/computer-computerd-linux-x64` | Experimental | @@ -517,6 +518,7 @@ on a computerd instance. | `@cloudflare/computer/modules/artifacts` | `createArtifactsModule()` for `codemode:artifacts`: Artifacts and git tokens from isolate JavaScript. | | `@cloudflare/computer/modules/assets` | `createAssetsModule()` for `codemode:assets`: publish a Workspace file from isolate JavaScript. | | `@cloudflare/computer/modules/tools` | `createToolBindings()` for `codemode:tools`: an agent's own tools, callable from isolate JavaScript. | +| `@cloudflare/computer/modules/browser` | `createBrowserModule()` for `codemode:browser`: an agents `Browser`, driven from isolate JavaScript. | | `@cloudflare/computer/tools` | AI SDK tools for agents: `read`, `ls`, `find`, `grep`, `write`, `edit`, `delete`, and optional `exec` and `publish`. | | `@cloudflare/computer/tools/core` | `defineExec()`: the `exec` tool's description, schema, and executor, with no agent library. | | `@cloudflare/computer/tools/ai-sdk` | `createAITools()`: the AI SDK tool set for a Workspace, and `forAISDKTools()` to serve AI SDK tools as `codemode:tools`. | diff --git a/packages/computer/package.json b/packages/computer/package.json index 9f1ae707..0334e3d1 100644 --- a/packages/computer/package.json +++ b/packages/computer/package.json @@ -51,6 +51,10 @@ "types": "./dist/modules/assets.d.ts", "import": "./dist/modules/assets.js" }, + "./modules/browser": { + "types": "./dist/modules/browser.d.ts", + "import": "./dist/modules/browser.js" + }, "./tools": { "types": "./dist/tools/index.d.ts", "import": "./dist/tools/index.js" diff --git a/packages/computer/src/modules/browser.test.ts b/packages/computer/src/modules/browser.test.ts new file mode 100644 index 00000000..3b3d2c21 --- /dev/null +++ b/packages/computer/src/modules/browser.test.ts @@ -0,0 +1,447 @@ +import { describe, expect, it } from "vitest"; + +import { parseModules } from "../backends/worker-javascript/module-graph"; +import { + type CodemodeModule, + codemodeModuleDefinition, + type WorkspaceModuleScope, + type WorkspaceModuleScopeEnd, +} from "../runtime/codemode-module"; +import { isUnsafeToRepeat } from "../runtime/replay-safety"; +import type { + WorkspaceModuleCallContext, + WorkspaceModuleFunction, + WorkspaceModuleHost, + WorkspaceRuntimePendingCall, +} from "../runtime/types"; +import { + type BrowserModuleSession, + type BrowserModuleSource, + createBrowserModule, +} from "./browser"; + +// A stand-in for an agents `Browser`, shaped like the parts the module +// uses. `busy` makes every lock wait until its signal aborts; `gate` +// makes it wait until the gate opens. +class FakeBrowser implements BrowserModuleSource { + readonly sessions: FakeSession[] = []; + busy = false; + gate: Promise | undefined; + waitedForLockMs = 0; + notices: string[] = []; + readonly calls: { name: string; input: unknown }[] = []; + readonly functionNames: string[]; + + constructor(functionNames = ["cdp", "attachToTarget", "spec", "getDebugLog", "clearDebugLog"]) { + this.functionNames = functionNames; + } + + functionDocs() { + return { + functions: "- `browser.cdp({ method })` sends a CDP command.", + rules: ["The browser persists between runs."], + }; + } + + session(options: { scope: string; lock?: boolean }): FakeSession { + const session = new FakeSession(this, options); + this.sessions.push(session); + return session; + } +} + +class FakeSession implements BrowserModuleSession { + closed = false; + locked = false; + readonly functions: Readonly Promise>>; + + constructor( + private readonly browser: FakeBrowser, + readonly options: { scope: string; lock?: boolean }, + ) { + this.functions = Object.fromEntries( + browser.functionNames.map((name) => [ + name, + async (input: unknown) => { + browser.calls.push({ name, input }); + return { name }; + }, + ]), + ); + } + + get waitedForLockMs() { + return this.browser.waitedForLockMs; + } + + lock({ signal }: { signal?: AbortSignal }): Promise { + const { gate } = this.browser; + if (gate !== undefined) { + return gate.then(() => { + this.locked = true; + }); + } + if (!this.browser.busy) { + this.locked = true; + return Promise.resolve(); + } + if (signal?.aborted) return Promise.reject(signal.reason); + return new Promise((_, reject) => { + signal?.addEventListener("abort", () => reject(signal.reason), { once: true }); + }); + } + + async close() { + this.closed = true; + return this.locked ? { notices: this.browser.notices } : undefined; + } +} + +// `pending` is read on every call, so a test can change it between checks. +function hostWith(pending: Partial[] = []): WorkspaceModuleHost { + // SAFETY: The module only reads `runtime.pending()` from its host. + return { + runtime: { pending: async () => [...pending] as WorkspaceRuntimePendingCall[] }, + } as unknown as WorkspaceModuleHost; +} + +function pausedEnd(): WorkspaceModuleScopeEnd { + return { reason: "paused", signal: new AbortController().signal, deadline: Date.now() + 60_000 }; +} + +function runScope( + overrides: Partial = {}, +): WorkspaceModuleScope & { abort: AbortController } { + const abort = new AbortController(); + return { id: "run-1", metadata: {}, signal: abort.signal, abort, ...overrides }; +} + +function callContext( + access: WorkspaceModuleCallContext["access"] = "read-write", +): WorkspaceModuleCallContext { + return { + signal: new AbortController().signal, + deadline: Date.now() + 60_000, + access, + callId: "test-call", + execution: { id: "run-1", metadata: {} }, + interrupt: () => Promise.reject(new Error("not used")), + resolvePath: async (path) => path, + }; +} + +function terminalEnd(): WorkspaceModuleScopeEnd & { logged: string[] } { + const logged: string[] = []; + return { + reason: "completed", + signal: new AbortController().signal, + deadline: Date.now() + 60_000, + log: (text) => logged.push(text), + logged, + }; +} + +// The session `start` opened, after the one read for the export names. +function runSession(browser: FakeBrowser): FakeSession { + const session = browser.sessions.at(-1); + if (session === undefined || session === browser.sessions[0]) throw new Error("no run session"); + return session; +} + +// What #293's start(run, host) gave: start the module's scoped +// "session" context for one scope, and call functions with it in ctx. +async function startScope( + module: CodemodeModule, + run: WorkspaceModuleScope, + host: WorkspaceModuleHost, +) { + const definition = codemodeModuleDefinition(module); + const context = definition?.contexts.find((item) => item.scoped && item.name === "session"); + if (definition === undefined || context === undefined) + throw new Error("no scoped session context"); + const create = context.create as unknown as ( + run: WorkspaceModuleScope, + host: WorkspaceModuleHost, + ) => Promise; + const session = await create(run, host); + const functions: Record = {}; + for (const [name, fn] of Object.entries(definition.functions)) { + functions[name] = (args, ctx) => + fn(args, { ...ctx, session: Promise.resolve(session) } as WorkspaceModuleCallContext); + } + const dispose = context.dispose as unknown as ( + value: unknown, + end: WorkspaceModuleScopeEnd, + ) => Promise; + return { functions, close: (end: WorkspaceModuleScopeEnd) => dispose(session, end) }; +} + +describe("createBrowserModule", () => { + it("exports the browser's functions and describes them with the Workspace rules", () => { + const browser = new FakeBrowser([ + "cdp", + "attachToTarget", + "spec", + "getDebugLog", + "clearDebugLog", + "listHosts", + ]); + const module = createBrowserModule({ browser }); + + expect(module.exports).toEqual([ + "cdp", + "attachToTarget", + "spec", + "getDebugLog", + "clearDebugLog", + "listHosts", + ]); + // Reading the names opens no locking session. + expect(browser.sessions.map((session) => session.options.lock)).toEqual([undefined]); + expect(module.description).toContain("- `browser.cdp({ method })` sends a CDP command."); + expect(module.description).toContain("- The browser persists between runs."); + expect(module.description).toContain('import * as browser from "codemode:browser"'); + expect(module.description).toContain("written to stderr after the run ends"); + expect(module.description).toContain("open the file with `read`"); + expect(module.description).toContain("about eight screenshots per run"); + expect(module.description).toContain("Target.createTarget"); + expect(module.description).toContain("Don't call the browser tool from code"); + // The backend lists it under its specifier, with its lines indented under that bullet. + expect(parseModules({ "codemode:browser": module }).description).toContain( + "- `codemode:browser`: A persistent Chrome browser, driven over the Chrome DevTools Protocol (CDP). It keeps its tabs, cookies and logins between runs.\n\n - `browser.cdp({ method })`", + ); + }); + + it('drives the "shared" browser scope by default, a string, or the one a function picks', async () => { + const browser = new FakeBrowser(); + await startScope(createBrowserModule({ browser }), runScope(), hostWith()); + expect(runSession(browser).options).toEqual({ scope: "shared", lock: true }); + + await startScope(createBrowserModule({ browser, scope: "team" }), runScope(), hostWith()); + expect(runSession(browser).options.scope).toBe("team"); + + const seen: unknown[] = []; + await startScope( + createBrowserModule({ + browser, + scope: (execution) => { + seen.push(execution); + return `conversation:${execution.metadata.conversation}`; + }, + }), + runScope({ metadata: { conversation: "7" } }), + hostWith(), + ); + expect(runSession(browser).options.scope).toBe("conversation:7"); + expect(seen).toEqual([{ id: "run-1", metadata: { conversation: "7" } }]); + }); + + it("rejects an empty browser scope", async () => { + const browser = new FakeBrowser(); + expect(() => createBrowserModule({ browser, scope: "" })).toThrow(/scope must not be empty/); + + const module = createBrowserModule({ browser, scope: () => "" }); + await expect(startScope(module, runScope(), hostWith())).rejects.toThrow( + /scope function returned "" for run run-1/, + ); + }); + + it("fails at once when a run that browsed and is waiting for a person holds the browser scope", async () => { + const browser = new FakeBrowser(); + const module = createBrowserModule({ + browser, + scope: ({ metadata }) => { + if (metadata.conversation === undefined) throw new Error("no conversation"); + return `conversation:${metadata.conversation}`; + }, + }); + const run0 = runScope({ id: "run-0", metadata: { conversation: "7" } }); + const held = await startScope(module, run0, hostWith()); + await held.close(pausedEnd()); + const run = runScope({ metadata: { conversation: "7" } }); + const sessions = browser.sessions.length; + + await expect( + startScope( + module, + run, + hostWith([{ id: "run-0", metadata: { conversation: "7" }, since: 0 }]), + ), + ).rejects.toThrow(/held by a run that's waiting for a person \(run run-0\)/); + expect(browser.sessions).toHaveLength(sessions); + + // A waiting run that never browsed, and the run itself, don't hold it. + await startScope( + module, + run, + hostWith([ + { id: "run-9", metadata: { conversation: "7" }, since: 0 }, + { id: "run-1", metadata: { conversation: "7" }, since: 0 }, + ]), + ); + expect(runSession(browser).options.scope).toBe("conversation:7"); + }); + + it("doesn't count a run that browsed once it has ended", async () => { + const browser = new FakeBrowser(); + const module = createBrowserModule({ browser }); + const run0 = await startScope(module, runScope({ id: "run-0" }), hostWith()); + await run0.close(terminalEnd()); + + await startScope(module, runScope(), hostWith([{ id: "run-0", metadata: {}, since: 0 }])); + expect(runSession(browser).locked).toBe(true); + }); + + it("doesn't count a wait that started after the run's session closed", async () => { + const browser = new FakeBrowser(); + const module = createBrowserModule({ browser }); + const run0 = await startScope(module, runScope({ id: "run-0" }), hostWith()); + await run0.close(pausedEnd()); + + // run-0 resumed, ended without browsing again, and a new run-0 is + // waiting; or run-0 itself is waiting again from a scope that didn't + // browse. Either way the wait began after the session closed. + const later = Date.now() + 1_000; + await startScope(module, runScope(), hostWith([{ id: "run-0", metadata: {}, since: later }])); + expect(runSession(browser).locked).toBe(true); + }); + + it("drops a claim once its run is no longer waiting", async () => { + const browser = new FakeBrowser(); + const module = createBrowserModule({ browser }); + const run0 = await startScope(module, runScope({ id: "run-0" }), hostWith()); + await run0.close(pausedEnd()); + + // A check while run-0 isn't waiting forgets it, so a wait from an + // earlier time no longer holds the scope. + await startScope(module, runScope({ id: "run-1" }), hostWith()); + await startScope( + module, + runScope({ id: "run-2" }), + hostWith([{ id: "run-0", metadata: {}, since: 0 }]), + ); + expect(runSession(browser).locked).toBe(true); + }); + + it("checks again once it has its turn, in case the run ahead paused meanwhile", async () => { + const browser = new FakeBrowser(); + const module = createBrowserModule({ browser }); + const run0 = await startScope(module, runScope({ id: "run-0" }), hostWith()); + + let open!: () => void; + browser.gate = new Promise((resolve) => (open = resolve)); + const pending: Partial[] = []; + const queued = startScope(module, runScope(), hostWith(pending)); + await new Promise((resolve) => setTimeout(resolve, 0)); + // run-0 marks itself waiting, then releases the lock. + pending.push({ id: "run-0", metadata: {}, since: 0 }); + await run0.close(pausedEnd()); + open(); + + await expect(queued).rejects.toThrow(/waiting for a person \(run run-0\)/); + expect(runSession(browser).closed).toBe(true); + }); + + it("gives up waiting for its turn with its own message, and closes the session", async () => { + const browser = new FakeBrowser(); + browser.busy = true; + const module = createBrowserModule({ browser, lockTimeoutMs: 10 }); + + await expect(startScope(module, runScope(), hostWith())).rejects.toThrow( + /gave up waiting for the browser after 0s: other runs in browser scope "shared" were using it/, + ); + expect(runSession(browser).closed).toBe(true); + }); + + it("stops waiting with the run's own reason when the run is cancelled", async () => { + const browser = new FakeBrowser(); + browser.busy = true; + const run = runScope(); + const started = startScope(createBrowserModule({ browser }), run, hostWith()); + await Promise.resolve(); + run.abort.abort(new Error("cancelled")); + + await expect(started).rejects.toThrow("cancelled"); + expect(runSession(browser).closed).toBe(true); + }); + + it("passes one input object to the session's functions and marks cdp unsafe to repeat", async () => { + const browser = new FakeBrowser(); + const run = runScope(); + const module = createBrowserModule({ browser }); + const { functions } = await startScope(module, run, hostWith()); + + expect(Object.keys(functions)).toEqual(module.exports); + await expect(functions.cdp?.([{ method: "Page.enable" }], callContext())).resolves.toEqual({ + name: "cdp", + }); + await functions.spec?.([], callContext()); + expect(browser.calls).toEqual([ + { name: "cdp", input: { method: "Page.enable" } }, + { name: "spec", input: undefined }, + ]); + await expect(functions.cdp?.([{}, {}], callContext())).rejects.toThrow( + /takes one input object/, + ); + + const exported = codemodeModuleDefinition(module)?.functions ?? {}; + expect(isUnsafeToRepeat(exported.cdp)).toBe(true); + for (const name of ["attachToTarget", "spec", "getDebugLog", "clearDebugLog"]) { + expect(isUnsafeToRepeat(exported[name])).toBe(false); + } + + expect(browser.calls).toHaveLength(2); + }); + + it("refuses every function on a read-only backend, before opening a session", async () => { + const browser = new FakeBrowser(); + const module = createBrowserModule({ browser }); + const exported = codemodeModuleDefinition(module)?.functions ?? {}; + // No ctx.session: reading it would throw, so the check comes first. + for (const name of module.exports) { + await expect(exported[name]?.([{}], callContext("read"))).rejects.toThrow( + `codemode:browser ${name} requires Workspace write access.`, + ); + } + expect(browser.sessions).toHaveLength(1); + }); + + it("closes the session when the run pauses, without logging", async () => { + const browser = new FakeBrowser(); + browser.notices = ["The browser was restarted."]; + const instance = await startScope(createBrowserModule({ browser }), runScope(), hostWith()); + + await instance.close({ + reason: "paused", + signal: new AbortController().signal, + deadline: Date.now() + 60_000, + }); + expect(runSession(browser).closed).toBe(true); + }); + + it("logs the session's notices and a noticeable wait when the run ends", async () => { + const browser = new FakeBrowser(); + browser.notices = ["The browser was restarted.", "New tabs opened by the page: []"]; + browser.waitedForLockMs = 2_400; + const instance = await startScope(createBrowserModule({ browser }), runScope(), hostWith()); + + const end = terminalEnd(); + await instance.close(end); + expect(runSession(browser).closed).toBe(true); + expect(end.logged).toEqual([ + "The browser was restarted.\n", + "New tabs opened by the page: []\n", + "This run waited 2s for its turn with the browser, while other runs in its browser scope used it.\n", + ]); + }); + + it("doesn't mention a short wait", async () => { + const browser = new FakeBrowser(); + browser.waitedForLockMs = 300; + const instance = await startScope(createBrowserModule({ browser }), runScope(), hostWith()); + + const end = terminalEnd(); + await instance.close(end); + expect(end.logged).toEqual([]); + }); +}); diff --git a/packages/computer/src/modules/browser.ts b/packages/computer/src/modules/browser.ts new file mode 100644 index 00000000..3792930e --- /dev/null +++ b/packages/computer/src/modules/browser.ts @@ -0,0 +1,268 @@ +// `codemode:browser`: an agent's persistent browser for isolate JavaScript. +// +// import * as browser from "codemode:browser"; +// await browser.cdp({ method: "Page.navigate", params: { url }, sessionId: "active" }); +// +// The browser is an agents `Browser`, and the exports are its `browser.*` +// functions, the same ones `browserTool` gives its code. Each run that +// calls the module opens one session on its first call and closes it +// when the run pauses or ends, which saves the run's active tab for the +// next run in its browser scope. +// +// Computer takes no dependency on agents, so the parts of the contract +// this module uses are typed here by shape. +// +// "Scope" means two things here. The browser scope is which tab a +// session drives ("shared", "conversation:"), as agents names it. +// A module scope is Computer's: one stretch of a run between starting +// in an isolate and pausing or ending, which `start` is called for. + +import { + type CodemodeModule, + createCodemodeModule, + type WorkspaceModuleScope, +} from "../runtime/codemode-module"; +import { unsafeToRepeat } from "../runtime/replay-safety"; +import type { + WorkspaceModuleExecution, + WorkspaceModuleFunction, + WorkspaceModuleHost, +} from "../runtime/types"; + +/** The agents `Browser`, as this module uses it. */ +export interface BrowserModuleSource { + /** The `browser.*` functions in Markdown, and the rules for using them. */ + functionDocs(): { readonly functions: string; readonly rules: readonly string[] }; + /** Open a session that connects on its first call. */ + session(options: { scope: string; lock?: boolean }): BrowserModuleSession; +} + +/** An agents `BrowserSession`, as this module uses it. */ +export interface BrowserModuleSession { + /** The `browser.*` functions bound to the session, by name. */ + readonly functions: Readonly Promise>>; + /** Take the browser scope's lock; rejects with the signal's reason when it aborts. */ + lock(options: { signal?: AbortSignal }): Promise; + /** How long the session waited for its lock, in milliseconds. */ + readonly waitedForLockMs: number; + /** Save the active tab, release the lock, and report what happened to the browser. */ + close(): Promise<{ readonly notices: readonly string[] } | undefined>; +} + +/** Options for {@link createBrowserModule}. */ +export interface BrowserModuleOptions { + /** The agent's `Browser`, from `agents/browser`. */ + readonly browser: BrowserModuleSource; + /** + * The browser scope a run drives: runs in one browser scope share an + * active tab and take turns. A string, or a function that picks one + * from the run's `id` and `metadata`. Defaults to `"shared"`. A + * function may throw when a run without the metadata it needs + * shouldn't browse. + */ + readonly scope?: string | ((execution: WorkspaceModuleExecution) => string); + /** + * How long a run waits for its turn with the browser before it gives + * up, in milliseconds. Defaults to 45000 (45 seconds). Keep it under + * the run's timeout and the backend's `maxHostCallMs`, so the run + * gets this module's error rather than timing out. + */ + readonly lockTimeoutMs?: number; +} + +const DEFAULT_BROWSER_SCOPE = "shared"; +const DEFAULT_LOCK_TIMEOUT_MS = 45_000; +// A run that waited less than this for its turn isn't told about it. +const NOTICEABLE_WAIT_MS = 1_000; + +const WORKSPACE_RULES = [ + 'Import the module as a namespace, `import * as browser from "codemode:browser"`, and call `browser.({ ... })` as above.', + "Notices about the browser (it restarted, your tab was closed, the page opened tabs) are written to stderr after the run ends.", + 'To look at a page, save a screenshot and open the file with `read`: `const { data } = await browser.cdp({ method: "Page.captureScreenshot", sessionId: "active" })`, then `await fs.writeFile(path, Uint8Array.from(atob(data), (c) => c.charCodeAt(0)))` with `node:fs/promises`.', + "One call returns at most 1 MiB, and a run's calls share 8 MiB each way, so take about eight screenshots per run at most, and print PDFs in a run of their own.", + "Runs that use the browser take turns, even when started together: a run waits until the runs ahead of it end. To work on several pages, do it in one run: open a tab per page with Target.createTarget and drive them with Promise.all.", + "Don't call the browser tool from code through `codemode:tools`: it waits for this run's turn, which only ends when this run does.", +]; + +/** + * Build the `codemode:browser` host module over an agents `Browser`. + * + * It exports the browser's `browser.*` functions. Each run that calls + * one opens a session in its browser scope, waits for its turn with + * the browser, and closes the session when it pauses or ends; when it + * ends, the session's notices go to its stderr. A run fails at once if + * a run waiting for an answer holds its browser scope. Every function + * needs Workspace write access, since browsing can change things on the + * sites the browser is logged in to. + * + * @param options - The browser, the browser scope, and how long to wait for a turn. + * @returns The module to pass as `modules["codemode:browser"]`. + * @throws When `scope` is an empty string. + */ +export function createBrowserModule(options: BrowserModuleOptions): CodemodeModule { + const { browser } = options; + const scope = options.scope ?? DEFAULT_BROWSER_SCOPE; + const lockTimeoutMs = options.lockTimeoutMs ?? DEFAULT_LOCK_TIMEOUT_MS; + if (scope === "") throw new Error("codemode:browser: scope must not be empty."); + const browserScopeOf = ({ id, metadata }: WorkspaceModuleExecution): string => { + const value = typeof scope === "function" ? scope({ id, metadata }) : scope; + if (typeof value !== "string" || value === "") { + throw new Error( + `codemode:browser: the scope function returned ${JSON.stringify(value)} for run ${id}; a browser scope must be a non-empty string.`, + ); + } + return value; + }; + + // The function table depends only on the browser, not the session. A + // session connects on its first call and, without `lock`, holds + // nothing until then, so reading its table needs no close. + const exportNames = Object.keys(browser.session({ scope: DEFAULT_BROWSER_SCOPE }).functions); + + const docs = browser.functionDocs(); + // The backend lists each module as one bullet, so every line after + // the first is indented under it. + const description = [ + "A persistent Chrome browser, driven over the Chrome DevTools Protocol (CDP). It keeps its tabs, cookies and logins between runs.", + "", + docs.functions, + "", + "Rules:", + ...[...docs.rules, ...WORKSPACE_RULES].map((rule) => `- ${rule}`), + ] + .join("\n") + .split("\n") + .map((line, index) => (index === 0 || line === "" ? line : ` ${line}`)) + .join("\n"); + + const module = createCodemodeModule({ description }); + + // Runs that paused with a session open, by run id: the browser scope + // and when the session closed. This lives in memory, so after the + // Durable Object restarts a waiting run holds nothing until it resumes + // and browses again. + const pausedWithSession = new Map(); + const sessionRuns = new WeakMap(); + + // One session per module scope, opened by the first live call that + // reads ctx.session. Calls racing it share the start; a start that + // throws stays failed for the scope. + module.context( + "session", + async (run: WorkspaceModuleScope, host: WorkspaceModuleHost) => { + const browserScope = browserScopeOf(run); + await assertNotHeldByPausedRun(host, run, browserScope, pausedWithSession); + + const session = browser.session({ scope: browserScope, lock: true }); + const timeout = AbortSignal.timeout(lockTimeoutMs); + try { + await session.lock({ signal: AbortSignal.any([run.signal, timeout]) }); + } catch (error) { + await session.close(); + if (timeout.aborted && !run.signal.aborted) { + throw new Error( + `codemode:browser: gave up waiting for the browser after ${Math.round(lockTimeoutMs / 1000)}s: other runs in browser scope ${JSON.stringify(browserScope)} were using it. Runs that use the browser take turns, so try again once they finish.`, + ); + } + throw error; + } + // The run that had the lock may have paused for a person while + // this one waited. A run marks itself waiting before it releases + // the lock, so checking again now can't miss it. + try { + await assertNotHeldByPausedRun(host, run, browserScope, pausedWithSession); + } catch (error) { + await session.close(); + throw error; + } + sessionRuns.set(session, { id: run.id, browserScope }); + return session; + }, + { + scoped: true, + async dispose(session, end) { + const owner = sessionRuns.get(session); + if (owner !== undefined) { + // Claimed before close releases the lock, so a run that gets + // the lock from this close sees the claim when it checks. + if (end.reason === "paused") { + pausedWithSession.set(owner.id, { + browserScope: owner.browserScope, + closedAt: Date.now(), + }); + } else { + pausedWithSession.delete(owner.id); + } + } + const report = await session.close(); + if (end.reason === "paused") return; + for (const notice of report?.notices ?? []) end.log(`${notice}\n`); + const waited = session.waitedForLockMs; + if (waited >= NOTICEABLE_WAIT_MS) { + end.log( + `This run waited ${Math.round(waited / 1000)}s for its turn with the browser, while other runs in its browser scope used it.\n`, + ); + } + }, + }, + ); + + const functions: Record = {}; + for (const name of exportNames) { + functions[name] = async (args, ctx) => { + if (ctx.access !== "read-write") { + throw new Error(`codemode:browser ${name} requires Workspace write access.`); + } + if (args.length > 1) { + throw new TypeError(`browser.${name}(input) takes one input object.`); + } + const session = await (ctx as typeof ctx & { session: Promise }) + .session; + const fn = session.functions[name]; + if (fn === undefined) throw new Error(`browser.${name} is not available.`); + return fn(args[0]); + }; + } + // A repeated CDP command can submit a form or buy something twice. + if (functions.cdp) unsafeToRepeat(functions.cdp); + + return module.module(functions); +} + +interface PausedClaim { + readonly browserScope: string; + /** When the run's session closed for the pause, in epoch milliseconds. */ + readonly closedAt: number; +} + +// A run waiting for an answer has closed its session, so it no longer +// holds the lock, but its browser scope's tab is where it left off. +// Another run there would move it, so it fails rather than waits: the +// answer may take hours. +// +// A waiting run holds its browser scope only if its session closed for +// this wait: a run marks itself waiting before its session closes, so +// a wait that started later, from a scope that didn't browse or a run +// that reused the id, holds nothing. Claims for runs no longer waiting +// are dropped here, since only a waiting run can hold a scope. +async function assertNotHeldByPausedRun( + host: WorkspaceModuleHost, + run: WorkspaceModuleScope, + browserScope: string, + pausedWithSession: Map, +) { + const pending = await host.runtime.pending(); + const waiting = new Set(pending.map((call) => call.id)); + for (const id of pausedWithSession.keys()) { + if (!waiting.has(id)) pausedWithSession.delete(id); + } + for (const call of pending) { + if (call.id === run.id) continue; + const claim = pausedWithSession.get(call.id); + if (claim?.browserScope === browserScope && call.since <= claim.closedAt) { + throw new Error( + `codemode:browser: browser scope ${JSON.stringify(browserScope)} is held by a run that's waiting for a person (run ${call.id}). Try again once it has its answer.`, + ); + } + } +} diff --git a/packages/computer/tests/script-runner-worker.ts b/packages/computer/tests/script-runner-worker.ts index 0f476edd..5d763fca 100644 --- a/packages/computer/tests/script-runner-worker.ts +++ b/packages/computer/tests/script-runner-worker.ts @@ -19,6 +19,7 @@ import { Workspace, } from "../src/index"; import { createArtifactsModule } from "../src/modules/artifacts"; +import { type BrowserModuleSession, createBrowserModule } from "../src/modules/browser"; import { createContainerModule } from "../src/modules/container"; import { createGitModule } from "../src/modules/git"; @@ -133,6 +134,37 @@ const testSession = sessionModule.module({ (await (ctx as unknown as { session: Promise<{ instance: number }> }).session).instance, }); +// A stand-in for an agents Browser. Each browser scope's sessions, +// locks, calls and closes go in sessionEvents under "browser:". +const fakeBrowser = { + functionDocs: () => ({ functions: "- `browser.cdp({ method })`", rules: [] }), + session({ scope, lock }: { scope: string; lock?: boolean }): BrowserModuleSession { + const id = `browser:${scope}`; + let locked = false; + if (lock === true) sessionEvents.push({ id, event: "open" }); + const fn = (name: string) => async (input: unknown) => { + sessionEvents.push({ id, event: `${name} ${JSON.stringify(input ?? null)}` }); + return { name }; + }; + return { + functions: { cdp: fn("cdp"), spec: fn("spec") }, + waitedForLockMs: 0, + async lock() { + locked = true; + sessionEvents.push({ id, event: "lock" }); + }, + async close() { + if (lock === true) sessionEvents.push({ id, event: "close" }); + return locked ? { notices: [`notice from ${scope}`] } : undefined; + }, + }; + }, +}; +const testBrowser = createBrowserModule({ + browser: fakeBrowser, + scope: ({ metadata }) => metadata.browserScope ?? "shared", +}); + // Objects: a job that starts runs. Each call counts itself; a run notes // when it's disposed, under the execution that made it. const jobsModule = createCodemodeModule({ @@ -225,6 +257,7 @@ function javascriptBackend( "codemode:artifacts": createArtifactsModule(), "codemode:container": createContainerModule(), "codemode:test-session": testSession, + "codemode:browser": testBrowser, "codemode:test-jobs": testJobs, "codemode:test-host": { async echo(args) { diff --git a/packages/computer/tests/script-runner.test.ts b/packages/computer/tests/script-runner.test.ts index a4fc619e..0c712264 100644 --- a/packages/computer/tests/script-runner.test.ts +++ b/packages/computer/tests/script-runner.test.ts @@ -1526,6 +1526,86 @@ describe("interrupts", () => { } `; + // codemode:browser on createCodemodeModule(), end to end. + const browsing = ` + import * as browser from "codemode:browser"; + import { approve } from "codemode:test-host"; + export default async function () { + await Promise.all([browser.cdp({ method: "A" }), browser.spec()]); + const answer = await approve("deploy"); + await browser.cdp({ method: "B" }); + return answer; + } + `; + + it("codemode:browser opens one session per scope, closes it on pause, and logs notices at the end", async () => { + await start("browse", browsing, { metadata: { browserScope: "b1" } }); + const call = await waitingCall("browse"); + await vi.waitFor(async () => + expect(await sessionEvents("browser:b1")).toEqual([ + "open", + "lock", + 'cdp {"method":"A"}', + "spec null", + "close", + ]), + ); + await respond("browse", call.callId, { ok: true }); + expect(await result("browse")).toMatchObject({ + status: "completed", + stderr: "notice from b1\n", + }); + // The resumed scope replays A and spec without a session, then + // opens one for B. + expect(await sessionEvents("browser:b1")).toEqual([ + "open", + "lock", + 'cdp {"method":"A"}', + "spec null", + "close", + "open", + "lock", + 'cdp {"method":"B"}', + "close", + ]); + }); + + it("codemode:browser fails a run at once while a run waiting for a person holds its browser scope", async () => { + await start("browse-held", browsing, { metadata: { browserScope: "b2" } }); + const call = await waitingCall("browse-held"); + await start( + "browse-second", + `import * as browser from "codemode:browser"; + export default () => browser.cdp({ method: "C" });`, + { metadata: { browserScope: "b2" } }, + ); + const second = await result("browse-second"); + expect(second.status).toBe("failed"); + // This harness keeps 64 bytes of stdio. + expect(second.stderr).toContain('codemode:browser: browser scope "b2" is held by a run'); + await respond("browse-held", call.callId, { ok: true }); + expect(await result("browse-held")).toMatchObject({ status: "completed" }); + }); + + it("codemode:browser doesn't count a waiting run that never browsed", async () => { + await start( + "approve-only", + `import { approve } from "codemode:test-host"; + export default () => approve("deploy");`, + { metadata: { browserScope: "b3" } }, + ); + const call = await waitingCall("approve-only"); + await start( + "browse-free", + `import * as browser from "codemode:browser"; + export default () => browser.cdp({ method: "D" });`, + { metadata: { browserScope: "b3" } }, + ); + expect(await result("browse-free")).toMatchObject({ status: "completed" }); + await respond("approve-only", call.callId, { ok: true }); + expect(await result("approve-only")).toMatchObject({ status: "completed" }); + }); + it("starts a scoped context once per scope, and resumes only after it closes", async () => { await start("session", session, { metadata: { holdClose: "1", closeLog: "closed" } }); const call = await waitingCall("session"); diff --git a/packages/computer/tsdown.config.ts b/packages/computer/tsdown.config.ts index c44761d4..9b80940d 100644 --- a/packages/computer/tsdown.config.ts +++ b/packages/computer/tsdown.config.ts @@ -39,6 +39,7 @@ export default defineConfig({ "modules/artifacts": "src/modules/artifacts.ts", "modules/tools": "src/modules/tools.ts", "modules/assets": "src/modules/assets.ts", + "modules/browser": "src/modules/browser.ts", "backends/container-legacy/index": "src/backends/container-legacy/index.ts", "backends/container/index": "src/backends/container/index.ts", "backends/worker-javascript/index": "src/backends/worker-javascript/index.ts",