From cd45c09a8498ad8bcc954e1189c1cf067eb16a92 Mon Sep 17 00:00:00 2001 From: Wojciech Apanowicz Date: Tue, 6 Oct 2026 09:54:30 +0200 Subject: [PATCH] docs(skills): add deepnote-dynamic-apps and document dataframe pagination Local agents can publish a static site with viewer API access, but no skill said how the page's JavaScript talks to Deepnote. deepnote-dynamic-apps covers the hosted client at /static/app-client/v1.js, what a viewer token can reach, returning a JSON result, and the failures a page has to handle. deepnote-runs now defines dataframe outputs once: rows hold one page (10 by default) while row_count is the total, and how values are encoded on the wire. deepnote-notebooks explains setting pageSize when creating a block, and the get_run entry lists fullOutputs, without which a capped block hides row_count. --- README.md | 3 +- plugins/deepnote/.claude-plugin/plugin.json | 2 +- plugins/deepnote/.codex-plugin/plugin.json | 2 +- .../skills/deepnote-dynamic-apps/SKILL.md | 157 ++++++++++++++++++ plugins/deepnote/skills/deepnote-mcp/SKILL.md | 2 +- .../skills/deepnote-notebooks/SKILL.md | 2 + .../deepnote/skills/deepnote-runs/SKILL.md | 18 +- .../skills/deepnote-static-sites/SKILL.md | 2 +- 8 files changed, 182 insertions(+), 6 deletions(-) create mode 100644 plugins/deepnote/skills/deepnote-dynamic-apps/SKILL.md diff --git a/README.md b/README.md index b01ec8b..9d53744 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Through the hosted MCP server the agent can: - Start notebook runs, optionally with input values or on a chosen machine type, and read run history, status, errors, and snapshot output - Choose a project's environment (Docker image), check whether its machine is running and how big it is, start, stop, or restart it, and change its machine type or idle shutdown - Read Deepnote docs -- Publish a small HTML/CSS/JavaScript site or serve an existing project file as a Streamlit app +- Publish a small HTML/CSS/JavaScript site, optionally one that runs the project's notebooks as its viewer, or serve an existing project file as a Streamlit app ## Good first prompts @@ -67,6 +67,7 @@ Through the hosted MCP server the agent can: - `Attach my Snowflake integration to this project.` - `Serve apps/dashboard.py from this project as a Streamlit app.` - `Copy utils/helpers.py from this project into my Sales Analysis project.` +- `Build a Deepnote dashboard page that runs this notebook and charts the result.` - `Look up the Deepnote docs for scheduled notebooks.` ## Repository layout diff --git a/plugins/deepnote/.claude-plugin/plugin.json b/plugins/deepnote/.claude-plugin/plugin.json index 4387dd3..56fe4a5 100644 --- a/plugins/deepnote/.claude-plugin/plugin.json +++ b/plugins/deepnote/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "deepnote", - "version": "0.6.0", + "version": "0.7.0", "description": "Search Deepnote workspaces, inspect and edit projects and notebooks, manage integrations and project machines, publish static sites and Streamlit apps, generate links, read docs, and run notebooks from Claude Code.", "author": { "name": "Deepnote", diff --git a/plugins/deepnote/.codex-plugin/plugin.json b/plugins/deepnote/.codex-plugin/plugin.json index 1694f41..137c2ae 100644 --- a/plugins/deepnote/.codex-plugin/plugin.json +++ b/plugins/deepnote/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "deepnote", - "version": "0.6.0", + "version": "0.7.0", "description": "Search Deepnote workspaces, inspect and edit projects and notebooks, manage integrations and project machines, publish static sites and Streamlit apps, generate links, read docs, and run notebooks from Codex.", "author": { "name": "Deepnote", diff --git a/plugins/deepnote/skills/deepnote-dynamic-apps/SKILL.md b/plugins/deepnote/skills/deepnote-dynamic-apps/SKILL.md new file mode 100644 index 0000000..2d5bef4 --- /dev/null +++ b/plugins/deepnote/skills/deepnote-dynamic-apps/SKILL.md @@ -0,0 +1,157 @@ +--- +name: deepnote-dynamic-apps +description: Use when writing the JavaScript of a Deepnote static site that runs the project's notebooks as its viewer, reads notebook results in the browser, or shows notebook data in a published dashboard, chart, or table. +--- + +# Deepnote Dynamic Apps + +A dynamic app is a static site whose JavaScript runs notebooks as the signed-in viewer and renders the results. Publishing is in `deepnote-static-sites`. Run inputs and dataframe outputs are defined in `deepnote-runs`. + +The page calls Deepnote from the viewer's browser with a short-lived token that Deepnote gives the viewer. + +## Requirements + +- The project shares its static site with viewer API access enabled: `staticFiles.sharingEnabled` and `staticFiles.apiAccessEnabled` on `update_project`, or `apiAccess: "enabled"` on `publish_static_site`. +- The viewer is signed in to Deepnote and has access to the project. +- The viewer opens the site's canonical URL (`staticFiles.url`). The page gets its token from the Deepnote page that frames it, so it cannot authenticate anywhere else. + +## Browser Client + +Load the client from the origin of the canonical URL, before the page's own scripts: + +```html + +``` + +It defines `window.Deepnote`. If that is missing because the script failed to load, show a message instead of an empty page. + +```js +const deepnote = window.Deepnote.connect({ onAuthExpired: showReloadMessage }) +const notebook = deepnote.notebook(notebookId) + +const inputs = await notebook.inputs() +const { run, result } = await notebook.run(values, { onProgress: (_, phase) => showProgress(phase) }) +const payload = run.status === 'success' ? result('sales-by-region/v1') : null +``` + +- `inputs()` returns the notebook's input blocks with `blockId`, `name`, `type`, `value`, and, where they apply, `label`, `min`, `max`, `step`, `options`, and `multiple`. Key `values` by input `name` and type them as in `deepnote-runs` Run Inputs. +- `run()` starts a detached run with read-only project storage, polls it every 2 seconds, and resolves when the run ends with any status, so check `run.status` and `run.error`. It rejects when a request fails, when `timeoutMs` (default 20 minutes) passes, or when the run succeeded but Deepnote could not save its results. `onProgress` receives `'executing'`, then `'storing-result'` while a successful run's results are saved. +- `result(schema)` returns the first `application/json` output whose `schema` field equals `schema`, or `null`. Always pass the schema: without it, `result()` returns the first JSON output of any block. +- `run.snapshotBlocks` lists every block as `{ id, type, outputs, metadata }`; output data is under `outputs[i].data[mimeType]`. Block source is never included. +- The client requests the token, renews it before it expires, and drops it after a 401 or 403 so the next call requests a new one. If no token arrives within 8 seconds, the call rejects with `Could not reach Deepnote`, `onAuthExpired` runs, and every later call rejects until the page reloads. + +The client ships no type declarations. For a TypeScript page, save these as `deepnote.d.ts`: + +```ts +type DeepnoteInputValue = string | boolean | string[] + +interface DeepnoteInput { + blockId: string + name: string + type: string + value: unknown + label?: string + min?: number + max?: number + step?: number + options?: string[] + multiple?: boolean +} + +interface DeepnoteRun { + runId: string + notebookId: string + status: 'pending' | 'running' | 'success' | 'error' | 'internal_error' | 'stopped' + snapshotStatus?: 'pending' | 'available' | 'unavailable' + snapshotBlocks?: Array<{ + id: string + type: string + outputs: Array<{ data?: Record }> + metadata: Record + }> | null + error?: string | null + createdAt: string + completedAt?: string | null +} + +interface DeepnoteNotebook { + inputs: () => Promise + run: ( + values: Record, + options?: { + onProgress?: (run: DeepnoteRun, phase: 'executing' | 'storing-result') => void + timeoutMs?: number + }, + ) => Promise<{ run: DeepnoteRun; result: (schema?: string) => Record | null }> +} + +interface Window { + Deepnote?: { + connect: (options?: { onAuthExpired?: () => void }) => { notebook: (notebookId: string) => DeepnoteNotebook } + } +} +``` + +## What The Viewer Token Can Reach + +| Request | Allowed | +| --- | --- | +| `GET /v2/notebooks/{notebookId}` | Notebook name, inputs, and block IDs and types; no block source | +| `POST /v2/runs` | Detached runs only; `detached: false`, `blockIds`, and `machineType` are rejected | +| `GET /v2/runs/{runId}` | The viewer's own runs, with `snapshotBlocks` once results are saved; never a raw snapshot or download URL | + +Every other endpoint answers 403, so do not offer notebook listing, run history, or other viewers' runs. The token can access notebooks in the site's project and in other projects of the same workspace where the viewer has direct access; starting runs in other projects also requires execute permission. Deepnote checks the viewer's access on every request, and turning off viewer API access fails the page's next call. + +Use the client. A page that calls the API without it must follow the handshake in the [cloud app example](https://github.com/deepnote/deepnote/tree/main/examples/local-runner/cloud-app), refresh the token through the handshake before its 15-minute expiry and on a 401, and send `detachedRunStorageMode: "readonly"` with every run. + +## Getting Data Into The Page + +### Return A JSON Result + +Have one notebook block display exactly what the page renders, under a schema name of your choice: + +```python +import json +from IPython.display import display + +rows = json.loads(df.to_json(orient='records', date_format='iso')) +display({'application/json': {'schema': 'sales-by-region/v1', 'rows': rows}}, raw=True) +``` + +`to_json` writes missing values as `null` and dates as ISO strings. `df.to_dict('records')` does not: a missing number reaches the page as the string `"nan"`, and a missing timestamp fails the block with `NaTType does not support strftime`. Integers past 2^53 lose precision in the browser's `JSON.parse`, so send them as strings, for example `df.astype({'id': str})`. + +Send only what the page shows and aggregate in the notebook. Deepnote replaces a block's outputs with a short notice when they pass 512 KiB as JSON, or when the notebook's outputs together pass 5 MiB, and `result()` then returns `null`. + +### Read A Dataframe Output + +When the page must read a block's table instead, find it under the `application/vnd.deepnote.dataframe.v3+json` output data and read it as `deepnote-runs` Dataframe Outputs describes: compare `rows.length` with `row_count` before charting it. In JavaScript: + +- In a numeric column, treat `null` and any value for which `Number(v)` is `NaN` as missing; that covers `"nan"`, `""`, and `"inf"`. +- In an integer column with a value past 2^53, keep the strings, which suits IDs, or use `BigInt(v)` for integer math; `Number(v)` rounds them. +- Pandas sends booleans as the strings `"True"` and `"False"`. Parse timestamps explicitly. +- Drop `_deepnote_index_column` from each row before rendering it. + +## Authoring Workflow + +1. Read the notebook with `get_notebook` for its ID and input names. +2. Choose the schema name and the exact JSON shape the page renders, and add or update the one block that displays it. +3. Run it as a viewer would: `create_run` with `detachedRunStorageMode: "readonly"` and the values the page will send, then `get_run` with `snapshotDelivery: "blocks"` and `fullOutputs: true`. Check the JSON before writing the page. +4. Write the page against that shape. Show a default view without a run, run the notebook only on an explicit submission, prevent duplicate submissions, and show loading, empty, and error states. +5. Publish with `deepnote-static-sites` with viewer API access enabled. +6. Open the canonical URL as a signed-in viewer and submit once, or ask the user to. A run through MCP does not test the page's token or rendering. + +## Failures To Handle In The Page + +| Symptom | Cause | Handle it by | +| --- | --- | --- | +| Calls reject after about 8 seconds and `onAuthExpired` runs | Page opened outside the canonical URL, viewer API access off, or the viewer signed out or without access | Message pointing to the canonical URL; reload to retry | +| `run.status` is `error`, `internal_error`, or `stopped` | The notebook failed or was stopped | Show `run.error`, or the status when it is empty | +| `result(schema)` is `null` after `success` | No JSON output with that schema, the output was over the size limit, or `run.snapshotBlocks` is still `null` | Check the notebook's output and the schema string | +| The chart shows 10 points | Reading `rows` of a dataframe output | Return a JSON result, or create the block with a larger page size | +| `run()` rejects with `did not finish in time` | The run took longer than `timeoutMs` | Show progress from `onProgress`; raise `timeoutMs` or speed up the notebook | + +## Safety + +- Never put API keys, personal tokens, `.env` contents, or other secrets in page source. +- Runs start as the viewer. Do not start them on page load, and do not have a page run notebooks with side effects such as sending messages or writing to external systems. +- A notebook the page runs must not depend on writing project files: viewer runs from the client have read-only project storage. diff --git a/plugins/deepnote/skills/deepnote-mcp/SKILL.md b/plugins/deepnote/skills/deepnote-mcp/SKILL.md index 3cced42..74a6afa 100644 --- a/plugins/deepnote/skills/deepnote-mcp/SKILL.md +++ b/plugins/deepnote/skills/deepnote-mcp/SKILL.md @@ -49,7 +49,7 @@ This is the authoritative summary of tool arguments. Workflow skills add only th - `generate_project_url`: return a canonical absolute project or notebook URL. Accepts `projectId`, `notebookId`, or both, and requires at least one; when both are supplied, the notebook must belong to that project. - `create_run`: start a notebook run by `notebookId`. It accepts optional `inputs` keyed by notebook input name, `detached` (default `true`), `detachedRunStorageMode` (`read_write` or `readonly`, detached runs only), `machineType` (an `id` from `list_machine_types`, detached runs only; defaults to the project's machine type), a non-empty unique `blockIds` list (live runs only), and `runDependentBlocks` (requires `blockIds`). For targeted execution, set `detached: false` and pass `blockIds`; omit `blockIds` to run the full notebook. - `list_notebook_runs`: list historical notebook runs newest first, with `pageSize` (default 20) and `pageToken` pagination. Rows carry `runId`, `notebookId`, `status`, `createdAt`, and `completedAt`. -- `get_run`: fetch run `status`, `error`, `createdAt`, `completedAt`, and run snapshots. Optional `snapshotDelivery` is `"downloadUrl"` (the default when omitted), `"inline"`, or `"blocks"`. +- `get_run`: fetch run `status`, `error`, `createdAt`, `completedAt`, and run snapshots. Optional `snapshotDelivery` is `"downloadUrl"` (the default when omitted), `"inline"`, or `"blocks"`; with `"blocks"`, `fullOutputs: true` lifts the 16,384-character cap on block outputs. - `list_machine_types`: list the machine types (sizes) projects can run on, with `id`, `name`, `vCpus`, `memoryGiB`, `gpuCount`, and `available` plus `unavailableReason` for the workspace's plan and free quota. A custom machine (`custom:`) only works for projects in its sandbox environment. - `get_machine`: get the project machine's `status` (`stopped`, `starting`, `running`, `stopping`, or `error`), `error`, machine `type` with its vCPUs, memory, and GPUs, and `idleShutdownSeconds` by `projectId`. This is the machine the editor, live runs, and hosted Streamlit apps use; detached runs use machines of their own. - `start_machine`, `stop_machine`, `restart_machine`: start, stop, or restart the project machine by `projectId`. They return without waiting, so poll `get_machine`. Starting a running machine or stopping a stopped one does nothing. Stopping or restarting clears kernel state for everyone working in the project; restarting recovers a machine in the `error` status. diff --git a/plugins/deepnote/skills/deepnote-notebooks/SKILL.md b/plugins/deepnote/skills/deepnote-notebooks/SKILL.md index 1409918..574d845 100644 --- a/plugins/deepnote/skills/deepnote-notebooks/SKILL.md +++ b/plugins/deepnote/skills/deepnote-notebooks/SKILL.md @@ -95,6 +95,8 @@ A text cell holds one line. A bullet list is one `text-cell-bullet` block per it For SQL blocks, resolve the integration with `list_integrations` when the user names a connection, pass it as top-level `integrationId`, and never put `sql_integration_id` inside `metadata`. Do not pass `integrationId` for non-SQL blocks; it must reference a SQL-capable integration in the same workspace. +A block that ends in a dataframe stores one page of rows in its output, 10 by default (see Dataframe Outputs in `deepnote-runs`). When a later reader needs more, such as a published page or a `get_run` read, create the block with `"metadata": { "deepnote_table_state": { "pageSize": 100 } }`, sized close to the rows it returns: every stored output and snapshot grows with it, and `0` turns the output into an error. A code block without its own `deepnote_table_state` inherits the table state last set during the run, and SQL blocks always set one, so give the block the reader uses its own `pageSize`. `update_block` cannot change metadata, so recreate the block to change the page size. + For input blocks, put block-type configuration in `metadata` and keep `content` for the visible or default textual content. Preserve existing naming and variable conventions when adding inputs near related blocks. ## Block Update diff --git a/plugins/deepnote/skills/deepnote-runs/SKILL.md b/plugins/deepnote/skills/deepnote-runs/SKILL.md index 336042a..b73dfe9 100644 --- a/plugins/deepnote/skills/deepnote-runs/SKILL.md +++ b/plugins/deepnote/skills/deepnote-runs/SKILL.md @@ -44,10 +44,26 @@ Use `list_notebook_runs` for recent runs, failed runs, run history, or anything This is the single definition of how to handle `get_run` snapshots. - Omit `snapshotDelivery` for routine status checks. The response then carries `snapshotContent: null` and, when a snapshot exists, a short-lived `snapshotDownloadUrl` to a `.snapshot.deepnote` file. That URL grants access to the snapshot: never paste it into an answer unless the user asks for a download or file handoff, and never fetch it on your own. -- Request `snapshotDelivery: "blocks"` to inspect outputs, summarize results, or diagnose a failure from execution outputs. It returns `snapshotBlocks` with block IDs, types, outputs, and metadata, but no source. +- Request `snapshotDelivery: "blocks"` to inspect outputs, summarize results, or diagnose a failure from execution outputs. It returns `snapshotBlocks` with block IDs, types, outputs, and metadata, but no source. A block whose outputs are longer than 16,384 characters as JSON gets a notice with `outputHead` and `outputTail` in their place; add `fullOutputs: true` to lift that cap when you need the output, such as a dataframe's `row_count`. A block over 512 KiB, or past 5 MiB of outputs in the notebook, still comes back as a notice. - Request `snapshotDelivery: "inline"` only when the full snapshot or block source is needed, such as mapping references visible in the snapshot. Inline snapshots can be large and sensitive: summarize the relevant blocks, outputs, failures, or data shape instead of dumping raw content. - If the current tool schema does not expose `snapshotDelivery`, use the fields `get_run` returns as they are and do not invent `snapshotContent` or `snapshotDownloadUrl`. +## Dataframe Outputs + +A block that ends in a dataframe, such as a SQL block, has an `application/vnd.deepnote.dataframe.v3+json` output. Its `rows` hold one page of the result, 10 rows unless a page size is set (`deepnote-notebooks`), while `row_count` is the total after the block's filters; for an output `type` of `query_preview`, the query itself ran with a 100-row limit. The `text/plain` and `text/html` outputs are pandas previews: a frame of up to 60 rows appears whole, a longer one only as its first and last 5 rows. + +- Report counts from `row_count` and `column_count`, and describe `rows` as a sample when `row_count` is larger. Never report that a query returned 10 rows because `rows` has 10. +- When the user needs all rows, change the notebook before running it: display the data as a JSON output, as in `deepnote-dynamic-apps`, or create the block with a larger `pageSize`, as in `deepnote-notebooks`. + +An output of `{ "error": "..." }` means the table could not be built. For pandas frames, values in `rows` do not follow `columns[].dtype`: + +- Missing values are the strings `nan`, `None`, `NaT`, or ``, also inside numeric columns. +- Booleans and nullable integers are strings, such as `"True"` and `"1.0"`. Timestamps are strings such as `"2024-01-01 00:00:00"`. +- When a value on the page is past 2^53 or infinite, the whole column is strings on that page. +- Every row has `_deepnote_index_column`, the frame's index. It is the last entry of `columns`, has no `stats`, and is not counted in `column_count`. + +Polars and Spark frames send missing values as `null` and numbers as JSON numbers. + ## Sensitive Outputs When snapshot content, download URLs, or errors include sensitive, proprietary, personal, or production-like data, minimize exposure. Summarize the result, shape, quality issues, aggregates, or failure mode instead of dumping raw records, presigned URLs, or long logs. diff --git a/plugins/deepnote/skills/deepnote-static-sites/SKILL.md b/plugins/deepnote/skills/deepnote-static-sites/SKILL.md index f16cc88..222c96d 100644 --- a/plugins/deepnote/skills/deepnote-static-sites/SKILL.md +++ b/plugins/deepnote/skills/deepnote-static-sites/SKILL.md @@ -5,7 +5,7 @@ description: Use when publishing, updating, sharing, or unpublishing a static HT # Deepnote Static Sites -Tool arguments and the capability boundary are defined in `deepnote-mcp`. +Tool arguments and the capability boundary are defined in `deepnote-mcp`. A page that runs notebooks or reads their results in the browser follows `deepnote-dynamic-apps`. ## Publishing Workflow