Skip to content

Diff GitHub pull requests in the browser with diffr compiled to wasm - #74

Draft
milanb17 wants to merge 17 commits into
mainfrom
mb/diffr-web
Draft

Diff GitHub pull requests in the browser with diffr compiled to wasm#74
milanb17 wants to merge 17 commits into
mainfrom
mb/diffr-web

Conversation

@milanb17

@milanb17 milanb17 commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Adds web/, a static page that diffs a GitHub pull request or comparison in the browser with diffr itself, compiled to WebAssembly. Replace github.com with the page's host in any pull request URL.

Engine

  • crates/diffr-core: the diff engine and the session around it (configuration, the plugin pipeline with the bundled plugins, file tags, and one file's record), extracted so it builds for wasm32-unknown-unknown. What a target may lack comes from the caller through plugin::Environment: plugins' git, a loader for WASM component plugins, and a clock for plugin timings. CI fails if a target_family gate appears outside the grammar loader.
  • diffr-cli keeps what needs the native platform: git, the threaded stream, Wasmtime, the config file on disk and the terminal. Its library no longer builds for wasm.
  • crates/diffr-web: a wasm-bindgen Differ, depending on diffr-core alone, that turns two sides of a file into its file record of the wire protocol, shaped by the bundled plugins as the CLI shapes it.
  • Rust 1.99 and the latest major version of every dependency, which the wasm build needs (see a4b7d9b).
  • A file that falls back to a line diff because of the graph limit or parse errors keeps its language, so its line diff is still highlighted. This changes --syntax output for the CLI and TUI too.

Page

  • File tree from @pierre/trees, and each file drawn in the DOM @pierre/diffs styles, filled from diffr's record: structural pairing, changed spans, tree-sitter colour and folds. Split and unified layouts, light and dark themes. Credited in NOTICE (Apache-2.0).
  • Files are diffed lazily from the viewport in a pool of up to eight workers sharing one compiled module. Each file holds room for its diff, sized from GitHub's patch, and rows are windowed, so a 3,000-file change scrolls at 60 fps.
  • Files that fell back to a line diff carry a warning label naming the reason.
  • File contents come from raw.githubusercontent.com without a token, so only the PR, its file list and the merge base count against GitHub's anonymous limit. An optional token is kept in localStorage and sent only to GitHub.
  • The engine ships gzipped (7 MB instead of 64) and is unpacked as it compiles, since static hosts often cap a file at 25 MiB.

Testing

Known limits

  • diffr-core's build reads plugins/shared/queries from the workspace, so the crate is not yet packageable on its own.
  • GitHub's file listing stops at 3,000 files.
  • Without a token, visitors share GitHub's 60 requests an hour per IP.

milanb17 added 17 commits October 1, 2026 17:17
@milanb17
Upgrade to Rust 1.99 and the latest major version of every dependency
a4b7d9b 
Wasmtime 34 -> 49, tree-sitter 0.27, git2 0.21, toml 1.1, dirs 7,
jsonschema 0.58, wit-bindgen 0.62, wasi 0.14.7 and the newest grammar
crates. Rust 1.89+ is also what a wasm32-unknown-unknown build needs: earlier
releases pass structs to tree-sitter's C with a non-standard ABI.

The WASM runner moves to Wasmtime 49's WasiView/WasiHttpView, stream and
FsPerms APIs, with the p2 and default-send-request features that Wasmtime 34
always included. lazy_static gives way to std::sync::LazyLock. Solidity keeps
structural diffing but loses highlighting, since its bundled highlights query
no longer compiles with tree-sitter 0.26.11+. The cc build dependency stays
at 1.2, which tree-sitter-sequel pins.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Separate the engine's inputs from config loading and the CLI
ade817d 
A new params module holds what the engine needs to diff one file: Params,
LanguageParams, DiffOptions with its default limits, and the compiled fold
query (moved from config/query.rs, now with its own QueryError). Config
compiles plugin queries into Params::new instead of building Params itself,
and Params no longer carries the [diff] limits, which the CLI reads from the
config before compiling. Params::default, which built a plugin pipeline,
becomes config::default_params. diff_file_content takes whether the after
side is absent instead of the CLI's FileArgument.

Outside tests, the engine, parser, diff and projection modules no longer
reach config, options, plugin or git: the first step towards a diffr-core
crate that builds for wasm32-unknown-unknown.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Extract the diff engine into a diffr-core crate that builds for wasm
d8c0acf 
crates/diffr-core holds what diffs one file and nothing that reads files,
runs git or spawns threads: tree-sitter parsing and the grammars (with
vendored_parsers/ and their C build), the AST diff, folds, params, and the
projection to the wire records. diffr-cli re-exports its modules under
their old paths, keeps the stream in protocol/stream.rs, and forwards the
lang-* features. Engine tests that need the bundled plugins' fold queries
(folds, projection) move to src/core_tests; the rest run in diffr-core with
Params::new([]). A test-support feature exposes the from_sources helpers.

`cargo xtask build-core-wasm` builds it for wasm32-unknown-unknown with
every language, and a CI job keeps it building. tree-sitter's grammar cache
is thread-local there, since its Language is not Sync on wasm. A Node smoke
test of the result diffs Rust structurally with no host imports; with every
grammar it is 119 MB raw, 5.6 MB with brotli.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Diff GitHub pull requests in the browser with diffr compiled to wasm
852ea5d 
diffr-cli becomes a library plus the diffr binary, and the library builds
for wasm32-unknown-unknown: git, threads, Wasmtime, the clock and the home
directory are native-only, and protocol::stream gains file_from_sources,
which tags, classifies, diffs and shapes one file from text the caller
already has. WASM component plugins report that they need the native diffr;
the bundled native plugins run as in the CLI. The plugin SDK gates its guest
code on wasm32-wasip2 so a wasm32-unknown-unknown host can link it.

crates/diffr-web wraps that in a wasm-bindgen Differ: built once from a
config (the default when none is given), it takes a file's two sides as JSON
and returns the file's wire event. `cargo xtask build-web` builds it and
writes the bindings to web/src/wasm, and `cargo xtask wasm <args>` runs any
cargo command with the C toolchain setup the grammars need.

web/ is a Vite app with no server. It reads a PR or comparison from GitHub
(public repositories need no token; one pasted into the page stays in
localStorage and goes only to api.github.com), diffs each file against the
merge base in a worker, and renders the rows with the TUI's own region, row
and theme model: split or unified, folds that toggle, files diffr hides
behind "Load diff". devdotfast/whiteboard#837 renders all eight files; the
engine is 64 MB raw, 7.2 MB gzipped, and diffs each of them in 15-105 ms.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Rebuild the browser frontend in the DiffsHub layout
ecbb4a6 
The page now follows DiffsHub's layout: a top bar with the PR URL, a file tree from
@pierre/trees with search and a Git status filter, Diff Stats (F2) and Engine (F3)
panels, and each file drawn in the DOM that @pierre/diffs' stylesheet styles. The
content is diffr's own: its changed spans, pairing, tree-sitter colour, and labelled
folds that open from the separator or the gutter.

Files are windowed: each keeps its exact height so the scrollbar is true, and only
files near the viewport hold rows, built in 64-row chunks. The landing page, token
dialog, light and dark themes, and a unified layout for narrow screens are new.

Loading a 2,188-file PR went from 75 s to about 7 s and from 6 GB to 2.75 GB of browser
memory. File-list pages are fetched in parallel. One compiled module feeds a pool of
workers that grows while files queue and shrinks when done. Workers parse diffr's
output and pass text as offsets and highlights as packed typed arrays. Off-screen
files drop their rows.

Pierre's Apache-2.0 components and the Geist fonts are credited in NOTICE.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Show GitHub's patch first and diff with diffr lazily from the viewport
fa5c73c 
Each file now shows GitHub's patch, taken from the file listing the page already loads,
as soon as the PR and the first page of files arrive. diffr then fetches and diffs only
the files on or near the screen, nearest first, and each one replaces its patch in
place. The viewer anchors the reading position while files around it change height.
Files GitHub lists without a patch reserve room for their changed lines. Diff Stats
use GitHub's totals for the change until every file is diffed.

Diffing starts once the merge base is known, without waiting for the rest of the
listing. On the 2,188-file bun PR, the first line now paints at 0.84 s (was 1.6 s)
against the live API, and browser memory went from 2.75 GB to 1.25 GB.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Colour GitHub's patch with diffr's highlighter until the file is diffed
2077d55 
The wasm engine gains a highlight call that guesses the language from the
path and returns tree-sitter spans for a fragment of text. The page joins
each side's patch lines into one fragment and highlights it on a worker of
its own, so colour for the screen never waits behind the diffs in the pool.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Show diffr's diff only, without GitHub's patch first
93c4e15 
A file waited as GitHub's line diff and then jumped to diffr's, which read
as two different diffs in a row. It now holds room for its diff, sized from
GitHub's patch, and shows diffr's diff once it is ready; diffing stays lazy,
from the viewport. Reverts the patch highlighter, which only served the
patch view.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Measure a file's placeholder from its patch once per layout
cd4141d 
Every file's height is read each time any diff lands, and the placeholder
was re-reading GitHub's patch each time. On a 3,000-file pull request that
doubled the slowest scrolling frames.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Diff heavy rewrites structurally in the browser and colour line diffs
f534295 
The web build raises diffr's graph limit from 3 to 10 million, so a heavy
rewrite of an ordinary source file gets a structural diff instead of a
line diff. Matching that large can take a gigabyte or more, so a file whose
removed times added lines pass 5,000, or which GitHub would not patch, is
diffed one at a time.

A file that still falls back because of the graph limit or parse errors
keeps its language, and its line diff is highlighted. Files over the byte
limit or generated stay plain.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Keep the 3M graph limit and mark line-diff fallbacks with a warning
cbf0075 
Back out the raised limit and the one-at-a-time queue for large rewrites.
Files that fall back to a line diff keep their syntax colour from the engine
change, and the header now says so with a warning label naming the reason;
its tooltip has diffr's full message.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Remove the Diffs and Trees credit from the sidebar
93b9097 
Pierre's components stay credited in NOTICE, as their Apache-2.0 licence
asks.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Remove the credit line from the landing page
fbd9a3a 
Pierre's components stay credited in NOTICE.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Keep the token dialog's pill clear of its close button
2d02f6f 
The title row leaves room for the close button, which now sits centred on
it, and the dialog opens with the token field focused rather than the
first link.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Ship the engine gzipped
7787a8e 
Static hosts often cap a file at 25 MiB and the engine is 64, so the build
writes a gzipped copy (7 MB) and the page unpacks it while it compiles. The
raw file wasm-bindgen's loader names is dropped from the build, since
workers always receive the compiled module.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
@milanb17
Pass the session's native-only parts in instead of gating them
26101df 
Each place the shared library reached for git, Wasmtime or a clock now
takes it from its caller, so the code that will move to diffr-core no
longer needs a wasm32 cfg:

- Plugins' git runs through a Git the pipeline is given, and component
  plugins load through its Components; both, and a stopwatch for the
  mutate timings, come in an Environment. The CLI's spawns git, runs
  Wasmtime and times calls; the default has none of them.
- Tags take attribute values as diffr's own AttrValue; looking them up
  with libgit2 moves to git.rs.
- One file's record (shaping, enrichment, wire errors, FileError and
  file_from_sources) moves out of the threaded stream into record.rs.

Tests run a bundled component plugin's own Rust code natively, from a
list build.rs now generates, unless they exercise Wasmtime itself.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
Agent-Session: 1b218ad3-5e21-4c8f-a23c-f0fa33bc6f0a
@milanb17
Move the diff session into diffr-core
6257576 
Configuration, the plugin pipeline and its bundled plugins, file tags and
one file's record now live in diffr-core beside the engine, and diffr-web
depends on diffr-core alone. diffr-cli keeps what needs the native
platform: git, the threaded stream, Wasmtime, finding and writing the
config file, and the terminal. It no longer builds for wasm32, so its
target-specific dependency block goes.

The bundled-plugin registry and query embedding move to diffr-core's
build script. Plugin and setting test helpers are shared through
diffr-core's test-support feature, so tests that need Wasmtime or the
stream (the summarizer's HTTP calls, external components, deferred
enrichment, config set) stay in diffr-cli and use them.

CI fails if a target_family gate appears in diffr-core outside the
grammar loader in parse/tree_sitter_parser.rs.

diffr-core's build now reads plugins/shared/queries from the workspace,
so the crate is not yet packageable on its own.

Written with AI assistance (Claude Code).

Agent-Session: 59d764a3-3e97-4c6d-b035-92634de2b6e4
Agent-Session: 1b218ad3-5e21-4c8f-a23c-f0fa33bc6f0a

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@milanb17