MORPH

Website · White paper · Contributing

MORPH, Model Optimized Representation for Prompt Handoffs, is a local compiler for structured model context. It accepts JSON-compatible data, an optional schema, a task, a tokenizer target, and constraints. It compares reversible physical layouts, measures the complete rendered text with the selected tokenizer, applies policy and budget gates, and returns a self-contained artifact with an explain report.

MORPH does not project fields, filter rows, calculate answers, summarize data, or claim that a smaller representation is easier for a model to understand. Preservation, token measurement, and model-task quality are separate outcomes.

Status

This public source repository contains the verified implementation for specification version 0.1.0. It includes the SDK, five native codecs, the optional official TOON adapter, an offline tokenizer profile, CLI, evaluation harness, browser workbench, project website, and white paper. Executed release evidence is recorded in RELEASE_REPORT.md. Model-task quality remains not-run, and no npm package or GitHub release has been published.

Area	Status	Meaning
Strict JSON and JavaScript-value IR	Implemented	Duplicate JSON keys and unsupported JavaScript values are rejected. JSON number lexemes are retained.
Five native codecs	Implemented	Compact JSON, MORPH-framed JSON Lines, typed delimited rows, columns JSON, and typed path/value are registered.
Official TOON adapter	Implemented, optional	Uses @toon-format/toon 4.1.1 and rejects inputs whose numeric spelling or value cannot meet MORPH equality.
local-o200k-base tokenizer profile	Implemented	Counts complete visible text with bundled js-tiktoken ranks. It has no provider or model binding.
Compatibility planner	Implemented	Compact JSON remains the default without applicable quality evidence.
Economy planner	Experimental	May select a smaller verified representation. This does not establish model comprehension.
Validated nonbaseline selection	Gate implemented, no bundled evidence	Declared identity, confidence-bound, case-count, dataset-count, task-family, and expiry checks exist. No quality profile or live model result ships with the repository.
Offline codec and token suites	Verified	The recorded release run has 70 passing conformance cases, 3 correctly inapplicable cases, 61 complete prompt measurements, and no conformance failure.
Composed SDK package	Implemented	@morph/sdk re-exports the local modules and provides createDefaultMorph().
Compact JSON source map	Implemented	Maps JSON Pointers to UTF-16 key and value spans in compact JSON output.
Project website and browser workbench	Implemented	A technical project dossier surrounds the real local compiler. It runs native codecs, gated TOON, and the tokenizer in a worker, and can strictly import, verify, render, and decode a saved artifact without uploading it.
Model evaluation	Bounded provider-neutral runner implemented	Builds matched compact-JSON and candidate trials, randomizes them by seed, and enforces request, concurrency, retry, timeout, output, and call-or-priced-cost limits. No provider adapter, quality profile, or Layer C run is bundled.
Schema registry	Implemented optional extension	Includes bounded in-memory and local-filesystem stores, content-addressed bundles, reference creation, and hydration back to self-contained artifacts.
Jev planner	Implemented optional adapter, live test not run	Provides bounded access-pattern classification and deterministic fallback. It is not wired into core selection and is not evidence of model quality.
Vercel project website	Deployed	The public project guide, real browser compiler, light and dark themes, approachable online white paper, and clearly labeled technical PDF are served by the existing Vercel project.
Source publication	Public	The GitHub repository is public for review and collaboration. Workspace packages remain private and unpublished.

Requirements

• Node.js >=22.12.0 <25
• pnpm 10.15.0, locked by the root packageManager field
• No account, API key, hosted database, or network connection at runtime after the dependencies are installed

The root and every workspace package manifest declare the same Node.js range.

Install exactly from the lockfile:

pnpm install --frozen-lockfile

Contributing

Start with CONTRIBUTING.md and the docs/README.md reading paths. The repository provides structured issue forms, a pull request verification checklist, CODEOWNERS routing, governance, security reporting, support boundaries, a code of conduct, and a decision-record template.

The normal contributor check is:

pnpm verify:pr

Changes to codecs, artifacts, security boundaries, package exports, benchmarks, or the workbench require the additional checks listed in CONTRIBUTING.md. Do not add a license, publish packages, change repository visibility, run paid evaluations, or create deployment resources without separate owner authorization.

First local run

Inspect the bundled synthetic customer fixture:

pnpm morph inspect --input fixtures/examples/customers.json

Compare complete candidate prompts with the experimental token objective:

pnpm morph compare \
  --input fixtures/examples/customers.json \
  --task-file fixtures/examples/lookup.task.txt \
  --schema fixtures/examples/customers.schema.json \
  --target local-o200k-base \
  --policy economy-experimental \
  --report reports/customers-comparison.json

Compile with the compatibility policy, then render, verify, and restore:

pnpm morph compile \
  --input fixtures/examples/customers.json \
  --task-file fixtures/examples/lookup.task.txt \
  --schema fixtures/examples/customers.schema.json \
  --target local-o200k-base \
  --policy compatibility \
  --output artifacts/customers.morph.json \
  --report reports/customers-explain.json

pnpm morph render \
  --input artifacts/customers.morph.json \
  --output artifacts/customers.context.txt

pnpm morph verify --input artifacts/customers.morph.json

pnpm morph decode \
  --input artifacts/customers.morph.json \
  --output artifacts/customers.restored.json

Output files are not overwritten unless --force is supplied. Output paths are restricted to the current workspace by lexical and resolved-parent checks. Use --input - to read JSON or an artifact from standard input where the command documents that option.

Project website and browser workbench

Vercel site: https://morph-one-jade.vercel.app

White paper: https://morph-one-jade.vercel.app/white-paper

The hosted site uses the same local-first compiler as this repository. It performs no provider call, automatic data upload, analytics, or input persistence. The appearance toggle follows the system theme on load and changes only the current page session.

Start the local Vite server:

pnpm workbench

Open the printed loopback URL. The site documents the use case, preservation contract, compiler pipeline, representation families, policy model, release evidence, trust boundaries, and canonical repository references. The embedded workbench includes synthetic uniform, nested, sparse, repeated-string, multilingual, and tiny examples. It compiles in a worker, rejects stale worker results, supports cancellation, and can download the context, artifact, explain report, and restored JSON. A user can import a local artifact of up to 20 MiB; the worker applies the strict artifact parser, verification, rendering, and decoding paths before displaying it. The file remains local. The workbench does not use analytics, browser storage, remote fonts, or model calls.

The dedicated white-paper route presents the implemented system in long form and links to the reviewed 23-page PDF at output/pdf/MORPH_White_Paper_v0.1.pdf. The visual layer uses project artwork and an architecture-editorial palette, gradients, soft elevation, pill controls, and theme behavior. The existing MORPH charts, compiler pipeline, and workbench layout remain intact.

See the website implementation notes for its information architecture, claim sources, asset provenance, and deployment boundary.

Offline verification commands

pnpm lint
pnpm typecheck
pnpm test
pnpm test:extended
pnpm build
pnpm bench:conformance
pnpm bench:tokens
pnpm bench:performance
pnpm smoke:package
pnpm morph doctor

The token suite reports tokenizer-specific counts for the complete MORPH-PROMPT/1 text. It does not report provider billing tokens or model accuracy. The conformance suite reports codec outcomes. The extended property command supplies a 5,000-round-trip test budget. The current test allocation derives 2,500 general JSON inputs and 625 uniform-table inputs from that budget, then exercises every applicable codec plan for each generated input. The installed CLI package carries the versioned synthetic benchmark manifest and its nine fixture files, so its offline bench commands do not depend on a source-workspace fixture path. The package smoke command is configured to pack all nine Node packages, including the optional schema-registry and Jev packages, before exercising a clean local consumer. Its final result belongs in the release report. See the final release report for commands actually run and their real outcomes.

Supported representations

Encoding	Applies to	Preservation notes
json-compact@1	Every accepted IR	Strict compact JSON and the primary baseline.
json-lines@1	Root arrays	One complete JSON value per physical LF-delimited line, with an explicit element count.
rows-delimited@1	Nonempty uniform arrays of primitive-only objects	Field names are declared once. Every cell is one JSON scalar token. Tab, comma, pipe, and semicolon plans are bounded alternatives.
columns-json@1	Nonempty uniform arrays of primitive-only objects	One JSON array per field. Row index alignment and source row order are retained.
path-value@1	Every accepted IR	Typed JSON Lines records use RFC 6901 pointers and emit empty containers explicitly.
toon@4.1.1	Inputs proven safe for the official adapter	Three official delimiter plans with indent size 2. Noncanonical, negative-zero, and precision-unsafe number lexemes are inapplicable.

All selected candidates are decoded twice: directly from their encoded section and again after parsing the self-contained model bundle. Both results must match the input semantic digest.

When a Draft 2020-12 schema is supplied, MORPH validates the accepted IR without coercion, defaults, property removal, format fetching, or data mutation. Remote schema references and other dialects are rejected. If exact numeric schema keywords meet a precision-sensitive raw number lexeme, compilation fails with SCHEMA_NUMERIC_VALIDATION_UNSUPPORTED rather than rounding.

What a token count means

The bundled profile is:

profile: local-o200k-base
tokenizer: o200k_base
revision: js-tiktoken@1.0.21:o200k_base:sha256:446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d
scope: complete visible MORPH-PROMPT/1 rendered text
certainty: exact-for-tokenizer
model binding: none

The measurement includes caller prefix and suffix, the compiler guide, layout metadata, included schema, payload, task text, and prompt framing. It excludes provider message framing, hidden instructions, tool schemas, images, outputs, and billing rules.

The seven committed tokenizer fixture cases were verified against official Python tiktoken 0.14.0 encode_ordinary with zero mismatches. Python was used only as a development oracle and is not a runtime dependency.

Design boundaries

• Re-encoding is not projection. Every accepted row and field remains present.
• Task and access-pattern fields are hints. MORPH does not execute the requested task.
• A checksum detects mismatches. It is not authentication or proof that content is safe.
• Encoding does not neutralize prompt injection in data or schema descriptions.
• A tokenizer profile is not a model profile unless an explicit verified binding says so.
• No quality percentage is produced without recorded evaluation evidence.
• Provider caching, local plan caching, schema lookup, and prompt size are separate concerns.
• estimated-request-cost is not available without a versioned pricing and request-accounting profile.

Documentation

• Documentation map
• Contributing
• Governance
• Code of conduct
• Security reporting
• Support
• Changelog
• Architecture
• Artifact, framing, and codec formats
• Semantic contract
• SDK API
• CLI
• Benchmarking and evidence
• Security and privacy
• Dependencies
• Release process
• Project website
• Technical white paper
• Current release report
• Product specification
• Primary references
• Decision records

Publication state

Every workspace package is marked private, so public source visibility does not publish an npm package. The repository and existing Vercel website are public. No npm publication or GitHub release has been authorized. No project license has been selected, so public visibility alone does not grant reuse rights. Verified remote state belongs in RELEASE_REPORT.md.