Skip to content

feat(functions-compiler): every build records a returned 5xx as one _b44_error line - #678

Merged
egidijusj merged 12 commits into
mainfrom
feat/functions-compiler-error-records
Oct 26, 2026
Merged

egidijusj merged 12 commits into
mainfrom
feat/functions-compiler-error-records

Conversation

@egidijusj

@egidijusj egidijusj commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Description

A backend function that catches its own error and returns a 500 leaves Cloudflare nothing to record — no exception reaches the runtime, so the Tail Worker sees a status and no reason. This PR makes every compiled entry (single-function and per-app; not actors) carry a small recorder that, for a handler-returned 5xx, logs exactly one _b44_error line for the Production Agent to consume. There is no build flag: the switch is the request's x-b44-feed-context header, which the backend sends only for production calls of apps that opted in, so an app is ready the moment it opts in with no rebuild.

Related Issue

None

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Other (please describe):

Changes Made

  • New packages/functions-compiler/src/error-records.ts — ERROR_RECORDS_PATCH, a self-contained (import-free) JS prelude inlined into generated entries after the console patch. It adds ~3 KB per module (~1.2 KB gzipped).
  • worker-entry.ts — both entry builders now route the handler through _b44RecordError(await handler(request, info)), inside telemetry and invocation logs so the recorder sees the handler's own response before it is re-wrapped, and both spread ..._b44ErrorStore(request) into the request store. The single-function entry's three handler-expression branches collapse into one shared HANDLER_CALL.
  • Message selection, in order: an Error logged during the request whose message the JSON body repeats (newest among the last 8, so a later unrelated warning cannot hide it); else the body (JSON ≤ 2,048 bytes read within 50 ms — first non-empty error/message/detail/details/error.message, skipping error when it is exactly "Internal server error" and details is present); else the last logged Error. No message means no record.
  • Stack is taken only from the Error that supplied the message, as its raw (minified) stack — Cloudflare remaps only exceptions it captures, so a consumer remaps it with the same version's source_map.
  • Scrubbing and bounds — every env-binding and runtime-secret string of 8+ chars (including values nested in JSON-valued ones, leading whitespace allowed) is replaced with [redacted], longest first, before any truncation; then name ≤ 64 chars, message ≤ 500 UTF-8 bytes (well-formed, cut on a code point), stack ≤ 4 KB (cut at a line end).
  • Request safety — the handler's response is returned as-is in every path (never rebuilt, never delayed, no header set); the work runs on a clone inside ctx.waitUntil. Any failure writes nothing and never reaches the request. The uncaught-throw path is untouched and stays Cloudflare's to record.
  • No banner change — the recorder is not a wrapper flag, so it moves the emitted bytes only when the package version does; compiler in the banner is what tells a consumer whether a build carries it.
  • README — new "Error records for a handled 5xx" section documenting the behaviour and the exact raw Tail log shape a consumer parses (level: "log", one wrapper argument, JSON.parse(wrapper.message)._b44_error), plus updates to the banner and byte-stability notes.

Testing

  • I have tested these changes locally
  • I have added/updated tests as needed
  • All tests pass (npm test)

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (if applicable)
  • My changes generate no new warnings
  • I have updated docs/ (AGENTS.md) if I made architectural changes

Additional Notes

New test/error-records.e2e.test.ts (~550 lines) runs compiled bundles end to end: it pins the raw Tail entry shape for both entry kinds, body-message selection (including the SSO "Internal server error" placeholder case), the logged-Error precedence and per-request isolation, the no-record cases (body over 2 KB, a stream that misses the 50 ms deadline, anything throwing) while asserting the client still gets the whole body, scrubbing and every bound, that the response is the same response and is not held back, and that an uncaught throw is unchanged. worker-entry.test.ts adds a recorder suite, and the entry snapshots are regenerated — the generated bytes of every non-pull app change, which is why the snapshot diff is large.

lint, typecheck, build and knip pass on CI; the functions-compiler and CLI test jobs were still running when this description was generated, so the two unchecked testing boxes reflect what could not be verified here rather than a known failure.


🤖 Generated by Claude | 2026-10-26 15:00 UTC | f69de2a

egidijusj and others added 11 commits October 20, 2026 09:42
`sourceMap: true` on `bundle`/`bundleApp` returns `map` beside `module`:
an esbuild map of the final compile (not the per-function probes) with no
sourcesContent and no sourceRoot. The module keeps the trailing
`//# sourceMappingURL=_bundled.mjs.map` comment Cloudflare reads to remap
stack traces, and otherwise matches the option-off build byte for byte.

The map's sources name the request's files: per-app `fn_<index>/` is
stripped and checked against that function's files, and a function's
entry becomes its optional `entryPath` (its real stored path). Every
other source stays in place, so the mappings' indices hold, behind a
`base44-internal:` prefix.

Off by default; the public CLI never sets it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`errorRecords: true` on `bundle`/`bundleApp` bakes a recorder into both
generated entries. It is inert unless the request carries
`x-b44-feed-context`. For a handler `Response` with status >= 500 it
returns that same response at once and, on a clone inside
ctx.waitUntil, logs one
`{ _b44_error: { v: 1, kind: "response", status, name?, message, stack? } }`
through the console patch.

The message comes from an Error logged during the request whose message
the JSON body repeats, else the body's first non-empty error / message /
detail / details / error.message (2 KB, 50 ms; the SSO
"Internal server error" placeholder skipped when details is present),
else the last logged Error. The stack comes only from that Error. Env and
runtime secrets of 8+ characters are scrubbed, longest first, before
name (64 chars), message (500 bytes) and stack (4 KB) are cut. Any
failure writes nothing; the crash path is unchanged.

Off by default; the public CLI never sets it, and flag-off entries and
banners keep their bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e_map

Cloudflare remaps only the stacks of uncaught exceptions it captures;
a stack the Worker logs itself (the per-app entry's console.error before
rethrowing, or user logs) stays minified. Run a throwing function in
workerd with a Tail Worker, for both entries, and map the exception's
frames back to the app's files by hand — local workerd does no remapping
— and pin that the logged per-app crash line stays minified. Say in the
README what is remapped automatically and what a consumer must remap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It is the raw Error.stack read inside the Worker, and Cloudflare remaps
only the exceptions it captures, never a logged string. Say so, and that
a consumer remaps it with the same version's source_map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y recent logged Error

Two review findings on the error recorder:

- A JSON-valued binding with leading whitespace (`\n{"db":{...}}`) was
  not parsed, so the secrets nested in it were logged unredacted.
  Detect JSON after trimStart(); the original string stays a candidate.
- A single saved Error was overwritten on every log, so a later
  unrelated warning lost the matching Error's name and stack. Keep the
  last 8 Errors logged in the request, use the newest one whose message
  the body repeats, and the newest one only as the fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The error recorder now replaces a matched secret with `[redacted]`
instead of `[secret]`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the `errorRecords` option. Both generated entries now always append
the recorder, route the handler's response through `_b44RecordError`
and seed `_b44ErrorStore(request)`, including the public CLI's own shard
builds and bundleApp's probe compiles. An app is ready for error
reporting the moment it opts in, with no rebuild.

The off switch is the request: the recorder does nothing unless it
carries `x-b44-feed-context`, which the backend sends only for
production calls of opted-in apps while its tail switch is on. Runtime
behaviour is unchanged, crash path included.

The banner loses its `errorRecords` field; `compiler` says which builds
carry the record. This changes every build's output once: about +3 KB
raw (+1.2 KB gzipped) per module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cord

The record is logged through the console patch, not raw: a Tail Worker
gets one entry with level "log" whose single argument is the patch's
wrapper `{ _b44_env, _b44_function?, level: "error", message }`, and
`message` is the JSON string of `{ "_b44_error": {...} }`. apper's error
tail is the consumer. Pin that entry for both entries, so a change to
the console patch fails here instead of silently in production, and
state the shape in the README and the module doc. No change to what is
emitted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Package Preview Available!


Install this PR's preview build with npm:

npm i @base44-preview/cli@0.2.3-pr.678.97827aa

Prefer not to change any import paths? Install using npm alias so your code still imports base44:

npm i "base44@npm:@base44-preview/cli@0.2.3-pr.678.97827aa"

Or add it to your package.json dependencies:

{
  "dependencies": {
    "base44": "npm:@base44-preview/cli@0.2.3-pr.678.97827aa"
  }
}

Preview published to npm registry — try new features instantly!

@egidijusj

Copy link
Copy Markdown
Collaborator Author

Description

Every build of a backend function now carries a small recorder in its generated Worker entry. When a function catches its own error and returns a 5xx, the recorder logs one _b44_error line saying what went wrong, so the Production Agent's Tail Worker has a reason to file, not just a status.

There is no build option. The off switch is the request: the recorder does nothing unless the request carries x-b44-feed-context. The backend sends that header only for production calls of apps that opted in, while its tail switch is on. Turning that switch off silences the record in every bundle, whatever built it.

Stacked on base44/cli#677 (source maps). This PR's base is #677's branch, so its diff shows only its own changes. When #677 merges, GitHub retargets this PR to main. It replaces base44/cli#669, which was opened from a fork, where CI's secret-dependent jobs can't run; the earlier review discussion is there.

Why it's in every build

  • An app is ready the moment it opts in, with no rebuild. Behind a build flag, only builds that set the flag would carry the record.
  • The public CLI's own builds get it too. The CLI compiles shards locally with its own COMPILE_MODES (packages/cli/src/core/version/functions.ts), which would never have set a flag.
  • It costs little. About +3.0 KB raw / +1.2 KB gzipped per module (single-function: 41,400 → 44,422 bytes; per-app: 43,159 → 46,191). Without the header, it costs one headers.has per request and one context lookup per console.* call. Source maps (feat(functions-compiler): opt-in external source maps with app paths #677) keep their flag, because they cost compile time and memory and are useless unless the caller stores them.

Request and response changes

None. An earlier revision of this PR added an errorRecords option; it's gone. Nothing shipped with it, so no caller sends it and no compatibility shim is needed. The build banner is unchanged from main: the errorRecords field is gone, and the banner's compiler version tells a consumer whether a build carries the record.

What a consumer sees (the contract)

The recorder calls console.error({ _b44_error: record }). That goes through the console patch, so a Tail Worker does not see that object. It gets one log entry:

Field Value
TailItem log level "log". The patch writes every level through the original console.log
message an array with one argument: the patch's wrapper object
wrapper's keys _b44_env, _b44_function (per-app bundles only), level, message
wrapper's level "error"
wrapper's message the JSON string of { "_b44_error": { v: 1, kind: "response", status, name?, message, stack? } } (name/stack omitted when absent)

A consumer keeps entries whose wrapper level is "error" and reads JSON.parse(wrapper.message)._b44_error. apper's error tail (workers/base44-log-tail/src/error-tail.ts) is that consumer, and today it parses the raw object, so it drops every record until it's changed (level === "error" on the wrapper, then JSON-parse message). Merging this before that change is harmless: records are simply dropped until then. The README and the module doc state this shape, and a test pins it.

Invocation logs. The record is logged inside waitUntil, after the response went back, so it never lands in the X-B44-Invocation-Logs header owners get:

  • the single-function entry has no capture store at all;
  • the per-app entry pushes the line into the capture buffer after _b44AttachInvocationLogs has already written the header.

Confirmed in workerd. It is still an ordinary stamped line, so views built on stamped Workers log lines (such as per-function log queries) show it as an error line whose message is the JSON. That applies only to production calls of opted-in apps, with secrets scrubbed.

Behaviour (src/error-records.ts), unchanged by this revision

  • Inert unless the request carries x-b44-feed-context. Presence is enough; the value isn't checked.

  • Handler Response with status ≥ 500: the same object is returned at once, never rebuilt, delayed or given a header. The recorder clones it and does the rest inside ctx.waitUntil. It sees the handler's own response, before telemetry or invocation logs re-wrap it.

  • Message, in order:

    1. an Error logged (any console.*) during this request whose message the JSON body repeats anywhere. The recorder keeps the last 8 Errors logged in the request and uses the newest one that matches, so a later unrelated warning doesn't hide it;
    2. otherwise the body: JSON only, from the clone, at most 2,048 bytes within 50 ms, then the reader is cancelled (over 2,048 bytes yields nothing). The first non-empty string among error, message, detail, details, error.message; error is skipped when details is present and error is exactly "Internal server error";
    3. otherwise the newest Error logged during this request.

    No message means no record.

  • Stack only from the logged Error whose message was chosen. It's minified: it's that Error's raw stack, read inside the Worker, and Cloudflare remaps only the exceptions it captures. The Production Agent remaps it with the same version's source_map from feat(functions-compiler): opt-in external source maps with app paths #677.

  • Scrub before cutting. Every string of 8+ characters from the request's env bindings and runtime secrets, longest first, becomes [redacted] in the name, message and stack. This includes strings nested in JSON-valued bindings, also when they start with whitespace (such as the private-data-sources manifest, which carries DB passwords). Binding objects (KV, services, Hyperdrive) are skipped.

  • Bounds: name ≤ 64 characters; message ≤ 500 UTF-8 bytes after toWellFormed(), cut on a code point; stack ≤ 4 KB, cut at a line end.

  • Never throws into the request. Any failure writes nothing.

  • Crash path untouched. The per-app entry's catch still does console.error(e) then throw e. A test pins that block's exact text.

This changes every build's output once

Snapshot change, in one line: each of the four pinned worker-entry snapshots (single-function and per-app, telemetry on and off) gains the recorder prelude. In each, exactly two existing lines change: the request store gains ..._b44ErrorStore(request), and the return becomes _b44RecordError(await handler(...)).

For the release: every build's bytes change once, by about +3 KB raw and +1.2 KB gzipped per module. So every function gets new bytes on its next deploy after apper's bundler picks up this version. Where a version's identity is the hash of the compiled output, as in the CLI's versions path, that mints one new version per app on its next publish. A package-version bump does the same anyway.

Byte comparison of the single-function lane

The README said bundle() stays bannerless "which keeps that lane byte-comparable with the engine apper still runs". I found nothing that still compares it.

  • apper's bundler service (infra/base44-userapp-bundler) pins and runs this package (0.1.7), so there's no second engine to compare against.
  • apper's own "byte-identical" notes (function_bundle.py) are about the bundle input, the flat main.ts submission, not the compiled output.
  • I found no hashing of the compiled module in apper's deploy path that decides redeploys.
  • bundle-banner.e2e.test.ts only asserts that bundle() has no banner, which still holds.

I kept bundle() bannerless, since it compiles one function and has nothing to name, and reworded the README. The caveat: this is from a local apper checkout I didn't update, so a newer apper commit could differ. No flag was kept for it.

Open question: client-sent x-b44-feed-context

The recorder treats the header's presence as the switch. Now that every app carries it, it matters whether the dispatcher strips a client-supplied x-b44-feed-context before calling a function. I couldn't find the header anywhere in the local apper checkout, so I can't verify this. The impact of a spoofed header is small: it only records a 5xx the function really returned, with secrets scrubbed, into that app's own logs. Please confirm the dispatcher drops it from client requests.

Tests

test/error-records.e2e.test.ts runs every case in workerd (Miniflare) and reads records through a real Tail Worker, the way production reads them.

  • The raw Tail log entry is pinned (added after review), for both entries. The test checks the log's level ("log"), the argument count (1), the wrapper's exact keys and level ("error"), and that JSON.parse(wrapper.message)._b44_error equals the record. Its comment names apper's error tail as the consumer, so a change to the console patch fails here, not silently in production. With no build option, the header is the only switch: every positive case also asserts there's no record without the header, and the cases that compared an option-off build now compare header-absent vs header-present on the same build.

  • 5xx JSON in each field shape: error, message, detail, details, error.message, first non-empty wins. Both SSO shapes: {error:"Internal server error", details} gives details, and {error:"Internal server error"} gives the placeholder. A near-miss isn't skipped. A 503 reports 503. The per-app entry works too and stamps _b44_function;

  • no record for: a 5xx text body, a 4xx, a 2xx, a 5xx with no message field, a non-Response return, a body over 2 KB, and a streamed body that misses the 50 ms deadline. The client still gets the whole body in the last two;

  • a recorder failure (a Response subclass whose clone() throws): the response stands and no record is written;

  • logged Errors: a matching one gives name, message and stack; it still wins when an unrelated warning is logged after it; the 8-Error bound is pinned (7 later Errors keep the match, 8 push it out); an unrelated one is ignored; the newest is the fallback for a text body; an Error from an earlier request doesn't leak;

  • non-ASCII cut to ≤ 500 bytes on a code point; a lone surrogate becomes U+FFFD; env secrets scrubbed in message and stack, longest first, with short values kept; JSON bindings scrubbed, including whitespace-led ones; stack ≤ 4 KB on a line end; name ≤ 64;

  • the response is identical (status, headers, body) with and without the header, and its headers arrive while the handler's stream is still open;

  • an uncaught throw, in both entries: the same status, body and Tail exceptions with and without the header, and no record.

test/worker-entry.test.ts:

  • the recorder is in both entries under every telemetry × runtimeSecrets combination;
  • it sits inside telemetry and invocation logs;
  • the crash block is unchanged;
  • the banner keys are exactly functions, telemetry, runtimeSecrets, compiler;
  • the four pinned snapshots are regenerated.

Results:

  • full compiler suite: 490/490;
  • typecheck across all packages: clean;
  • lint and knip: clean;
  • the CLI's tests/core/version-functions.spec.ts, its function-compiling publish path: 4/4.

Notes for review

  • Version and changelog. No version bump or changelog entry in this PR, per this package's convention: the Manual Functions Compiler Publish workflow bumps the version in its own release commit, and the root CHANGELOG.md is the CLI's. The release notes should say: every build now carries the _b44_error record, inert until the backend sends x-b44-feed-context. The README says the same.
  • What gets no record:
    • responses the wrapper builds rather than the handler: the per-app "failed to initialize" 500 and the 503/404 routing answers;
    • the per-app crash response that X-B44-Capture-Logs turns an uncaught throw into, which is the crash path;
    • actor entries.
  • Overlap with base44/cli#662 (edge errors on outbound fetches): it also edits worker-entry.ts and its snapshots, so whichever lands second needs a rebase and a snapshot regeneration.
  • "Not delayed" in Miniflare. Miniflare's dispatchFetch hands a streamed body to the caller only once it closes, so the test times the headers.

🤖 Generated with Claude Code

netanelgilad
netanelgilad previously approved these changes Oct 26, 2026
Base automatically changed from feat/functions-compiler-source-maps to main October 26, 2026 13:56
@egidijusj
egidijusj dismissed netanelgilad’s stale review October 26, 2026 13:56

The base branch was changed.

…er-error-records

# Conflicts:
#	packages/functions-compiler/README.md
@egidijusj
egidijusj enabled auto-merge (squash) October 26, 2026 15:05
@egidijusj
egidijusj merged commit 67676dd into main Oct 26, 2026
14 checks passed
@egidijusj
egidijusj deleted the feat/functions-compiler-error-records branch October 26, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants