Skip to content

Harden dependencies and remote fetches - #109

Merged
scalvert merged 4 commits into
mainfrom
chore/deps-and-remote
Oct 10, 2026
Merged

scalvert merged 4 commits into
mainfrom
chore/deps-and-remote

Conversation

@scalvert

@scalvert scalvert commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

Remote-fetch and package hardening. A redirect can no longer downgrade a remote snippet to plain HTTP, remote bodies are size-limited, a deprecated package is gone, and the published tarball shrinks from 1.3 MB to 31 KB.

Remote snippets

Redirects bypassed allowInsecureHttp: false. Node's fetch follows HTTPS → HTTP redirects silently. Checked against a live endpoint:

fetch('https://httpbingo.org/redirect-to?url=http://example.com/')
→ status 200, redirected true, final http://example.com/

fetchRemoteContent now follows redirects itself (redirect: 'manual', at most 10). It resolves relative Locations and validates every hop before requesting it:

  • http: is rejected unless allowInsecureHttp is on. That includes an http: hop in the middle of a chain (https → http → https), where a network attacker could otherwise point the last hop anywhere:
    Remote file redirected to an insecure HTTP URL: https://… -> http://example.com/. Use HTTPS or set allowInsecureHttp: true in config.
    
  • Non-HTTP schemes (file:, data:) are rejected, as are URLs with credentials (without echoing them) and unparseable Locations.
  • More than 10 redirects fails with Too many redirects.

No size limit. The response used to be read with response.text(), however large. Bodies are now streamed and rejected past 5 MB: up front from Content-Length, or as soon as the streamed body passes the limit. The limit counts decompressed bytes, so a gzip bomb is caught. Decoding matches response.text() exactly: BOM stripped, invalid UTF-8 becomes U+FFFD, multi-byte characters split across chunks decode correctly. The timeout still covers the whole redirect chain and the body. Error responses have their body cancelled, so a stalled 500 can't keep the process alive. The limit is a FetchOptions.maxBytes parameter, not a config option. The README documents both rules.

Dependencies and package

  • Removed mdast@3.0.0, a deprecated stub (npm: "mdast was renamed to remark"); the code only uses its types. @types/mdast moved to devDependencies, because dist/index.d.ts has no imports. A consumer project compiled with skipLibCheck: false and no @types/mdast.
  • markdown-code.png is no longer published. The README uses the GitHub raw URL (checked: 200 image/png), and the smoke test's allowed-files list is updated.
  • Markdown files are processed in sorted order, so output is the same on every platform. Snapshots didn't change; the only visible difference is which path of a symlinked file is reported as updated.
  • fast-glob stays. I tried replacing it with tinyglobby to clear the remaining micromatch/braces advisories (stack exhaustion on deeply nested brace patterns, no fixed release). The review found tinyglobby 0.2.17 drops the working directory's own files from a ../ pattern: md-code check --markdown-glob '../**/*.md' run from docs/ found nothing. Glob patterns come only from the user's own config or CLI, so the advisory isn't reachable by an outside attacker. glob.test.ts now covers the ../ case so a future swap gets caught. The lockfile diff only removes mdast.

Tests

  • remote.test.ts: stubs now return real Responses, since the body is streamed.

    • Redirects: HTTP hop rejected without being requested; HTTP in the middle of a chain; HTTP allowed with the flag; relative Location; non-HTTP scheme; credentials; invalid Location; 10-redirect limit; 3xx without Location.
    • Error responses have their body cancelled.
    • Size: Content-Length over the limit, streamed body over the limit, exactly at the limit, and UTF-8 split across chunks.
  • glob.test.ts: ../ includes the working directory's own files; symlinked directories under a symlinked absolute root are followed.

  • End to end: a local redirect chain (/start → /middle → final?x=1) works, and the live HTTPS→HTTP redirect is rejected.

  • I have run npm run lint and npm run format:check.

  • I have run npm run build to verify the project builds.

  • I have run npm test to ensure all tests pass. (340 passed, plus npm run test:package)

Reject HTTPS snippets that redirect to HTTP unless allowInsecureHttp is set, and stream remote bodies with a 5 MB limit. Replace fast-glob with tinyglobby, removing the only production advisories (micromatch/braces); sort results for a stable order. Drop the deprecated mdast package, move @types/mdast to devDependencies, and stop publishing the 1.3 MB logo.
@scalvert scalvert added the bug Something isn't working label Oct 10, 2026
tinyglobby 0.2.17 drops the working directory's own subtree from a ../ pattern and, in some layouts, symlinked directories under a symlinked absolute root, so md-code check --markdown-glob '../**/*.md' found nothing. Go back to fast-glob, keep sorted results, drop the empty-pattern guard that was only needed for tinyglobby, and add glob tests for both cases. Regenerate the lockfile from main so it only drops mdast.
Checking only the final URL let an HTTP hop in the middle of a redirect chain through (https -> http -> https), and an insecure hop was requested before being rejected. Follow up to 10 redirects with redirect: 'manual', resolving relative Locations and rejecting http: (unless allowInsecureHttp) and non-HTTP schemes before requesting them.
@scalvert

Copy link
Copy Markdown
Owner Author

Second-pass review

Four passes: dependencies, then remote fetch three times.

Dependencies. All 9 fixtures run through check/sync/extract/eject gave identical output to main. The pass also found that the tinyglobby swap broke file discovery:

  • ../ patterns drop the working directory's own files. md-code check --markdown-glob '../**/*.md' run from docs/ found 5 files on main and none on the PR. This was tinyglobby 0.2.17, the latest release.
  • Symlinked directories under a symlinked absolute root are skipped (e.g. /tmp → /private/tmp on macOS). I couldn't reproduce this one in three layouts.
  • Smaller changes: !docs/** and docs/ excludes behaved differently, and the default excludes now applied to absolute patterns.

Fixed: I kept fast-glob and added a ../ test that fails with tinyglobby. The remaining braces advisory needs control of your own glob patterns, which come only from your config or CLI. Removing mdast, moving @types/mdast to dev (a consumer compiled with skipLibCheck: false and no @types/mdast), and the README image URL (200) all check out. The lockfile diff only removes mdast.

Remote fetch, first pass: real node:http/https servers. It found that checking only the final URL let https → http → https through, and that an insecure hop was requested before being rejected.

Fixed: redirects are followed manually (redirect: 'manual', at most 10). Every hop is validated before it's requested.

Remote fetch, recheck: the http hop is never requested in either https → http or https → http → https. Also confirmed:

  • relative, protocol-relative and cross-host Locations resolve;
  • an A↔B loop stops at 11 requests in 2 ms;
  • a 3xx without Location, a 304 and a 300 aren't followed;
  • every hop sends the same method and headers;
  • the timeout covers the whole chain and the body;
  • check through 5 redirects takes 0.15 s on 2 connections.

Other checks:

  • Size limit: 6 MB is caught with and without Content-Length. A gzip body that expands to 20 MB is caught, because the limit counts decompressed bytes. Exactly 5 MB passes.
  • Decoding: BOM handling and invalid UTF-8 match response.text().

Fixed in 328c09f: redirects to URLs with credentials are rejected without echoing them, an invalid Location names the original URL, and error responses have their body cancelled. Before, a stalled 500 body kept check alive for 8 s; that was also true on main.

Not changed: if a body is shorter than its Content-Length, undici's error is just terminated, with no URL. This was the same with response.text().

CI green on Ubuntu, macOS and Windows × Node 22 and 24, including test:package. 340 tests pass locally.

@scalvert
scalvert merged commit d5ce418 into main Oct 10, 2026
7 checks passed
@scalvert
scalvert deleted the chore/deps-and-remote branch October 10, 2026 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant