esm: resolving each .js module re-parses the whole package.json "exports" map (memory grows with edges x exports size)

[kryptobaseddev]

Version

v24.21.0 (also seen on 24.16.0)

Platform

darwin arm64 (the logic is platform-independent)

Subsystem

esm, modules

What steps will reproduce the bug?

No dependencies are needed. Create one package with 200 chained modules and a large exports map:

// make.mjs: one package, 200 chained modules, an exports map of about 130 KB
import { mkdirSync, writeFileSync } from 'node:fs';
const N = 200, EXPORTS = +process.argv[2];
mkdirSync('node_modules/big/lib', { recursive: true });
const exp = { '.': './lib/m0.js' };
for (let i = 0; i < EXPORTS; i++) exp[`./sub/feature-${i}/deep/path`] = { types: `./lib/s${i}.d.ts`, import: `./lib/s${i}.js`, require: `./lib/s${i}.cjs`, default: `./lib/s${i}.js` };
writeFileSync('node_modules/big/package.json', JSON.stringify({ name: 'big', type: 'module', exports: exp }, null, 2));
for (let i = 0; i < N; i++) writeFileSync(`node_modules/big/lib/m${i}.js`, (i + 1 < N ? `import './m${i + 1}.js';\n` : '') + `export const v${i} = ${i};\n`);

$ node make.mjs 700 && node --input-type=module -e "const t=performance.now(); await import('big'); console.log(Math.round(performance.now()-t)+'ms', Math.round(process.resourceUsage().maxRSS/1024)+'MB')"
90ms 140MB
$ node make.mjs 0   && node --input-type=module -e "...same..."
18ms 55MB

How often does it reproduce? Is there a required condition?

Every time. The cost grows with (import edges inside a package) × (size of that package's exports map).

What is the expected behavior? Why is that the expected behavior?

Resolving a module's format should not depend on the size of its package's exports map. getFileProtocolModuleFormat only reads type, pjsonPath and exists.

What do you see instead?

For every resolved .js URL, getFileProtocolModuleFormat (lib/internal/modules/esm/get_format.js) calls getPackageScopeConfig(url) to read the package type. getPackageScopeConfig returns { ...data, exists, pjsonPath }. The ...data spread fires the lazy exports (and imports) getters from deserializePackageJSON, and those getters JSON-parse the raw string on every access. So the whole exports map is parsed again for every import edge inside the package, although format detection needs only type.

Real-world case: drizzle-orm 1.0.0-rc.4 ships a 290 KB package.json with 718 exports. await import('drizzle-orm/sqlite-core') peaks at 301 MB RSS and takes 181 ms. The same code loaded through drizzle's CJS build, which never takes this path, peaks at 60 MB. A heap-sampling profile that includes collected objects attributes about 137 MB of allocations to package_json_reader get / getPackageScopeConfig, under defaultResolve -> defaultGetFormatWithoutErrors.

Additional information

Possible fixes, any one of which would do:

• call getPackageType(), which already exists and returns only the type;
• read the three fields without spreading data;
• cache the parsed exports/imports per package.json instead of re-parsing on each getter access.

We work around it in our CLI with a module.registerHooks resolve hook that caches the package-scope type per directory. That takes the drizzle case from 301 MB to 63 MB. Today every ESM consumer of a package with a large exports map pays this cost.

[inoway46]

Looks like #65812 addresses this.