TS v2.3.1 runs out of memory compiling FP library

TypeScript Version:: 2.3.1

Repo: https://github.com/gcanti/fp-ts

After upgrading typescript from 2.2.2 to 2.3.1 the compilation (npm run build) fails with the following error:

<--- Last few GCs --->

   25725 ms: Scavenge 1407.7 (1457.4) -> 1407.7 (1457.4) MB, 46.2 / 0 ms (+ 24.7 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
   27079 ms: Mark-sweep 1407.7 (1457.4) -> 1406.4 (1456.4) MB, 1354.1 / 0 ms (+ 1071.3 ms in 617 steps since start of marking, biggest step 25.4 ms) [last resort gc].
   28376 ms: Mark-sweep 1406.4 (1456.4) -> 1405.9 (1457.4) MB, 1297.0 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1985f0ce3ac1 <JS Object>
    1: recursiveTypeRelatedTo(aka recursiveTypeRelatedTo) [/Users/giulio/.nvm/versions/node/v5.5.0/lib/node_modules/typescript/lib/tsc.js:~27814] [pc=0xf9f8ea8a65b] (this=0x1985f0c04189 <undefined>,source=0x173e2ea53611 <a Type with map 0x1089a6556cc9>,target=0x95cb168fbb1 <a Type with map 0x1089a6556b69>,reportErrors=0x1985f0c04299 <false>)
    2: isRelatedTo(aka isRelatedTo) [/Users/giulio/.n...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6

Somehow seems related to the modules Tuple.ts and Apply.ts

In the module Tuple.ts if I comment the following lines

declare module './HKT' {
  // interface HKT<A> {
  //   Tuple: Tuple<any, A>
  // }
  interface HKT2<A, B> {
    Tuple: Tuple<A, B>
  }
}

the compilation doesn't crash.

Alternatively if I don't touch Tuples.ts but I comment out all the liftA* functions in Apply.ts the compilation doesn't crash.

Expected behavior:
Compilation success

Actual behavior:
Compilation fail

Here's another repro which involves other modules.

Worth noting that with the following example the code completion feature in VSCode with TypeScript v2.2.2 hangs with CPU 100%

import * as array from 'fp-ts/lib/Array'
import * as validation from 'fp-ts/lib/Validation'

export function pouet(
  xs: number[],
  f: (x: number) => validation.Validation<string[], number>
): void {
  const x = array.traverse(validation. .getStaticApplicative(array))(f, xs) // Long hanging due to this space
}

In the Array.ts module, if I remove the currying in the definition of traverse, i.e from

export function traverse<F extends HKT2S>(applicative: StaticApplicative<F>): <L, A, B>(f: (a: A) => HKT2<L, B>[F], ta: Array<A>) => HKT2<L, Array<B>>[F]
export function traverse<F extends HKTS>(applicative: StaticApplicative<F>): <A, B>(f: (a: A) => HKT<B>[F], ta: Array<A>) => HKT<Array<B>>[F]
export function traverse<F extends HKTS>(applicative: StaticApplicative<F>): <A, B>(f: (a: A) => HKT<B>[F], ta: Array<A>) => HKT<Array<B>>[F] {
  return (f: any, ta: any) => {
    const snocA2 = liftA2(applicative, curriedSnoc)
    return reduce((fab, a) => snocA2(fab, f(a)), applicative.of(empty()), ta)
  }
}

to

export function traverse<F extends HKT2S, L, A, B>(applicative: StaticApplicative<F>, f: (a: A) => HKT2<L, B>[F], ta: Array<A>): HKT2<L, Array<B>>[F]
export function traverse<F extends HKTS, A, B>(applicative: StaticApplicative<F>, f: (a: A) => HKT<B>[F], ta: Array<A>): HKT<Array<B>>[F]
export function traverse<F extends HKTS, A, B>(applicative: StaticApplicative<F>, f: (a: A) => HKT<B>[F], ta: Array<A>): HKT<Array<B>>[F] {
  const snocA2 = liftA2(applicative, curriedSnoc)
  return reduce((fab, a) => snocA2(fab, f(a)), applicative.of(empty()), ta)
}

both the compilation crash and the VSCode hanging go away

Nathan Shively-Sanders (@sandersn)

Nathan Shively-Sanders (@sandersn) One repo showing the last issue Giulio Canti (@gcanti) reported (in case it could help) :
https://github.com/sledorze/vscode-hanging

I can't repro this with typescript@next. Can you try typescript@next?

With TypeScript 2.4.0-dev.20170502

compilation succeeded
Code helper still hangs with 100% CPU (snippet or repo above)

I can't repro that exact behaviour in VS Code. For me, initial load time is very long and then if I type a dot after validation I get completions one time. Afterwards I just get Loading... indicating that the language service is not responsive. Does VSCode itself become unresponsive for you?

Are you using "typescript.tsdk" to point to typescript@next? Here's my settings.json

{
    "typescript.tsdk": "/media/nathansa/src2/ts/built/local",
    "typescript.tsserver.trace": "off",
    // ... other config stuff ...
}

Yours will be somewhat different since you want to point to typescript@next installed from npm instead of built locally from source.

I have a project that uses fp-ts and compilation still fails with this memory error on typescript@next:

❯ tsc --version
Version 2.4.0-dev.20170503

❯ yarn run compile
yarn run v0.22.0
$ rm -rf target && mkdir target && tsc && cp -r ./src/client ./target

<--- Last few GCs --->

   46574 ms: Mark-sweep 1367.1 (1435.0) -> 1367.1 (1435.0) MB, 1402.6 / 0.0 ms [allocation failure] [GC in old space requested].
   48162 ms: Mark-sweep 1367.1 (1435.0) -> 1367.1 (1435.0) MB, 1587.9 / 0.0 ms [allocation failure] [GC in old space requested].
   49752 ms: Mark-sweep 1367.1 (1435.0) -> 1371.7 (1419.0) MB, 1589.5 / 0.0 ms [last resort gc].
   51100 ms: Mark-sweep 1371.7 (1419.0) -> 1376.3 (1419.0) MB, 1347.6 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x27062c3cfb51 <JS Object>
    1: inferFromTypes [/Users/OliverJAsh/Development/twitter-paper/node_modules/typescript/lib/tsc.js:~28893] [pc=0x286345341249] (this=0x27062c3e6f19 <JS Global Object>,source=0x3e76e59c9ba1 <a Type with map 0xb208b868ac1>,target=0x3df62ba20169 <a Type with map 0xb208b86b039>)
    2: inferFromProperties [/Users/OliverJAsh/Development/twitter-paper/node_modules/typescript/lib/tsc.js:~29042] [pc...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/OliverJAsh/.nvm/versions/node/v6.9.5/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/OliverJAsh/.nvm/versions/node/v6.9.5/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/OliverJAsh/.nvm/versions/node/v6.9.5/bin/node]
 4: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [/Users/OliverJAsh/.nvm/versions/node/v6.9.5/bin/node]
 5: v8::internal::Factory::NumberToString(v8::internal::Handle<v8::internal::Object>, bool) [/Users/OliverJAsh/.nvm/versions/node/v6.9.5/bin/node]
 6: v8::internal::Runtime_NumberToStringSkipCache(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/OliverJAsh/.nvm/versions/node/v6.9.5/bin/node]
 7: 0x2863447092a7
sh: line 1: 73747 Abort trap: 6           tsc
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Nathan Shively-Sanders (@sandersn)

For me, initial load time is very long and then if I type a dot after validation I get completions one time. Afterwards I just get Loading... indicating that the language service is not responsive

Nathan Shively-Sanders (@sandersn) 👍 That's the exact behaviour I'm experimenting

Does VSCode itself become unresponsive for you?

No, only the Code Helper is unresponsive (I must close and reopen vscode to regain the functionality)

Are you using "typescript.tsdk" to point to typescript@next?

Yes

Stéphane Le Dorze (@sledorze)

Stéphane Le Dorze (@sledorze) the inference-too-deep cap isn't configurable right now. Setting it too low will start to create bogus inferences of {}. Maybe we should revisit the value, but I don't think it should be configurable. The only time you need it, you're avoiding an architectural problem of the compiler.

Node crashes don't give much of the stack, as you can see. I debugged using the following:

$ node --inspect --debug_brk ~/ts/built/local/tsc.js

Then I opened Chrome to the provided URL and let the compiler run for a while. Each time I did this there were many recursive calls in inference on the stack, so I guessed that inference was causing the out-of-memory crash. (You can substitute any copy of tsc.js that you have, you don't need to build a copy yourself.)

Anders Hejlsberg (@ahejlsberg)

Update from talking to Anders Hejlsberg (@ahejlsberg). The inference-too-deep cap should maybe be removed entirely — it was a mistake from when we believed that inference should behave similarly to assignability checking.

Anders Hejlsberg (@ahejlsberg) is working on a fix that just quits when inference sees an instantiation of a type that it has seen before. That should fix this bug.

I'll also add fp-ts to our real-world code test suite after the bug is fixed so that we can catch compiler bugs in this library and similar FP/monad libraries.

fp-ts is now part of our RWC corpus so we should notice any performance regressions in the future.

This fixed the issue for https://github.com/OliverJAsh/ts-2.4-memory-issue, but I ran into another scenario where the memory issue pops up again. I've created a small test case in https://github.com/OliverJAsh/ts-2.4-memory-issue/tree/2.4.0-dev.20170519.

Nathan Shively-Sanders (@sandersn)

Nathan Shively-Sanders (@sandersn) Do you have any idea what is causing the above?

Nathan Shively-Sanders (@sandersn)

Nathan Shively-Sanders (@sandersn) note that the problem happens with fp-ts when there's a type error in the program, not compiling an actually well typed program (this is from my observations).

Also considering this problem, the worst case is not when one changes some code and have the compiler getting lost in inference, the real issue is when one upgrade a library and it breaks the compiler without any clue where the type error is and how to fix it.

Nathan Shively-Sanders (@sandersn)

Nathan Shively-Sanders (@sandersn) it should be reopened IMHO

Stéphane Le Dorze (@sledorze)

Stéphane Le Dorze (@sledorze) I opened a new bug based on Oliver Joseph Ash (@OliverJAsh)'s repro: #16029. If you have another repro please link to it there.

I'm keeping this bug closed just to make bookkeeping simpler. And it's unlikely that the compiler is getting stuck in inference again (or at least, for the same reason).

I'm not sure whether or not the pain point I had today was caused by this issue, but I also was running into the problem of Node running out of space on the heap.

I reverted to 2.3.2, and was able to successfully compile. If you guys can nail down this bug, it would be greatly appreciated!

Bryan Rayner (@bryanerayner)

Bryan Rayner (@bryanerayner) there are, unfortunately, many reasons for typescript to run out of memory while compiling fp-ts. The root cause of this issue is definitely fixed: type inference no longer recursively examines types that it has already seen. But (for example) assignability checking still does recursively examine types. It's likely that the issue you're hitting is different. Can you open a new issue and give some detail about your code there?

Navigation Menu

Repository navigation

TS v2.3.1 runs out of memory compiling FP library #15443

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TS v2.3.1 runs out of memory compiling FP library #15443

Description

Activity

gcanti commented on Apr 29, 2017

sledorze commented on May 2, 2017

sandersn commented on May 2, 2017

gcanti commented on May 3, 2017

sandersn commented on May 3, 2017

OliverJAsh commented on May 3, 2017

gcanti commented on May 3, 2017

15 remaining items

sandersn commented on May 15, 2017

sandersn commented on May 15, 2017

sandersn commented on May 17, 2017

OliverJAsh commented on May 20, 2017

OliverJAsh commented on May 22, 2017

sledorze commented on May 23, 2017

sledorze commented on May 23, 2017

sandersn commented on May 23, 2017

bryanerayner commented on Aug 13, 2017

sandersn commented on Aug 16, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions