[API Question] Is there a type resolution depth limit difference between compiling and running the language service?

TS 3.3.0-dev.20190103

I don't exactly have a minimal repro but I figured I'd ask my question first.

I'm experiencing a problem that I can only think is because TS is "giving up" on resolving the type in the IDE. But it resolves the type correctly during compile time.

IDE

I'm currently seeing this in my IDE,

And it seems to think that the following type, ReturnType<SelectDelegateT>[index] is any.

It also thinks that ReturnType<SelectDelegateT> is any.

However, when I just use SelectDelegateT, it gives me the full type, which is correct.

Compile-time

It was really weird because I was pretty sure I shouldn't get an error. I gave up on the IDE and decided to run tsc. I configured it to have noEmitOnError : true.

So, if there really is an error, it should spit out the same error as the IDE and not emit any types.

However, tsc ran fine. No errors.

I checked the emitted .d.ts file and the emitted type was resolved correctly.

So... Is my hunch right? Is there a "depth limit" difference between the language service that runs for the IDE (VS Code) vs the compiler?

And if so, is there a way for me to make it so that both have the same limit?
If possible, I'd like to make the limit as large as possible for both VS code and during compile-time.

The other weird thing is that part of my code looks like this,

{
    [index in Extract<keyof ReturnType<SelectDelegateT>, string>]: (
        ReturnType<SelectDelegateT>[index] extends IColumn ?
        (
            ReturnType<SelectDelegateT>[index] extends ColumnUtil.FromColumnRef<ColumnRefUtil.FromQueryJoins<QueryT>> ? never :
            ["Invalid IColumn", ReturnType<SelectDelegateT>[index]]
        ) : 
        never
    )
}

And the red squiggly line gives me ["Invalid IColumn", any]

Intuitively, to me,

If ReturnType<SelectDelegateT> really is being resolved to any, then Extract<keyof ReturnType<SelectDelegateT>, string> should be string.

ReturnType<SelectDelegateT>[index] would be any and so,

ReturnType<SelectDelegateT>[index] extends IColumn ? should be true
ReturnType<SelectDelegateT>[index] extends ColumnUtil.FromColumnRef<ColumnRefUtil.FromQueryJoins<QueryT>> ? should be true

And if the above two are true, then it should have been never and not ["Invalid IColumn", any]

The other weird thing is that I tried to shift some types around to change the error output and I get,

'{ [x: string]: ["Invalid IColumn", any, string]; }'.
    Index signature is missing in type '() => [Expr<{ readonly usedRef: { readonly merchant: { readonly appId: Column<{ tableAlias: "merchant"; name: "appId"; assertDelegate: AssertDelegate<bigint> & { __accepts: bigi/*snip*/

After the Index signature is missing in type ' part, it gives me the full type, resolved correctly. and not any! So, it is inferring the type of SelectDelegateT correctly, for sure.

But the fact that it thinks the type is '{ [x: string]: ["Invalid IColumn", any, string]; }'. means that it failed to resolve ReturnType<SelectDelegateT>[index] correctly =/

I'm so confused =(

Just updated to TS 3.3.0-dev.20190119

It seems like the type resolution depth limit is the same as the one in the language service now because even tsc is giving me errors.

Now I just gotta' figure out how to increase this limit =/

So, I just shifted some code around and it got rid of that any error ._.

Before:

export function isEnabled () {
    return o.and(
        t.merchantEnabledLog.latestValueOfEntity(
            c => c.enabled
        ),
        business.isEnabled(),
        appPlatform.isEnabled(),
        payOutMethod.isEnabled(),
    );
}

After:

export function isEnabled () {
    return o.and(
        o.and(
            t.merchantEnabledLog.latestValueOfEntity(
                c => c.enabled
            ),
            business.isEnabled(),
            appPlatform.isEnabled(),
        ),
        payOutMethod.isEnabled(),
    );
}

o.and() is a function that takes a tuple. It seems like splitting the tuples to lengths of 3, 2 instead of using a single tuple of length 4 helped. I don't know why it helped but it did.

The resultant type for both "Before" and "After" is the same.

I'm also having this weird problem where VS code highlights folders and files in red (to say they contain errors).

And when I open the files, VS code underlines some text with red squiggly lines, initially. Hovering over the lines gives me TS errors, all related to types not resolving properly (Becoming any).

But, after a few moments (seconds to minutes), VS code removes the red squiggly lines. I assume because it types checks successfully.

Closing the files makes VS code highlight those folders and files again after a few moments (seconds to minutes).

Makes me think that the depth limit is different for "background" vs "foreground" type checking. If that makes any sense at all.

I think I'm most definitely sure it has to do with some depth limit.

I have a bunch of calls to functions that are generics and rely on type inference.

declare function and</*Some Generic Type Params*/> (.../*Some Params*/) : /*Some Return Type*/;
function isEnabled1 () {
    return and(
        /*expr*/,
        /*expr*/
    )
}
function isEnabled2 () {
    return and(
        isEnabled1(),
        /*expr*/,
        /*expr*/,
        /*expr*/
    )
}
function isEnabled3 () {
    return and(
        isEnabled2(),
        /*expr*/,
        /*expr*/
    )
}
/*etc.*/

The return type of all these functions resolves fine on their own.

But when I start doing stuff like,

/*expr*/.select(() => [isEnabled3()])

And select() has types that use ReturnType<>, it all breaks down and resolves to any

I'm also having this problem where, on occasion, types are getting inferred as never and then the never types go away after a while.

But they always appear when I run tsc.

At this point, I'd be happy to mail my code to someone to look at what I'm doing wrong =/
I guess I'll try and get a minimal working repro this weekend or something.

Wesley Wigham (@weswigham)

I think it's got to do with some kind of iterative type resolution thing?

For example,

.where(canStartCreate)

Hover over canStartCreate, see an incorrect type resolution of type never.
Go to declaration of canStartCreate.
Start hovering over all expressions that make up canStartCreate.
Every time an incorrect type resolution is found, go to that declaration and hover over all sub expressions.
At the end, hover over canStartCreate again. It now has the correct type resolution.

I guess when I run tsc, it doesn't search deep enough and gives up and resolves to any or never.

Wesley Wigham (@weswigham) Sorry I'm tagging you directly but this is a little frustrating for me. I'm not too familiar with compiler internals, let alone tsc's but I was wondering if there is some kind of quick fix I could do on my end to bump the depth limit. Maybe change a constant in a file somewhere =/

I finally got it to compile successfully!

So, after a lot of random Ctrl+F > depth and searching through the code base, I finally found what I had to change,

I changed if (instantiationDepth === 50) { to if (instantiationDepth === 500) { in checker.ts and it compiled successfully in 123.49s.

Before the change, it failed in 71.92s.

Files:          3895
Lines:        130128
Nodes:        823248
Identifiers:  257678
Symbols:      702177
Types:        249365
Memory used: 569673K
I/O read:      0.52s
I/O write:     5.67s
Parse time:    9.24s
Bind time:     2.24s
Check time:   50.22s
Emit time:    61.79s
Total time:  123.49s

I ran into a lot of dead-ends trying to bump the flow depth limit, constraint depth limit, etc. But it seems like instantiationDepth was what I really needed.

I'm pretty happy with this because v2 of a library I worked on would take 180+s to compile.
Having a project use v3 and compiling in 123.49s is a pretty huge speed boost to me.

[EDIT]

Changed the limit to 250 and it still compiled successfully. Similar compile time as 500 (makes sense).

Files:          3895
Lines:        130128
Nodes:        823248
Identifiers:  257678
Symbols:      702177
Types:        249365
Memory used: 628652K
I/O read:      3.74s
I/O write:     1.51s
Parse time:   15.28s
Bind time:     2.30s
Check time:   49.50s
Emit time:    54.96s
Total time:  122.05s

I'm going to keep trying to decrease the limit until I get compile errors again.

I'm pretty happy with this because v2 of a library I worked on would take 180+s to compile.
Having a project use v3 and compiling in 123.49s is a pretty huge speed boost to me.

😮 How many LoC is this project? 50s check time and 54s emit time is uuuge.

I've noticed that the instantiationDepth is hard-coded to 50 with the following comment,

// We have reached 50 recursive type instantiations and there is a very high likelyhood we're dealing
// with a combination of infinite generic types that perpetually generate new type identities. We stop
// the recursion here by yielding the error type.

Would be nice if we could make this configurable with a tsconfig.json.

I just noticed you asked me about LoC, I'll respond in a second :x
Need to look some things up.

Here's v3 of the library I'm working on. On npm, it's currently at version 1.13.1.

https://github.com/AnyhowStep/typed-orm/tree/v3

You can download the project and run npm run sanity-check and it'll recompile the library and run a bunch of tests on it.

It's an experiment in compile-time structurally-safe MySQL query building ;D

It takes about 40s to compile that library on my laptop.

That library doesn't suffer from this instantiation depth problem because I never wrote compile-time tests for nested MySQL expressions more than 2 or 3 levels deep.

However, in another project I'm working on (for work), it has MySQL expressions that are 5-6 levels deep, using the v3 library. So, I ran into this instantiation depth limit.

I think v3 of the library is about 50k LoC.

I think the emit time is really long because the generated types are really large. Not because I have many lines of code :x

For example, this is a compile-time test input file,

https://github.com/AnyhowStep/typed-orm/blob/v3/test/compile-time/input/query/select/from-table.ts

14 lines of code.

The output is here,

https://github.com/AnyhowStep/typed-orm/blob/v3/test/compile-time/expected-output/query/select/from-table.d.ts

81 lines of code.

[EDIT]

This project using the v3 library is currently at about 27k LoC, but only about 14k are using the library directly.

[EDIT]

Brought the limit down to 75 and it still compiles successfully!

Gonna' keep going lower.

[EDIT]

65 was too low and I exceeded the instantiation depth =(

[EDIT]

70 is still too low xD

[EDIT]

73 was still too low.
It compiled successfully with 74.

It seems like 74 is the lowest I can go at the moment before I get those errors again. I'm pretty sure I'll have to bump this higher in future when I start having more complex queries.

In addition to being able to configure the max instantiationDepth, it would be nice if an error message was added. Something like "Type resolution exceeded max instantiationDepth; consider simplifying the type or increasing the max instantiationDepth".

Then again, I'm not even sure how many people have realistically run into this problem to make this a worthwhile endeavor =(

It came up once before when someone was nesting 200-some conditional types deep because they'd written what amounted to an else-if block of every syntax kind in TS. They just worked around the limit by reordering their type, though (so they had 200 top-level conditionals in a union instead of 200 levels of nesting).

I finally got around to checking where my instantiation depth was going crazy because I was finally getting 800-ish seconds compile times =P

Updating to TS 3.4 gave me 300+ errors.

It seems like most of my problems come from using Expr.as() as opposed to ExprUtil.as().

However, Expr.as() is just,

class Expr {
    as (alias) {
        return ExprUtil.as(this, alias);
    }
}

I've noticed a similar thing before where using classes+methods gave me a lot of problems with type-checking/compile-times while using interfaces+functions worked far better.

That's why I have IXxx interfaces for every Xxx class I have.

And every class method is just a wrapper around functions that take interfaces as arguments.

I'm not sure why this is the case.

Another big contributor is that ExprUtil.as() was returning,

As<> = Expr<> & IExprSelectItem<>;

When I replaced the class Expr<> (which has a bunch of methods) with the interface IExpr<> (which has no methods), the instantiation depth got better in many places.

However, replacing with the interface means I can no longer use that fancy "fluent API" design, though.

After a lot of... Shifting function calls around (about 200-ish of them), and reverting to TS 3.3, it finally type checks successfully in two minutes!

Files:           4768
Lines:         427316
Nodes:        1704858
Identifiers:   589521
Symbols:      1744579
Types:         440293
Memory used: 1176821K
I/O read:       0.48s
I/O write:      0.00s
Parse time:     9.92s
Bind time:      2.45s
Check time:   121.89s
Emit time:      0.00s
Total time:   134.26s

I'm so happy T^T

So, after 3 days and more code, it started taking 8 minutes to do a full build.
And about 4 minutes to type check.

I decided to experiment some more and check what's giving me these issues.

It seems like when I write expressions/sub-queries that require many-table joins, I end up with long check times.

For example,

import * as o from "typed-orm";

const manyJoinExpression = o.and(
    dao.merchant.isEnabled(),
    dao.business.isEnabled(),
    dao.payOutMethod.isEnabled(),
    /* snip */
);

//merchant.ts
export function isEnabled () {
    return o.and(
        t.merchant.columns.enabled,  //Requires merchant table
        user.isEnabled(), //Requires user table
        /* snip */
    )
}

At the end of it all, the emitted .d.ts of manyJoinExpression ends up being 300-ish lines after compiling!

In the below screenshot, you'll see a lot of o.Column<> types. These are all required by the expression for it to be used in other queries.

I decided to refactor about 7 of these long expressions to only require one table join but to use sub-queries to get access to the other tables.

const expr1 = o.exists(o.requireParentJoins(tableA)
    .from(tableB)
    .where(o.innerJoinPk(tableA, tableB).eq)
    .where(dao.tableB.isEnabled)
);
const expr2 = o.exists(o.requireParentJoins(tableA)
    .from(tableC)
    .where(o.innerJoinPk(tableA, tableC).eq)
    .where(dao.tableC.isEnabled)
);
/* snip */

const longExpr = o.and(
    expr1, expr2, expr3, expr4, /* snip */
);

So, for this longExpr, while it may "access" many columns, to use this expression in a query, you only need tableA in the FROM/JOIN clause.

The emitted type will only list tableA's columns as required.

The result is that it now takes roughly 2.5 minutes to type check

Even though both queries have the same effect, their compile times are very different ._.

It feels like that regex quote,

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

In my case,

I had a problem that writing SQL queries is generally not type-safe. Using = when you meant <=>, adding strings and numbers, pretending a string is a date, joining tables using columns of different data types, joining the same table twice without aliasing, etc.

I decided to write a type-safe query builder. Now, I have more problems.

After refactoring one query that was joining 5 tables, the check time is 105s or 1m45s.

After refactoring a second query that was joining 6 tables... No improvement.
It actually took 114s but that's not significantly different. Maybe a GC pause or thread interrupt or something.

After refactoring a third query that was joining 5 tables... No improvement.
112s.
Something else improved, though. I started getting live type checks and auto-complete suggestions on VS code again!

I had to wait about 5-20s for them but they did come up.
Before, I'd get nothing. Or had to wait 10+ minutes for type/syntax errors to show up.
Then, after fixing errors, nothing again.
Had to restart VS code and wait a long time for type/syntax errors to show up.

After a fourth query with 7 tables... No improvement.
117s

After a fifth query with 7 tables... No improvement.
123s

At the end of it all, it took 126s.
Beyond the first refactored query, it looks like deleting code increased my compile time.
126s is still better than 2.5 minutes and significantly faster than 4 minutes, though.

The overall build time has improved somewhat,

Files:           4829
Lines:         432389
Nodes:        1724119
Identifiers:   596241
Symbols:      1780290
Types:         452835
Memory used: 1188208K
I/O read:       0.66s
I/O write:      8.05s
Parse time:    13.38s
Bind time:      3.10s
Check time:   116.93s
Emit time:    106.00s
Total time:   239.41s

tsc reported 3.9 minutes but I ran date ; (tsc -d --diagnostics) ; date and it actually took 5m53s.

Before refactoring,

Files:           4829
Lines:         432334
Nodes:        1724079
Identifiers:   596216
Symbols:      1813474
Types:         458844
Memory used: 1227975K
I/O read:       0.73s
I/O write:     13.73s
Parse time:    14.80s
Bind time:      3.39s
Check time:   210.63s
Emit time:    187.08s
Total time:   415.90s

tsc reported 6.9 minutes but it was actually 8+ minutes

Anders Hejlsberg (@ahejlsberg)

Hi Anders Hejlsberg (@ahejlsberg) Wesley Wigham (@weswigham) AnyhowStep,

I recently published an article about how to create types for curry and ramda. Quite a few people are excited and waiting for me to add these types to DefinitelyTyped. But I can't pass the lint tests yet.

https://medium.freecodecamp.org/typescript-curry-ramda-types-f747e99744ab
https://github.com/pirix-gh/medium/tree/master/src/types-curry-ramda

To solve these kind of problems, I detail how I make use of recursive types. Indeed TS 3.4.0-dev.20190302 is breaking these types. In the latest stable version (3.3.3333) warnings arise (only) when we recurse more than 45 times. A recursive type then returns any if the limit is exceeded (which is nice).

In TS 3.4.0-dev.20190302 it appears that "Type instantiation is excessively deep and possibly infinite", which is a breaking behaviour. But in fact it is only possibly infinite, and this is why the previous behaviour should be preferred. any could just be returned anytime that limit appears to be exceeded.

I referenced this issue in a new ticket #30188

I haven't moved any of my code to TS 3.4 because it also breaks a lot of my code. I don't have the time to look through the hundreds of new errors that popped up. Maybe some day =/

That said, having the compiler introduce an any type silently is a bad idea, in my opinion. For one, it disables type checking and can allow otherwise erroneous code to compile. Or, in my case, it just broke type checks and said something was not assignable when it clearly was (to a "human compiler")

It was very confusing for me to see types resolved as any when I try really hard to avoid them in my projects. And even more confusing was that the cause was not obvious, and would happen in some cases and not others.

Silently introducing any is like silently adding a potential run-time error.

Also, cool article. Have you used it on a large-ish project? Does it impact compile times? I remember experimenting with tuple type concatenation, push, pop, etc. and compile-time math with recursive types but it made compile times ridiculously slow in large projects (where I initially had an 18 minute compile time). But this was quite some time ago.

AnyhowStep

AnyhowStep No compile problems so far, it's fast!

And to come back to what you said, it would indeed be nice to be able to set that recursion limit. In TS 3.3 it is set to 45 and in TS@next it is set to 42. I would come with an in-between solution. A limit value could be defined in tsconfig.json to increase the recursion count. If exceeded, any would be attributed to a recursive type's return. This way, no behavior is broken in any of the cases.

Just thought I'd update this issue with a workaround to avoid reaching the max instantiation depth.

This workaround will not always apply to your situation but it can be useful.

You have a variable that looks like this, const complexType : UnexpandedType<Foo>;
You try to use complexType in an expression, bar(complexType)
You get the error complaining about the max instantiation depth being reached

Workaround

This workaround abuses "composite projects".

Create a different project; we'll call it project-complex; it will be a composite project
Shove const complexType : UnexpandedType<Foo> into project-complex.

In your original project, make it composite and make it reference project-complex.
Something like,

//in tsconfig.json
"references" : [
    {
        "path": "./../project-complex",
    },
]

Build both project-complex and your original project.

Now, in your original project, you may use bar(complexType) and it should be fine.

The idea behind the workaround is that resolving each type involves a certain amount of "depth currency". If it costs too much of this currency to resolve the type, TS complains.

When building project-complex, there is a good chance that the emitted .d.ts file will not look like,

export const complexType : UnexpandedType<Foo>;

Instead, the type of complexType will probably be "expanded",

export const complexType : {
    my : {
        ridiculously : {
            complex : Foo,
        }
    },
    someArray : [1,2,3,Foo,"blah"],
    /*snip 100+ lines and objects nested 7 levels deep*/
};

Since complexType will already be expanded, it will not have to go as deep to resolve the type when used in your original project.

Before workaround,

Original project check start
Expression bar(complexType) encountered
Allocate 50 (or whatever arbitrary amount) to resolving type of expression
Spent 40 to resolve complexType
Spent 10 to try to resolve bar(complexType)
No more "depth currency"; still haven't finished resolving
Complain

After workaround,

project-complex build start
complexType encountered
Allocate 50 (or whatever arbitrary amount) to resolving type of expression
Spent 40 to resolve complexType
Emit type of complexType, but "expand" the type in the .d.ts file
project-complex build successful
Original project check start
Expression bar(complexType) encountered
Allocate 50 (or whatever arbitrary amount) to resolving type of expression
Spent 20 to resolve complexType (Expanding a type doesn't necessarily mean the cost to resolve it is zero)
Spent 25 to resolve bar(complexType)
Total cost is 45
No complaints; bar(complexType) resolved successfully

Your mileage will depend on how much you can get TS to expand the type when emitting project-complex. If the emitted type isn't expanded, this will not help you at all.

There are ways to get TS to expand the type for you during compile-time, you'll have to experiment or look for them elsewhere, though. I don't have a definitive guide for that.

@pirix-gh
I just saw your repo https://github.com/pirix-gh/ts-toolbelt

It looks very tempting to try.
I love that there are so many convenient helper types in there.
My only concern is how much "depth currency" it'll eat up in my projects =/

Anders Hejlsberg (@ahejlsberg)

Anders Hejlsberg (@ahejlsberg)

I just had a random thought. Since the maxInstantiationDepth is an implementation detail and subject to change in the future, I was thinking that, instead of configuring it with "absolute numbers", it could be configured with vague settings instead.

Instead of maxInstantiationDepth : number, it could be something like,
maxInstantiationDepth : "default"|"high"|"very-high"|"infinity".

I probably shouldn't even be using the word maxInstantiationDepth but I feel like the concept of "how-deep-the-checker-has-to-go-into-a-type-to-resolve-it" will stick with TS no matter how advanced the checker gets.

If not maxInstantiationDepth, I can't think of a good enough word for the concept.
Thoughts?

Just adding a link for me to personally keep track of limits related to type instantiation,

#32079

Just adding more notes to myself. It seems like someone else asked for higher depth limits/customizable depth limits wayyyyy before me.

#13052

AnyhowStep

AnyhowStep FWIW the style of super-long and multiple updates is very confusing from our side of the fence - it's not clear when you're "done", which comments have been superseded by later content, and it's not at all obvious which comment is the one that summarizes the situation.

I'm going to close this; if you have a concise and concrete thing to work from that demonstrates some kind of manifest bug, we're of course happy to take a look at a fresh issue. Thanks again!

Navigation Menu

Repository navigation

[API Question] Is there a type resolution depth limit difference between compiling and running the language service? #29511

IDE

Compile-time

8 remaining items

Workaround

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[API Question] Is there a type resolution depth limit difference between compiling and running the language service? #29511

Description

IDE

Compile-time

Activity

AnyhowStep commented on Jan 21, 2019

AnyhowStep commented on Jan 21, 2019

AnyhowStep commented on Jan 21, 2019

AnyhowStep commented on Jan 22, 2019

AnyhowStep commented on Jan 25, 2019

AnyhowStep commented on Jan 25, 2019

AnyhowStep commented on Jan 25, 2019

AnyhowStep commented on Jan 25, 2019

weswigham commented on Jan 25, 2019

AnyhowStep commented on Jan 25, 2019

AnyhowStep commented on Jan 25, 2019

AnyhowStep commented on Jan 25, 2019

AnyhowStep commented on Jan 25, 2019

weswigham commented on Jan 25, 2019

8 remaining items

AnyhowStep commented on Feb 12, 2019

AnyhowStep commented on Feb 12, 2019

AnyhowStep commented on Feb 12, 2019

AnyhowStep commented on Feb 15, 2019

AnyhowStep commented on Feb 15, 2019

millsp commented on Mar 2, 2019

AnyhowStep commented on Mar 2, 2019

millsp commented on Mar 2, 2019

AnyhowStep commented on Jun 23, 2019

Workaround

AnyhowStep commented on Jun 23, 2019

AnyhowStep commented on Jun 23, 2019

AnyhowStep commented on Jun 26, 2019

AnyhowStep commented on Aug 5, 2019

RyanCavanaugh commented on Aug 22, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions