Expose incremental Compile-on-Save builder API

Sheetal Nandi (@sheetalkamat)

In #10879, Sheetal Nandi (@sheetalkamat) is working on bringing the same logic in the language server to Compile-on-Save. We would like to expose that as an API so that other build tools (e.g. Webpack loaders, gulp plugins, Broccoli plugins) and editors (any editor using TS Server) can get:

automated resolution logic to detect which files need to be "updated"
incremental type-checks to only check the bare minimum of files
- this means that, for example, requesting errors & building will only trigger a type check on updated/new/impacted files. By "impacted files", I mean files whose type information could potentially have been changed by other updated/new files.

Some prior work that motivated the scenarios here can be seen on PR #3687 by Dirk Bäumer (@dbaeumer).

We'd like to elicit feedback from others in the community. Some folks that immediately come to mind are Sean Larkin (@TheLarkInn) John Reilly (@johnnyreilly) James Brantly (@jbrantly) Stanislav Panferov (@s-panferov) Matt Bierner (@mattbierner) Johannes Rieken (@jrieken) Dirk Bäumer (@dbaeumer) Alex Eagle (@alexeagle) Chuck Jazdzewski (@chuckjaz) Ivo Gabe de Wolff (@ivogabe) and Kris Selden (@krisselden) (Forgive me if I excluded anyone, that's already a decent number of people 😄).

Piotr Oleś (@piotr-oles)

Exciting!! cc Piotr Oleś (@piotr-oles) Basarat Ali Syed (@basarat) Blake Embrey (@blakeembrey) Herrington Darkholme (@HerringtonDarkholme) Morten Houston Ludvigsen (@MortenHoustonLudvigsen)

It sounds like the addition of this will make for simpler build tools. Put simply, I figure certain tools will need to do less work with this is in place. Perhaps it would be useful for people to list "things their tools do" which, with the addition of this functionality, they think they won't need to anymore.

If we cover that we will hopefully ensure that what lands will work in the way that everyone hopes.

James Brantly (@jbrantly) - I know you're less involved with ts-loader these days but whenever you pitch in your insights are really useful. If you could get a moment to comment on this I think it would be really valuable for everyone. (Obviously no worries if you can't)
❤️ 🌻

John Reilly (@johnnyreilly)

Will this be an api that has both sync and async variants?

Additionally, this sounds awesome so far. I think this lets both John Reilly (@johnnyreilly) and Stanislav Panferov (@s-panferov) remove the soon-to-be deprecated accessing of tapable instances inside the loaders. (Which means tools like happypack Hans Larsen (@hansl) and friends can use @ngtools/webpack maybe as a loader instead (if not much has changed there). Just many many opportunities are now possible, etc.

💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯

Sean Larkin (@TheLarkInn)

Hi Sean Larkin (@TheLarkInn),

soon-to-be deprecated accessing of tapable instances inside the loaders

Do you have a link you could share to this please Sean? I hadn't heard this was happening and would like to read up on it. Thanks!

Which means tools like happypack Hans Larsen (@hansl) and friends can use maybe as a loader instead

As an aside, you can use ts-loader with happypack right now. It essentially works by flipping ts-loader into transpile mode and preventing reporting of errors to webpack in the way that ts-loader normally does. (Type checking is moved into another process via a plugin.)

The reason I mention that is the preventing of reporting to webpack etc is I suspect where we make use of tapable instances. I just want to be clear that I understand where we're hooking into this if that's going to need to change. Thanks!

Sheetal Nandi (@sheetalkamat)

To answer the question about sync vs async, I don't think we will have an async API since the compiler runs synchronously.

Otherwise, something that would be helpful for Sheetal Nandi (@sheetalkamat) to know is what sort of API you all expect to be able to use. Do we have users who need to be able to query for only new diagnostics from files, or is it whole-program errors every time?

Currently the stripped down design is something like the following

createWatchMode(
    sysHost: System, 
    callback: (program: Program, changedFiles: string[]) => void,
): void

With that, a callback could potentially always just query the program for when it needs new diagnostics, or needs to emit those specific files. It could be that consumers are responsible for caching diagnostics themselves.

Daniel Rosenwasser (@DanielRosenwasser)

Daniel Rosenwasser (@DanielRosenwasser) for additions/removals would one diff the prior program source files to the program passed to the callback?

Kris Selden (@krisselden)

Kris Selden (@krisselden) The changedFiles is list of addition/removed/changed files name list.

Do we have users who need to be able to query for only new diagnostics from files, or is it whole-program errors every time? .... It could be that consumers are responsible for caching diagnostics themselves.

My feeling is that having "whole-program errors every time" is simpler from a consumption perspective. Less scope for getting things wrong, less work to do (selfish I know!)

Looking at the API, I'm assuming the idea is you'd call the callback with changed files and then query for diagnostics immediately after? And this would be fine as the API is sync not async?

Obviously you're only at design stages right now but if you have a chance, I'd love to see some simple code examples of how you would expect this API to be used. Personally, I find it easier to think about things when I see examples of how something plugs together (and also I've found that when I take things out of the abstract to the practical it shows up considerations I hadn't pondered.) 🌻

Alex Eagle (@alexeagle)

Also I guess this is #9282, sorry Alex Eagle (@alexeagle).

This is interesting. Could you provide some more details on what features this compile-on-save API exposes? Is it a "read-only" API that simply informs external services of TypeScript's dependency model, or can it also be used to inject other tasks into the dependency graph? Something like the latter would be pretty useful for doing codegen.

With the current Compiler API, a custom CompilerHost can be provided with adjusted methods for e.g. getSourceFile. Will this proposed createWatchMode API allow such modifications (such as for createProgram)? I haven't worked on any open source tool, so I don't know if that would be required. Though it would be useful for an internal watch script I made recently with the Compiler API.

It will be great if command line interface of TypeScript compiler (tsc) will also support incremental building. This mode allows embed tsc in command pipeline much easier, e.g.

# Compile and run tests
tsc -p tests --outDir tests-out && mocha tests-out

Now this command has two disadvantages:

need wait for full rebuild of the project described by tests/tsconfig.json;
or need keep watch mode (tsc --watch) as separate process.

ikokostya

ikokostya there are projects like ts-watch that allows you to do watch + trigger onSuccess.

is the builder API still on track for 2.7?

ikokostya

ikokostya for now you can use tsc-watch to re-run a task on successful build

Should be fixed by #20234

Navigation Menu

Repository navigation

Expose incremental Compile-on-Save builder API #17493

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Expose incremental Compile-on-Save builder API #17493

Description

Activity

johnnyreilly commented on Jul 29, 2017

TheLarkInn commented on Jul 31, 2017

johnnyreilly commented on Jul 31, 2017

DanielRosenwasser commented on Aug 1, 2017

krisselden commented on Aug 1, 2017

sheetalkamat commented on Aug 1, 2017

johnnyreilly commented on Aug 2, 2017

DanielRosenwasser commented on Aug 4, 2017

2 remaining items

masaeedu commented on Sep 28, 2017

csvn commented on Nov 1, 2017

ikokostya commented on Jan 8, 2018

kenotron commented on Jan 8, 2018

OneCyrus commented on Jan 8, 2018

gilamran commented on Jan 13, 2018

mhegazy commented on Jan 20, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions