Repository navigation
Online Module Summit 2018-04-03 #9
Description
Activity
Would people find a "participating in open source" or "how the node project works" useful?
Reacted by Gil Tayar, Wassim Chegham and Matt DuLeoneGiven that I have both a presentation and a blog post about the current state of things, I can volunteer to do a presentation about that (i.e. current state, and differences between ESM and CJS).
Another topic to add to this:
- The current proposals for NodeJS modules (e.g. the npm proposal and the loader:esm proposal), and how they affect things like ESM-CJS interop, migration from CJS to ESM, browser compatibility and interoperability, etc.
In no particular order, these are all the (very specific) questions I have about the current state of affairs:
- Why does this need to ship in Node 10? I understand the LTS cycle and the desire to see this done, but this group is only being formed in Feb. - 3 months feels like a short time to come to consensus on and implement all semver-major changes - and shipping something that could ultimately be substandard should be unacceptable. Whatever is shipped will be here forever (or close to it) - the time should be taken to get it right. I understand the desire to see it shipped unflagged (and I would, too), but I don't think it should be a hard requirement if we're not satisfied.
- Who are the abstract stakeholders for this feature? What are each group's needs and how do each of these groups needs differ? Which groups are the most important to satisfy with this feature? Off the top of my head, I've got the following: Library authors (utilities, typically cross-platform, sometimes platform-specific), framework authors (front-end, back-end, fullstack - each have different desires), application authors (again, front-end, back-end, and fullstack), and transpiler or tool authors (since they end up needing to paper over the differences in any of the platforms and runtimes that surface).
- Who are the concrete stakeholders for this feature (is it everyone who was invited to this group? 😛 )? Who's already committed to migrating to node's esmodule implementation when it comes out, and who's committed to starting new projects with it? What are their concerns? Their needs? Their reasons for moving in the first place?
- What, specifically, is going to make this feature more attractive than a transpilation step to commonjs, especially if running on multiple platforms is going to need a buildstep, anyway? This answer is likely different for each of the stakeholder groups identified above.
- What upcoming es features, if any, end up affecting this? How do we reconcile future syntactic changes to modules in the spec (like new
exportorimportforms) with a desire for a stable, broadly compatible platform?
And more in the technical space of the current state of affairs:
- Does module execution order have to be maintained compared to what the user wrote? If interop allows it and they have commonjs and esm imports, what about then? (It seems like the answers are yes and no, respectively, based on the current state of things, but I'd like more insight and specific reasoning documented here)
- What parts of the module spec must be followed to the
Tin the node es module implementation and why, from a technical perspective? (JS engines have a long history of implementation spec nonconformance in some edge cases - usually for good reasons, such as compatibility, though I think most would prefer not to admit that) I don't think giving up on a large set of the end-user features we'd like to have a good transition or ecosystem experience because we've been told that it'd be nonconfoming in ways that are unlikely to matter in practice is not a great reason to give up (IMO, this indicates a failing or the original spec to account for the node use-case particularly well, and may warrant revision of the spec by the tc39 in the future, once all the usecases are known and possibly there's even a nonconforming implementation to standardize on). - What browser esm do we expect to be executable in node (without processing by a tool)? What node esm would we expect to be executable in the browser (without processing by a tool)? (There's no conditional imports in es6 modules, so all feature detection must happen via globals, and all imports must be unconditional - these are some big caveats!) I, personally, would expect something like lodash or another util library to be able to to be cross-consumed; but today there are a pile of constraints on that (mostly by the browser) - only relative paths, all extensions must be specified (and browsers/webservers are going to lag with adding support for detecting/serving up
.mjsfiles as javascript, which is another challenge imposed on evolving the node ecosystem by browsers). Realistically, this means most code written in a node-first way will need preprocessing before being consumable by the browser. Additionally, any browser code that uses fully qualified URLs will need preprocessing before being consumed by node. In some ways, this is a huge step backward from commonjs and amd module loaders, since you'll definitely need to at least check your modules in some tool before consuming them on some platform. 😞
Reacted by Daniel Rosenwasser, Benjamin Gruenbaum and Steveni can answer
Does module execution order have to be maintained compared to what the user wrote?with a strong yes. check out nodejs/node#16675 near the end.i can answer Does module execution order have to be maintained compared to what the user wrote? with a strong yes. check out nodejs/node#16675 near the end.
I'm aware, but the notes from the tc39 meeting also say
DH: It feels like sometimes a spec is trailing reality, not leading it. We can just go away to implementations and after studying what the implementation details are we can spec it. I feel from the room we have a consensus of wanting something in spec but not knowing enough about the problem.
And
Conclusion/Resolution
This goes against the sort of spirit of the specification, though we have no particular normative spec text to add right now. Spec may in the future add normative text around this.Which implies "that sounds nice, but we can't actually guarantee it (at least in simple terms), and it isn't actually true if you consider that imports are cached".
Additionally, that only matters within a specific module graph anyway - consider that if any import can be a dynamic import object using a loader hook, then theoretically any import could cause a new subgraph to be created (via the execution of a new vm context during resolution) which inherently has ordering unbound form the parent graph (though, do these share a cache like
requires do? If they don't you get multiple execution instead of out-of-order execution, which is arguably worse - though you could make a global cache yourself... I suppose you'd have to in order to handle circularities). There's a lot of nice-to-haves there that can't actually be guaranteed (at least not in the flexible, extendable world node would like to present).And to top that off, for internal modules in nodejs/node#18131 it was being discussed that that PR (internally) messes with the execution order by making sure that all the internal libs get executed before public packages so they can get named imports.
So while the answer is
yes, I definitely feel more like it's a qualifiedyes insofar as nothing surprising should happen, withsurprisingbeing defined a little bit differently for everyone involved...18131 is possible because we know internal modules won't error on evaluation, and that's why the other pr was turned down, it creates inherent instability in the module record if it fails like that.
To add to @weswigham excellent list of questions, and to refer specifically to this part of his last point:
Realistically, this means most code written in a node-first way will need preprocessing before being consumable by the browser.
- Can we (and should we?) approach browser vendors and/or TC39 with a proposal that will enable them to implement additions to browse ESM so that the browser can consume Node ES Modules without any preprocessing?
- Obviously, I believe we can. I believe Node and the browsers can meet halfway, and if we come with a concrete proposal that is easily implementable, this would be to the benefit of both sides. The npm proposal for ES Modules has a hint of how this can be done, but we need to fully flesh this out.
- Can we (and should we?) approach browser vendors and/or TC39 with a proposal that will enable them to implement additions to browse ESM so that the browser can consume Node ES Modules without any preprocessing?
Because paths/URLs differ across environments, the only possible way I can think of to make that work is if browsers are willing to add loader hooks that would allow you to configure them to load modules "as if" it were node - or, if node users were willing to use URLs exclusively, and use loaders in node as needed. The latter, imo, is unacceptable.
TC39 has no bearing on the conversation unless browsers are first convinced.
Reacted by Benjamin GruenbaumCan we (and should we?) approach browser vendors and/or TC39 with a proposal that will enable them to implement additions to browse ESM so that the browser can consume Node ES Modules without any preprocessing?
This is a very pessimistic view, but browser implementers have no concrete incentive to implement anything module-related to align with what node needs for an ergonomic es module implementation at present. There's no existing ecosystem for them to benefit from, and it will still cost implementers time and tech debt - essentially you're selling them on the future promise of the node es module ecosystem. They also can just choose to take a "wait and see" approach in that regard - they can see if node es modules don't flounder on delivery, then if they become wildly successful they can extend their minimal module support to support loading them. (They do have their own strange compatability issues to worry about) I'm not saying we shouldn't try, and I don't think we'd get a straight "no", but the incentives don't seem like they've been there in the past when es modules were being designed. Looking over notes from the past few years, almost every time someone has gone to the tc39 to ask for changes or permissiveness in the existing spec to standardize something node wanted as permissible, the answer has not been positive (save
import.meta). That's part of what's driving everything towards the current spartan state of affairs. My observation is, perhaps, a little more bleak than is warranted, as I've only been an observer of these processes.But I think, based on the last year of tc39 notes, what it looks like the committee needs to see is node doing what it must to make es modules optimally usable for its ecosystem, at which point it can come back and create or update the standard around what's been done. In fact, I can quote @wycats stating almost exactly that, from one of the links above from last year:
YK: approach, get as close to the spec as possible, and come back to the committee with concrete points that need to be fixed.
This has been a bit of a tangent. sorry. I've been watching this space for years, and the near zero apparent external progress here has just been flabbergasting given how much es modules have been adopted by transpiler-using-programmers since even before the es6 spec officially dropped - this feeling is probably what's helping drive this rush to implement something before node 10 drops.
Reacted by Benjamin GruenbaumAs an aside... There is no need to rush anything before Node 10 drops. We can remove the flag prior to LTS if it stabilizes. It would also be possible to drop the flag in a semver minor release during active LTS. So the runway is more like 9 - 27 months
Reacted by Jordan Harband, Andrea Giammarchi, Evan Plaice, Benjamin Gruenbaum, Steven and Geoffrey Boothsome answer for @weswigham hopefully interesting for others too.
the time should be taken to get it right
I couldn't agree more.
What, specifically, is going to make this feature more attractive than a transpilation step to commonjs, especially if running on multiple platforms is going to need a buildstep, anyway?
There is no need for a build step to run cross platform a self contained module.
Actually, there is no need for build steps for any ESM module (more later).
Build steps are rather a best practice for the Web, JavaScript is not a Web or Node only language.
- SpiderMonkey already has ESM (static, not dynamic)
- JSC already ships both
- GJS (GNOME JS bindings) will use SpiderMonkey mechanism
- Kinoma, Espruino and others might use ESM soon too
What browser esm do we expect to be executable in node (without processing by a tool)?
Not browser esm, any code that doesn't use NodeJS core modules could be shared.
What node esm would we expect to be executable in the browser (without processing by a tool)?
Not browser only, any code that does not use NodeJS core modules can land in npm and be used env-agnostic.
There's no conditional imports ... these are some big caveats!
Dynamic
import(stuff ? that : other)partially solve this but I agree few will struggle with this caveat.any browser code that uses fully qualified URLs will need preprocessing before being consumed by node
Two slightly off assumptions here:
- node will be incapable of resolving fully qualified URL in
import - fully qualified URLs are needed at all
CommonJS Example
Two modules, acjs and bcjs, both installed, hence inside node_modules folder.
acjs/index.js
const bcjs = require('../bcjs'); console.log(bcjs);
bcjs/index.js
module.exports = Math.random();
The execution of
node -e 'require("acjs")'will log a random numberESM Example
Two modules, aesm and besm, both installed, hence inside node_modules folder.
aesm/index.js
import random from '../besm/index.js'; // print via JSC or SpiderMonkey print(random);
besm/index.js
export default Math.random();
The execution of
js52orjsc -m node_modules/aesm/index.jswill log a random number.Does it work on the Web ?
TL;DR Yes, if you use the same CDN to publish your modules you can actually import in the same relative way.
Example, instead of
import ... from 'hyperhtml/esm'in this file which indeed requires bundling alchemies to work,
I could've written (and I will ASAP)import ... from '../../hyperhtml/esm/index.js'and that would've worked out of the box for development, where I would bundle anyway the module for production.Also notice-able that specific project with Custom Elements runs in NodeJS too thanks to basicHTML, yet I need to convert ESM to CJS because there's no ESM in NodeJS (like you said
.mjsis not a portable solution).As summary
It would be great if we could stop thinking binary (Node / Browser) when the official JS module system can, and will, affect many other situations but also we should start considering that, as shown in these examples, cross portability is already reality.
Please let's get this right, thanks for reading.
Reacted by Evan PlaiceFor bare imports
Can we (and should we?) approach browser vendors and/or TC39 with a proposal that will enable them to implement additions to browse ESM so that the browser can consume Node ES Modules without any preprocessing?
Browser vendors, no. TC39, yes. If it goes through TC39, the functionality eventually land in browsers. The downside being, TC39 is slow so addressing this sooner than later will be best.
Also keep in mind the potential for
npm assetfor front-end specific modules. Depending on how that pans out 'in the wild' it may not even be necessary to provide bare imports tonode_modules.Are we using import.meta?
If so, keep in mind that the spec is still only Stage 3. If we're going to use a non-finalized spec we should reach out to TC39 to identify any potential footguns ahead of time.
@evanplaice I think you misunderstand how TC39 works; browsers are members of it, and have actively blocked some of the changes that have been requested for purposes of the node integration. It's not that TC39 is "slow", it's that it has many stakeholders - appropriately many - and so it's often harder to ship bad decisions (but not impossible, obviously).
@ljharb Copy. Scratch that last comment.
Reacted by Jordan Harband45 remaining items
Looks like someone else just started a meeting on the same zoom account. Give me 10 minutes
Reacted by Ben Newman, Jan Olaf Martin, snek, Wesley Wigham, Gil Tayar, Michaël Zasso, Jeremiah Senkpiel, Jordan Harband, Refael Ackermann and Hassan SaniDue to technical difficulties we are moving to hangouts on air
participants please check out https://hangouts.google.com/hangouts/_/tmdgbvvtyraerlk4twtlqqejfqe
A new stream link will be available v soon
Ok finding another options :sigh:
Original Webinar is working again
I've pinged foundation to see if we can avoid getting booted
Reacted by Jan Olaf Martin and Refael AckermannCan other people get in? It just says "If you are the host, please sign in to start the webinar"
@justinfagnani I can switch times with you
Great work today everyone! All the videos are available individually on youtube now, shared in a larger play list
https://www.youtube.com/watch?v=IbPxOaGveXA&list=PLfMzBWSH11xb9LoHLO0q3Yo7HXjMOm11N
Reacted by snek and Charles SamborskiReacted by Hassan Sani, John-David Dalton and Jordan HarbandCan you please post links to the various presentations that were shown today? Not the video, but the Google Docs/Slides/etc. documents that people presented. Thanks!
I'll cross post over on youtube when people post stuff here.
@GeoffreyBooth if this gets lost in the noise I'll start asking folks for the links. Please feel free to ping again if this falls off my radar
Slides: https://goo.gl/WLufhg
Twitter: @jdalton
Repo: standard-things/esm
Reading:
Time
Decided via doodle
UTC Tue 03-Apr-2018 14:00 (02:00 PM):
Or in your local time:
Proposed schedule
https://docs.google.com/document/d/1oAvF_qukb1dIUWji9GjHNCL5-0uJ3_M-yPG-lzJVBbs/edit#
Joining the Call
Members of the team can join the zoom:
https://zoom.us/j/486767135
Streaming
Part 1: http://www.youtube.com/watch?v=O-7rXALxL9Y
Part 2: https://www.youtube.com/watch?v=Eaofz2Cj9Lw
So there is a LOT of context to the module implementation. I think it would be useful for us to schedule a day of content to document the current state of things and to get individuals who have not been as involved up to speed. I'm imaging we likely could co-ordinate a day of content around this. Please use reactions to signify interest and comment topics you would like to see or volunteer to provide. We can also invite guests from outside the group (such as TC39, webpack, or participants from Frameworks)
I'll keep this original issue updated with content as it develops.
Topics