Repository navigation
Description
Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11615
The Arrow crate successfully compiles to WebAssembly (e.g. https://github.com/domoritz/arrow-wasm) but the DataFusion crate currently does not support thewasm32-unknown-unknown target.
Try out the repository at https://github.com/domoritz/datafusion-wasm/tree/73105fd1b2e3ca6c32ec4652c271fb741bda419a.
{code}
error[E0433]: failed to resolve: could not find unix in os
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/dirs-1.0.5/src/lin.rs:41:18
|
41 | use std::os::unix::ffi::OsStringExt;
| ^^^^ could not find unix in os
error[E0432]: unresolved import unix
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/dirs-1.0.5/src/lin.rs:6:5
|
6 | use unix;
| ^^^^ no unix in the root
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:98:9
|
98 | sys::duplicate(self)
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:101:9
|
101 | sys::allocated_size(self)
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:104:9
|
104 | sys::allocate(self, len)
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:107:9
|
107 | sys::lock_shared(self)
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:110:9
|
110 | sys::lock_exclusive(self)
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:113:9
|
113 | sys::try_lock_shared(self)
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:116:9
|
116 | sys::try_lock_exclusive(self)
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:119:9
|
119 | sys::unlock(self)
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:126:5
|
126 | sys::lock_error()
| ^^^ use of undeclared crate or module sys
error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:169:5
|
169 | sys::statvfs(path.as_ref())
| ^^^ use of undeclared crate or module sys
Compiling num-rational v0.3.2
error: aborting due to 10 previous errors
{code}
Activity
alippai commentedon Apr 26, 2021
Polars proof of concept (shows that arrow-rs and datafusion like API can work): https://github.com/ritchie46/polars/blob/master/js-polars/app.js
alippai commentedon Apr 29, 2021
#218 is a great step helping this
seddonm1 commentedon Aug 25, 2021
@alamb @Dandandan @jorgecarleitao
I have done some digging into this. There are two blockers:
dirs-rs
problem
dirs-rsis pulled in due toprettyprint-rpulled in due to pretty printing dataframes https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/dataframe_impl.rs#L34.dirsdoes not have awasm32-unknownimplementation and the owner seems reluctant to add it: wasm-pack support missing. dirs-dev/dirs-rs#17.solution The arrow crate has a feature flag which enables pretty printing. If we agree we can add this to the datafusion (default enabled) to prevent this first problem.
lz4
problem the
parquetcrate depends onlz4which is generated bylz4-sys(not native rust) and does not compile towasm32-unknown-unknown. I have done a test of ripping parquet out of datafusion and everything compiles.solution we could either put anything
parquetbehind another feature flag or swap out thelz4for a pure rust implementation like the redox: https://lib.rs/crates/lz4-compress.It would be relatively easy for me to do the two feature flags unless someone has an objection?
alamb commentedon Aug 25, 2021
Note that apache/arrow-rs#656 from @PsiACE has removed the
pretty-tabledependency in arrow-rs upstream. This will be included in the 6.0 arrow release (in 2ish months); I am not sure if/how this affects your decisionI think lz4 is an optional dependency of parquet: https://github.com/apache/arrow-rs/blob/master/parquet/Cargo.toml#L40 thus perhaps we could just have a lz4 feature flag for datafusion?
seddonm1 commentedon Aug 25, 2021
Thanks @alamb .
I will do some experiments but seems like a good solution.
ivanceras commentedon Nov 8, 2021
I think the biggest hurdles is using
tokiospecific types in the underlying implementation of datafusion, as I've found in this codeI understand this is for performance reason, but also there is no easy way to abstract this instead of using executor specific types. In wasm, we can use
wasm-bindgen-futuresas a way to run async functions.alamb commentedon Nov 8, 2021
@ivanceras I am not sure if we are using tokio specific
futuresstuff due to performance or just convenienceIt might be worth trying to replace tokio specific structs with things from the
futurescrate (if that works onwasm32) and see what it looks likeseddonm1 commentedon Nov 8, 2021
@ivanceras what are you experiencing? I have managed to compile to WASM with very slight code modifications.
roee88 commentedon Nov 9, 2021
@seddonm1 compile and run?
I have experimented with that yesterday. I tried wasm32-wasi first and a simple sample works in single threaded mode after disabling some parquet features. See this gist for the example: https://gist.github.com/roee88/91f2b67c3e180fa0dfb688ba8d923dae
For wasm32-unknown-unknown adding getrandom with js as a dependency of the sample makes it compile IIRC, but actually running it is a different story. I tried to get a sample working with wasm-pack and it stops execution on the datafusion context creation, I suspect that it uses some sync primitives that are unsupported in wasm32-unknown-unknown but I didn't investigate further.
I didn't try wasm32-unknown-emscripten yet since my local rust version is incompatible with my installed emcc version (both latest at the time of this writing).
Edit: re tokio, the sample above worked on wasm32-wasi with other executors in single threaded mode including futures 0.3, https://github.com/richardanaya/executor, and async-global-executor.
As long as you don't hit code paths that use things like tokio::spawn (used in hash aggregate) it might be fine to use another executor. I'm not sure what's the best approach for library code that needs to spawn tasks. I have seen opinions for 1) a library should never spawn, 2) futures should be universally supported, 3) a library should accept an executor trait (as implemented in https://github.com/najamelan/async_executors). I didn't check the state of futures and WebAssembly recently.
I didn't try wasmbindgen-futures because it's officially no longer compatible with wasi and emscripten and as I said I couldn't get anything running with wasm32-unknown-unknown.
roee88 commentedon Nov 9, 2021
I got the basic sample from the gist in the previous message working with wasm-pack (wasm32-unknown-unknown) on single threaded tokio after:
tokio-stream = { version = "0.1", default-features = false }Instant::now()which is unsupported in wasm32-unknown-unknown and panics. For the basic sample that I'm running I just removed some lines from ScopedTimerGuard, but it's used in other places too if you run more complex stuff. Maybe https://github.com/sebcrozet/instant can be used instead.jsfeature of getrandom:getrandom = { version = "0.2", features = ["js"] }. I did this in the sample application itself and not in datafusion (as it breaks other targets like wasm32-wasi).wasmbindfeature of chrono:chrono = {version = "0.4", features = ["wasmbind"]}. I did this in the sample application itself and not in datafusion (as it breaks other targets like wasm32-wasi).Tested in chrome and seems to work. Again, there are definitely some code paths that lead to panic as not all of std is supported in wasm32 targets and I only tested something basic. Also, for multi-threading to work in the browser some parts of tokio can't be used directly from the datafusion codebase (this is a more complex topic).
milesrichardson commentedon Nov 1, 2022
Good news, fellow WebAssembly enthusiasts! It looks like the stars are finally aligning, and with relatively minimal patching, I successfully compiled the code from the gist (create, insert and query a
MemTable) towasm32-wasiandwasm32-unknown-unknown, and ran it inwasmedgeand the browser (viawasmpack):I pushed the proof-of-concept to a public repository at
splitgraph/experimental-datafusion-webassembly. There are two branches:wasm32-wasiwasm32-unknown-unknownwasm32-wasiand the diff ofwasm32-wasi..wasm32-unknown-unknownshows the changesIn the near future, I intend to cleanup these changes and submit a PR to DataFusion feature-flagging WebAssembly support.
In general, the summary of requirements for
wasm-wasi:arrowto use its upstream Git repository fixed some initial compilation issues, but otherwise no changes were required. It seems like the latest release onmasterofarrow-rs(v26.0.0) can compile to both webassembly targets, and so DataFusion just needs to upgrade to that. (There was also one minor change todatafusion/physical-exprrequired to support the upgraded Arrow package).RUSTFLAGS="--cfg tokio_unstable"is necessary, and will benefit from recently stabilized wasm support in Tokioobject_store, as they are not compatible with thewasm32-unknown-unknowntarget (they may be withwasm32-wasiin some runtimes, but I disabled them).bzip2reqwestto add some wasm compatibility (note: I'm not sure how much of this was/is necessary, and/or if it forces tokio to resolve to a version that other packages incidentally benefit from)spawn_blockinginsortwithspawn, making the compiler happy but possibly causing runtime/logic errorsfor
wasm32-unknown-unknown, in addition to all those requirements, it was also necessary to:std::timewithInstant, in bothdatafusionandarrowgetrandomis also passing it thejsfeature flag, which I did by just patchinggetrandomand making that the defaultTo get it to run (without a runtime error related to
std::timebeing unreachable), a few more changes were made:mainruntime, even withflavor = current-thread. Instead, use wasm-bindgen-futures to await a future that performs the asynchronous task that calls datafusionThis is all very messy. I will clean it up and submit a PR to DataFusion once I have a better sense of the most minimal changes required and the proper way to feature flag them. Also, general disclaimer that I'm new to Rust and YMMV, especially on the
wasm-unknown-unknownpatch - after all, I barely got it to run. But it does compile and create and query a small in-memory table, which is pretty good!alamb commentedon Nov 1, 2022
This sounds very cool @milesrichardson - DataFusion should be upgraded to arrow 26.0.0 shortly: #4039. I think @jimexist is in the process of making bzip support optional #3993
In terms of being messy / submitting a PR -- if it is possible I suggest trying to do it incrementally -- like for example we can probably sort out the calls to
spawn_blockingin a separate PRBut all in all this is pretty exciting
REASY commentedon Mar 15, 2023
Hello, folks.
I'm trying to add WASM support to DataFusion's dependencies. Started with bzip2-rs trifectatechfoundation/bzip2-rs#93
REASY commentedon Apr 19, 2023
Posting an update on trifectatechfoundation/bzip2-rs#93, had a discussion with @alexcrichton
Not sure how to go from here...
milesrichardson commentedon Apr 19, 2023
@REASY In my experiment (the one linked above), I put bzip behind a configuration flag and disabled it for the wasm targets. Datafusion still compiled. I don't know enough about DF to say how important bzip is, or which parts of DF would be broken without it, however. It seemed limited in scope, since it should only affect files that are encoded with bzip.