Skip to content

Support a pthread and dynamic link build for wasm #96254

Closed as not planned
Closed as not planned

Description

@rchiodo

Feature or enhancement

The tools/wasm/wasm_build.py is missing a build feature that supports pthreading and dynamic linking for node.

Pitch

Add this build feature

Previous discussion

Activity

  1. tiran commented on Aug 25, 2022

    @tiran
    Member

    Thanks for your interest. 👍

    The build rule is omitted on purpose. Last time I checked builds with dynamic linking and pthread proxying were unstable and crashed. Did you check that the target is passing all tests?

  2. rchiodo commented on Aug 25, 2022

    @rchiodo
    Author

    There are some tests that fail (5 out of 435) but I thought this is all experimental anyway?

    Here's the failing tests:

    5 tests failed:
        test_capi test_code test_imp test_importlib test_xxlimited
    

    I'm using this build rule to try and get https://github.com/Microsoft/DEBUGPY working in some form of a WASM build. Debugpy requires dynamic linking and pthreads.

  3. rchiodo commented on Aug 25, 2022

    @rchiodo
    Author

    Would it be possible to add it and put a comment like this one?

    # wasm64-emscripten (currently not working)

  4. tiran commented on Aug 25, 2022

    @tiran
    Member

    That comment is outdated already :). wasm64-emscripten is working and all tests are passing -- or were passing Emscripten broke two days ago. Sam is working on a fix, emscripten-core/emscripten#17720

    test_capi test_code test_imp test_importlib test_xxlimited are literally all tests that load a shared extension. The other tests use statically linked extensions. I guess dlopen() is still broken and corrupts memory with threading.

    I would be willing to accept a emscripten-node-dl-pthreads-debug if you get it working, report problems to Emscripten project, and work with Emscripten upstream to get the bugs fixed. A non-debug build is ok once tests are passing and stable.

  5. tiran commented on Aug 29, 2022

    @tiran
    Member

    For the record, pthreads and dynamic linking are experimental and currently broken for CPython. Every attempt to load a shared extension module leads to an immediate crash:

    emcc: warning: -sMAIN_MODULE + pthreads is experimental [-Wexperimental]

    Error [RuntimeError]: memory access out of bounds
        at PyType_IsSubtype (<anonymous>:wasm-function[2366]:0x1fd5c4)
        at _PyImport_LoadDynamicModuleWithSpec (<anonymous>:wasm-function[3777]:0x2c0f80)
        at _imp_create_dynamic (<anonymous>:wasm-function[3770]:0x2c076e)
        at cfunction_vectorcall_FASTCALL (<anonymous>:wasm-function[1987]:0x1e9a8d)
        at _PyVectorcall_Call (<anonymous>:wasm-function[880]:0x19c433)
        at _PyObject_Call (<anonymous>:wasm-function[883]:0x19c754)
        at PyObject_Call (<anonymous>:wasm-function[884]:0x19c801)
        at _PyEval_EvalFrameDefault (<anonymous>:wasm-function[3269]:0x2855b1)
        at _PyEval_Vector (<anonymous>:wasm-function[3270]:0x28b334)
        at _PyFunction_Vectorcall (<anonymous>:wasm-function[887]:0x19c976)
    Emitted 'error' event on process instance at:
        at emitUnhandledRejectionOrErr (internal/event_target.js:579:11)
        at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:403:9)
        at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)
    
    RuntimeError: invalid index into function table
        at module_dealloc (<anonymous>:wasm-function[2027]:0x1eb420)
        at _Py_Dealloc (<anonymous>:wasm-function[2113]:0x1ef34b)
        at PyModule_FromDefAndSpec2 (<anonymous>:wasm-function[2015]:0x1eabfa)
        at _PyImport_LoadDynamicModuleWithSpec (<anonymous>:wasm-function[3777]:0x2c0fdf)
        at _imp_create_dynamic (<anonymous>:wasm-function[3770]:0x2c076e)
        at cfunction_vectorcall_FASTCALL (<anonymous>:wasm-function[1987]:0x1e9a8d)
        at _PyVectorcall_Call (<anonymous>:wasm-function[880]:0x19c433)
        at _PyObject_Call (<anonymous>:wasm-function[883]:0x19c754)
        at PyObject_Call (<anonymous>:wasm-function[884]:0x19c801)
        at _PyEval_EvalFrameDefault (<anonymous>:wasm-function[3269]:0x2855b1)
    Pthread 0x008c0948 sent an error! undefined:undefined: invalid index into function table
    
    /home/heimes/dev/python/cpython/builddir/emscripten-node-dl-pthreads-debug/python.js:156
       throw ex;
       ^
    Error [RuntimeError]: invalid index into function table
        at module_dealloc (<anonymous>:wasm-function[2027]:0x1eb420)
        at _Py_Dealloc (<anonymous>:wasm-function[2113]:0x1ef34b)
        at PyModule_FromDefAndSpec2 (<anonymous>:wasm-function[2015]:0x1eabfa)
        at _PyImport_LoadDynamicModuleWithSpec (<anonymous>:wasm-function[3777]:0x2c0fdf)
        at _imp_create_dynamic (<anonymous>:wasm-function[3770]:0x2c076e)
        at cfunction_vectorcall_FASTCALL (<anonymous>:wasm-function[1987]:0x1e9a8d)
        at _PyVectorcall_Call (<anonymous>:wasm-function[880]:0x19c433)
        at _PyObject_Call (<anonymous>:wasm-function[883]:0x19c754)
        at PyObject_Call (<anonymous>:wasm-function[884]:0x19c801)
        at _PyEval_EvalFrameDefault (<anonymous>:wasm-function[3269]:0x2855b1)
    Emitted 'error' event on process instance at:
        at emitUnhandledRejectionOrErr (internal/event_target.js:579:11)
        at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:403:9)
        at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)
    
  6. brettcannon commented on Jun 8, 2023

    @brettcannon
    Member

    Closing as a duplicate of #101538 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions