Repository navigation
emscripten cross-compile wasm-ld: error: duplicate symbol _Py_LibHacl_Hacl_Hash_* #133042
Description
Activity
Mmh, EMScripten with shared dependencies is experimental so I should actually force static linking in this case. You will likely encounter the same issues with the other HACL* primitives as they would also have duplicated symbols. As you can see on your PR, the change is breaking for the regular builds.
I'll see what I can do (probably a similar issue as we had with WASI).
Thx for the quick reply! Yes, just removing the objects is just breaking the regular builds. However, this is the only part that fails my shared build.
One part of the configure script also caught my eye:
Lines 7960 to 7966 in 614d792
WASI requires static linking but I'm unsure whether emscripten can have the corresponding built-in modules. If needed, I can just.. disable the HACL* primitives. I can't test locally because I have a "python.mjs" not found and I don't know how to fix it.
Question: would forcing
LIBHACL_LDEPS_LIBTYPE=STATICbe sufficient on your side to fix the EMscripten build?OK I think I know the issue. At the final makefile step, we have something like that:
-lm -lm -lm -lm -lm Modules/_decimal/libmpdec/libmpdec.a -sUSE_ZLIB -sUSE_BZIP2 -sUSE_ZLIB Modules/_hacl/Hacl_Hash_MD5.o Modules/_hacl/Hacl_Hash_SHA1.o Modules/_hacl/Hacl_Hash_SHA2.o Modules/_hacl/Hacl_Hash_SHA3.o Modules/_hacl/Hacl_Hash_Blake2s.o Modules/_hacl/Hacl_Hash_Blake2b.o Modules/_hacl/Lib_Memzero0.o Modules/_hacl/Hacl_HMAC.o Modules/_hacl/Hacl_Streaming_HMAC.o Modules/_hacl/Hacl_Hash_MD5.o Modules/_hacl/Hacl_Hash_SHA1.o Modules/_hacl/Hacl_Hash_SHA2.o Modules/_hacl/Hacl_Hash_SHA3.o Modules/_hacl/Hacl_Hash_Blake2s.o Modules/_hacl/Hacl_Hash_Blake2b.o Modules/_hacl/Lib_Memzero0.oThe first occurrences are of Hacl* files are coming from each individual modules, but the last ones are indeed coming from HMAC. It's not an issue for regular compilers but for emscripten it appears that it's an issue (though I don't know why??) I'll make it so that the configure script can figure out the duplicates.
Ok it's much more complex than I thought. Question, can you check if the following would solve your issue:
-if test "$ac_sys_system" = "WASI"; then +if test "$ac_sys_system" = "WASI" -o "$ac_sys_system" = "Emscripten"; then
I don't have time now to fix the build entirely (sorry) but I'll have a look again tomorrow. If this still fails, we should make it so that only unique
.oare retained when linking the Python executable.I'm marking it as a release blocker because I'm leaving tomorrow and won't be back before the next beta release so we need to find a solution in the next 24 hours (otherwise it's annoying for those working with Emscripten). It's not a true release blocker though but it's to keep in mind that this must be addressed before releasing the next sources.
Reacted by Peter Bierma10 remaining items
- added a commit that references this issue
on May 4, 2025 Ok, so here's the story:
- Let's say I build md5 and HMAC. Everything works fine until building the interpreter actually. IOW, I can build both modules and their corresponding shared extensions without issues.
- During the linking phase for the interpreter, we're passing
MODLIBSto it. It contains all*_LDFLAGSvariables. Now, in my case, I have built MD5 and HMAC. Since HMAC depends on MD5, the_LDFLAGSvariable for HMAC contains the MD5 objects, that isModules/_hacl/Hacl_Hash_MD5.o. SoMODULE__MD5_LDFLAGSisModules/_hacl/Hacl_Hash_MD5.owhileMODULE__HMAC_LDFLAGScontainsModules/_hacl/Hacl_Hash_MD5.o. - In
make python.mjs, we will have a duplicatedModules/_hacl/Hacl_Hash_MD5.oentry, and emcc complains at this point.
So how can I fix this? well, the dirty way to do it is to do everything normally, and only for
python.mjsshould we re-computeMODLIBScorrectly, eliminating the duplicated.o. While I can do it using GNU Make, it's quite hard to do it for a portable Makefile.An alternative I had in mind is to install, alongside libpython.so, libpythonhacl.so as a dynamic library. All other vendored libraries are actually statically linked, but we wanted to get rid of the static archives of libhacl as they were annoying for
freeze(why does freeze work for the rest is still a mystery to me).So I need some help from someone who would know the build systems even more. We can keep the status quo of not having a HACL* HMAC for Emscripten (we don't have OpenSSL and so Emscripten would rely on HACL* MD5 & co + Python implementation of HMAC, which is fine IMO). OTOH, I would really be interested in being able to make it available on Emscripten without changing the entire build configuration...
Is this broken also natively if we statically link everything
I wasn't able to make it work properly for that. The issue seems to be the fact that I'm specifying objects twice, whether it's static or dynamic.
Why is emscripten specifically affected?
In Emscripten, apparently, the compiler complains if the linker flags contain duplicated
.ofiles, but gcc doesn't seem to complain. I don't know why WASM isn't affected, but maybe it's because it's statically linked. I'll try again.Btw, this is orthogonal, but I also need to hack my own wasm.py script because
python3isn't picking up the correct interpreter, even if I doPYTHON=python3.12 python3.12 Tools/.... build. Inmake_emscripten_python, I needed to replace:call( ["make", "--jobs", str(cpu_count()), "all"], env=updated_env(), quiet=context.quiet, )
by
call( ["make", "PYTHON=python3.12", "--jobs", str(cpu_count()), "all"], env=updated_env(), quiet=context.quiet, )
otherwise I can't compile in the end. I don't know why this is the case but I'll open a separate issue later.
Ok, so I confirm that static linking still fails (sorry, I forgot to regen configure). Now the issue is as follows (for instance):
wasm-ld: error: duplicate symbol: _Py_LibHacl_Hacl_Hash_SHA1_update_multi >>> defined in Modules/_hacl/libHacl_Hash_SHA1.a(Hacl_Hash_SHA1.o) >>> defined in Modules/_hacl/libHacl_HMAC.a(Hacl_Hash_SHA1.o)To be precise: it's because libHacl_HMAC contains all its dependencies as well. So my alternative is the following: instead of passing the entire
.awhen creatingpython.mjs, what we can do is instead to build a static object that only contains the HMAC.a definitions, even if incomplete. However, this requires to changemakesetupand other magical stuff I think and I really don't want to go down that road :cIs there some magical flag for the compiler that I can pass in order to suppress those duplicate entry points if they are actually doing the same?
I don't know of any. In pyodide-build we use a compiler wrapper that drops duplicated libs:
# WASM link doesn't like libraries being included twice # skip second one if arg in used_libs: return None used_libs.add(arg) return arg
https://github.com/pyodide/pyodide-build/blob/main/pyodide_build/pywasmcross.py#L130-L133
Of course that wouldn't fix this particular problem because you have to compare object file hashes.cc @sbc100
Opened upstream issue:
emscripten-core/emscripten#24375I'm having trouble finding a minimal reproducer for this. I tried to make two libs with a duplicated object file:
x.c,y.c,z.cint x(void) { return 3; }
and similar
main.cint x(); int y(); int z(); int main() { x(); y(); z(); return 0; }
Compile and link
emcc -c x.c y.c z.c main.c ar -mc liby.a x.o y.o ar -mc libz.a x.o z.o emcc main.o -ly -lz -L.
It links fine...
I will be away for a few days until Friday, so I'll try to make it work on my side. I'll give more results when I'm back and hopefully I'll see what we can do.
No rush.
The third beta of 3.14 is tomorrow.
I'm not sure this is a deferred blocker -- it might be blocking Emscripten from becoming a Tier 3 platform, but, even T3 platforms don't necessarily block releases. Also, AFAICS, there's a workaround in place.
@hugovk, will you delay 3.14.0 for this issue?No, I don't think this needs to delay 3.14.0 or be a deferred-blocker (but please let me know if you think it should). As a bug it can be fixed as normal.
Bug report
Bug description:
For the emscripten build, during building of the python executable, all the
LIBHACL_*_LIB_SHAREDare used as linker flags, however theLIBHACL_HMAC_LIB_SHAREDflag contains objects that are already contained in otherLIBHACL_*_LIB_SHAREDflags. This happens, because LIBHACL_MD5_OBJS, LIBHACL_SHA1_OBJS, LIBHACL_SHA2_OBJS, LIBHACL_SHA3_OBJS and LIBHACL_BLAKE2_OBJS are all included inLIBHACL_HMAC_LIB_SHAREDthroughLIBHACL_HMAC_OBJSvia the Makefile.pre.in file.The configure script then sets
LIBHACL_HMAC_LDFLAGS=LIBHACL_HMAC_LIB_${LIBHACL_LDEPS_LIBTYPE}which for a shared emscripten build leads to duplicate dependencies in the final build command which leads to duplicate symbols.Since I don't know the purpose of the object file references in the
LIBHACL_HMAC_OBJS, I do not feel comfortable just removing them, even though that change fixes the emscripten build.CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
LIBHACL_HMAC_*#133043