Repository navigation
Internally use objcopy --add-section on macOS and Linux #63
Description
objcopy --add-section works on Linux - https://man7.org/linux/man-pages/man1/objcopy.1.html.
After https://reviews.llvm.org/D66283 landed, llvm-objcopy on macOS too started supporting this feature: https://llvm.org/docs/CommandGuide/llvm-objcopy.html#cmdoption-llvm-objcopy-add-section
This demo is using the Node.js binary that has been built from nodejs/node#45038 on macOS:
$ ls
index.js node
$ cat index.js
console.log('Hello, world!');
$ /usr/local/opt/llvm/bin/llvm-objcopy --add-section __POSTJECT,__NODE_JS_CODE=index.js node sea
$ ./sea
Hello, world!Advantages:
- We can avoid maintaining any complex code that adds a new section into a binary and reuse existing tools.
- Much faster than the current state of postject.
$ /usr/bin/time /usr/local/opt/llvm/bin/llvm-objcopy --add-section __POSTJECT,__NODE_JS_CODE=index.js node sea 0.37 real 0.22 user 0.11 sys $ /usr/bin/time postject sea NODE_JS_CODE index.js 37.59 real 40.05 user 1.12 sys - Possibly works on AIX also? I would need access to such a system to make sure.
Disadvantage:
objcopyis present by default on Linux but users would have to download it manually on macOS along with other tools usingbrew install llvm.
@nodejs/single-executable wdyt?
@bnoordhuis I'd also like to hear your thoughts on this since you were sharing some insights on the resource injection part in nodejs/node#45066.
Activity
jviotti commentedon Oct 31, 2022
Makes total sense to me!
tony-go commentedon Oct 31, 2022
+1 👏
RaisinTen commentedon Oct 31, 2022
Another problem I can think of is that, it won't let you embed your source code into a an executable format that's not used on that platform, like you won't be able to embed data into a Mach-O file on Windows. Pkg is currently able to produce ELF, Mach-O as well as PE executables regardless of the host platform. @jesec, is that a hard requirement of pkg?
jviotti commentedon Oct 31, 2022
That is a good point indeed, and worth checking what Node.js folks think too.
ovflowd commentedon Oct 31, 2022
That feels like would become a dependency of
nodeas ifseais part of node-core, all dependencies/pre-reqs need to be installed with node or bundled with node, no?And feels here like
llvmwould be an external runtime dependency, right?RaisinTen commentedon Nov 2, 2022
I didn't mention it during the meeting but it seems like the cross-platform binary creation might actually not even work if your project has native bindings, so maybe this still makes sense?
To @ovflowd's point, maybe we can find a way to ship just
llvm-objcopyas a standalone executable?feat: use objcopy instead of LIEF
RaisinTen commentedon Nov 3, 2022
PR - #64
dsanders11 commentedon Nov 9, 2022
I'm a fairly strong -1 on this idea for several reasons.
This is going to be a somewhat lengthy comment, so before getting further into it, I want to point out that
objcopyandllvm-objcopyare not the same. This issue and PR #64 use them interchangeably which is ripe for confusion. Whilellvm-objcopymay try to support the same options (and more) asobjcopyso that it's a drop-in replacement in most cases, the implementation can and is quite different, like the support for Mach-O files. In particular the implementation of--add-sectionis different for ELF (I'm pretty sure) than inobjcopyand the documentation for the option is different as a result, withllvm-objcopyproviding more details on the implementation (regarding note sections). In what follows I'm explicit about which one I'm discussing and use the name accordingly.That said, first and foremost, I don't think it works.
When exploring ways to accomplish the injection (before the Postject project existed) I evaluated
objcopyand discarded it as not a solution to the problem. With ELF executables if you useobjcopy --add-section, you're adding a section, not a segment - we went through this once already with the initial implementation for ELF. In ELF, sections are for the linker, but serve no purpose after that point. You can strip the section header table (SHT) from the binary and it will continue to work just fine. When usingobjcopy --add-sectionfor ELF you'd also need to use--set-section-flagsto setallocandreadonly. If you do that you'll see the warningwarning: allocated section '.foo' not in segment- becauseobjcopyisn't adding a segment, it's adding a section. Without the section being in a segment, it won't be paged into memory and can't be accessed at run-time like we need for Postject. Here's a very brief forum post explaining that issue and a more in-depth mailing list post. In ELF binaries sections with the same flags are grouped together into segments with those same properties (such as read-only) at linking time, and then the SHT keeps track of where the sections ended up (virtual address) but that information isn't needed at run-time for normal execution, which is why the SHT can be stripped and execution is normal. I don't really know what the intended use-case ofobjcopy --add-sectionis, but it doesn't fit our needs since we need the section to be in a loadable segment which can be accessed at run-time. cc @robertgzr as we've previously discussed the differences between sections and segments in ELF and how sections are not a run-time concept.I have not explored
llvm-objcopy --add-sectionto see if the above is still true for their implementation on ELF. It may be, it may not be. There appear to be other problems withllvm-objcopy --add-sectionthough, as pointed out on #64. For Mach-O it is adding the segment afterLC_CODE_SIGNATURE, which causes problems. It doesn't appear to support--set-section-flagsfor Mach-O at all (we need that to make the segment read-only). Finally, when testing it with Electron, it appears to be significantly altering the executable - just runningllvm-objcopy --add-sectionchanges the size of the executable from 315 MB to 267 MB and understandably it no longer runs.Beyond not being suitable for the purpose (IMO), I'd like to address the advantages/disadvantages discussed here.
Is this referring to the code using LIEF in Postject, or upstream LIEF? I'd argue that relying on
llvm-objcopyadds more maintenance complexity, not less. It's still under development, and we can and will find bugs (see previous point about it drastically changing the Electron executable), which we'll have to fix upstream, and LLVM is a more complex codebase than LLVM (IMO, I've worked in both). We also have no control over when those fixes actually land if we're having users bring their own copy ofllvm-objcopy, throughbrewfor example. Finally, if it's bring your own copy ofllvm-objcopy, then we significantly increase our maintenance headache since we don't control it and we'd be stuck with the problem of versions for that tool, we'd have to determine which ones are supported, test new ones when they are released, etc. With LIEF we're freezing to a known good version and that will never change under our feet.It's also just weird for
postjectas a CLI tool to then rely on the user to separately installllvm-objcopy.I don't think bundling
llvm-objcopyis feasible either. The whole reason we changed to WASM was to avoid native dependencies, so bundling it would just bring that back. Even if we don't bundle it and rely on the user to bring their own copy, we're still back into needing native dependencies, which is not where we want to be.Now, it is potentially interesting to consider building
llvm-objcopyto WASM and consuming it programmatically that way, like we currently do with LIEF, which would eliminate the native dependency issue and the users bringing their own copy issue since we'd pin it and consume a known good version. That said, there's still the "it doesn't actually work" issue detailed above. Also the downside that while LIEF is a solution for PE executables as well,llvm-objcopyis not, so we'd end up increasing the size of Postject even more. There could be an LLVM solution for PE executables, but research would need to be done.There are improvements that can be made here at the WASM level, that I haven't opened a PR for yet. That said, I don't think the speed of the injection is a major concern, and shouldn't be a major motivator for a significant implementation change like this.
See opening points, but for consistency we do not want to mix
objcopyandllvm-objcopy, that would be confusing and error prone. We'd want to usellvm-objcopyacross the board, which means you wouldn't be using the systemobjcopyon Linux machines.I think this is a major drawback. Being platform-agnostic for the injection is a major benefit and makes Postject easy to get started with immediately for testing. I'd want strong reasons to abandon this benefit, especially after we did the work to use WASM and make Postject completely platform-agnostic (heck, you could provide a website that does the injection in-browser and not even require the user to install anything).
That sounds like a bug - cross-compiling is the standard solution there and is supported by tools like
node-gyp.I'm in no way opposed to replacing LIEF or the underlying implementation - that's one of the main benefits to building to WASM and being entirely self-contained. But I don't think this is the right direction and I don't really see what is gained.
RaisinTen commentedon Nov 9, 2022
I was referring to maintaining a pure JS implementation of a resource injector for all binary formats.
nodejs/node-gyp#829 tells me that it is not supported. 🤔
Anyways, excellent points, thanks! I'll close this and instead work on the pure JS resource injector.
dsanders11 commentedon Nov 9, 2022
Good catch, I was thinking cross-compiling, not cross-platform. That indeed might not be possible.