Rpath, or why lld doesn’t work on NixOS

`patchelf` solves a different problem: it fixes up an already built binary. Here, I am building the binary myself, so I’d rather make that just work without any extra build steps.

I'm pretty sure NixOS isn't the only one doing this hack. The Yocto build system does something similar. It contains its own build-time binary glibc, and patches its tools to point to its own internal library installation. Or something like that. In effect, Yocto has its own build-time distro, which has to run from any filesystem location.

I'm super interested in trying out mold, but I was a bit taken about seeing that it's AGPL licensed; I have no idea what the implications of that license for a linker would be. Would using it to link into a final binary require sharing the source of the binary when its distributed? What about if its used to link statically into a library, and then that library is linked into another binary?

Really looking forward to MacOS support on Mold!

As mentioned by another commenter, the NixOS project does have a similar little program called patchelf, which may be a bit better known?

RPATH and RUNPATH are indeed different. However, I believe the -rpath flag is used to set either (--enable/disable-new-dtags determines which one is used), so the confusion is understandable.

What is the distinction between them? I always thought they were the same thing.

That's not exactly right.

A library itself decides if it is relocatable or fixed. If it is fixed the MH_DYLIB records its install name as /path/to/binary (generally by setting DYLIB_INSTALL_NAME_BASE so xcodebuild will merge that with the library name automatically). The binary must be at that path. However this can (and often is) a symlink just like other systems use where /usr/lib/somelib.dylib -> /usr/lib/somelib.1.3.dylib so that minor version updates can be made without rebuilding programs.

If a library wants to be relocatable it specifies an install name of @rpath/binary.

At runtime dyld creates a "run path list". Every time it encounters a load command with an @rpath name it tries substituting paths from the run path list until it finds the library. The main binary along with any dependencies can add entries to the run path list. These can be absolute paths or relative paths anchored from @executable_path or @loader_path. The former being the main binary and the latter being the path to the binary itself (eg if the main app loads a plugin the plugin can reference dependencies relative to the main app or itself as needed).

You can push your own paths in the mix with DYLD_LIBRARY_PATH (searched first) or DYLD_FALLBACK_LIBRARY_PATH (searched last). Check "man dyld" and "man ld" if you want more details.

None of the above requires modifying binaries and so doesn't invalidate code signatures. If you want to use install_name_tool on binaries you build pass "-headerpad_max_install_names" to ld so it will pad out the load commands which makes it easier to edit them.

There have been a bunch of security vulnerabilities around the Windows strategy of auto-loading any dependency from the same directory as the binary so YMMV.

That's a pretty poor solution compared to just finding libraries relative to the root of the program's installation.

NixOS changes some standard conventions of Linux filesystem layout and where tools can expect to find things. These are for good reasons and are due to the core of what NixOS is trying to achieve. For building most things those changes are relatively abstracted away (see ld wrapper script for details) and you don't have to know about them. Fiddling around with something low level like a linker is I think a forgivable situation where the abstraction leaks - it is a process that is inextricably linked to where the system puts things.

My issue is the last sentence:

    So… .. turns out there’s more than one lld on NixOS. There’s pkgs.lld, the thing I have been using in the post. And then there’s pkgs.llvmPackages.bintools package, which also contains lld. And that version is actually wrapped into an rpath-setting shell script, the same way ld is.

This is only semi-related, but would you mind explaining what libtool actually does and what problem it solves? I've never been able to fully grok it.

> For all those who keep harping on how static linking is better than dynamic linking, what I would suggest is that what must be done to make static linking not suck is to enrich .a files with the kinds of metadata that ELF adds to shared objects so we can get the same "list only direct dependencies" semantics when static linking as when dynamic linking.

Or just use cmake which does that automatically

> Cause the situation on Linux is absurd! You pass some flags -L and -l to the compiler/linker, the linker links something and nobody knows what. Then when you run your executable it has to locate this something again, and you can only pray that your libc and binutils/llvm agree on search paths & order. In most cases this does not work, and you must manually pass -Wl,-rpath,/some/path to add a search path. Nobody guarantees that what the linker links is what the runtime linker uses.

The issue you're missing is that the build directory is usually not the location of the final binary objects. The actual absolute path of libfoo.so may well be in /builds/runner/foo-package-4df78af0/build/prefix/lib/libfoo.so, which is unlikely to exist on anyone other than the CI's machine (and even on the CI machine itself for too much longer). The actual location will usually be /usr/lib64/libfoo.so, but the library that is linked against may well not be there at the time of linking (particularly in the case where a package is building both a library and an executable that depends on said library in the same package).

What you really want is for the relative path to the library to stored in the executable. Unless what you want is to actually use the globally-installed library and not one you're building at the same time. There's no single solution that fits every use case!

> Cause the situation on Linux is absurd!

Your complaints are reasonable, but gcc already does this. Here, I'll show you how:

> You pass some flags -L and -l to the compiler/linker, the linker links something and nobody knows what.

I agree, it would be really nice to be able to specific exact shared object paths instead of using -L and -l. Build systems typically already know the full paths to all the objects and the abstraction here is often unhelpful.

This could be remedied fairly easily by allowing (for example) -l to take an absolute path to an object rather than searching -L paths. But gcc already does this - you can just put the shared object on the command line directly like so:

Change this: gcc -lfoo bar.c

Into this: gcc bar.c /path/to/foo.so

The effect is the same, but more explicit. The foo.so.x object will be linked and added to the SONAME list.

> Nobody guarantees that what the linker links is what the runtime linker uses.

This part, however, is by design. We explicitly do NOT want what the linker links to be what the runtime uses. This is how we update shared objects between minor versions to fix bugs without reinstalling every binary on the entire system!

Letting libfoo.so.1 link to libfoo.so.1.x is a huge feature. Locking in an explicit minor version would defeat the entire purpose of dynamic linking.

As a rule, NixOS doesn't patch binaries, it causes RPATH to get baked in at build time. Also, as a rule, it doesn't have some sort of materialized FHS-like subtree for each package, it manages a complex filesystem tree where the (bit-exact) stuff from the packages is stored immutably by the hash of the full build description.

Binary patching only comes in when they're trying to get closed source binaries to run on NixOS and is fully managed by the packaging process to happen the same way on every system. These days, though, the approach of using filesystem namespacing to give packages a custom FHS-like view of the world seems to be growing more common instead of the patching.

IMO RPATH is fine, but $ORIGIN-relative RPATH values are best.

That said, it's generally very difficult to build deploy-time relocatable code in Unix-land. The problem is that there's nothing like $ORIGIN for finding static assets, and all the autoconf tooling just makes it so easy to make all paths in object code by absolute paths that include the install $prefix/$bindir/$libdir/$sharedir/$statedir/$etcdir, etc.

Not that one cannot write deploy-time relocatable code -- I've done it plenty. But that it requires so much foreknowledge, intent, and know-how, that it just doesn't get done.

> I suspect that NixOS is playing with this in order to have a relocatable install

Kinda the opposite: NixOS doesn’t really have a sysroot.

nixOS is built around hashing the outputs of its builds, so you can verify if a build produces the expected output given the same inputs. The reason nixOS patches binaries is so that they can actually find the shared objects they expect when they are not stored in /usr/lib. Since every build output gets its own unique path, this allows for a binary to link against two slightly different versions of the same library, which in practice is never an issue one needs to resolve. However, a more practical issue this solves is that you can have a single sysroot with two binaries that need two slightly different versions of the same library.

> A possibly nice hack would be to extend the meaning of the rpath variable. Give it a syntax, like say that if it starts with @, then the rest of it denotes a relative sysroot path fragment.

This sounds vaguely similar to dyld's @executable_path variable on macOS.

This is the only sane way...

And for people saying "Buy you can't get security updates".

I would rather have a dynamically-linked binary that includes all the dependences in it, at which you can upgrade the dependencies with a tool over the binary, than the madness of shared libraries in system paths. (Well, you kinda get that with appimage and similar).