Loading Relocatable Objects at Runtime - Very Expensive

Rod Evans — Wednesday February 02, 2005

Surfing with the Linker-Aliens

The runtime linker, ld.so.1(1), is capable of loading relocatable objects. This capability arrived somewhat by accident, and as a side effect of some other projects related to the link-editor, ld(1). But, it was thought that this capability could prove useful within a development environment. A user can preload relocatable objects, and this technique might provide a quick prototyping turnaround instead of having to first combine the relocatable objects into some huge shared object.

However, this capability doesn't come cheap, and it was never thought to be useful for production software. Recently, a customer problem uncovered a regression in relocatable object processing. But, it was a surprise to see that this technique, in this case the dlopen(3c) of a relocatable object, was being used in production code.

The runtime linker only knows how to deal with dynamic objects, that is, dynamic executables and shared objects. When a relocatable object is encountered, the runtime linker first loads the link-editor. The link-editor then converts the relocatable object into a shared object memory image within the process. The runtime linker then processes this memory image as it would any other shared object.

The link-editor is really a family of libraries, and you can see these if you inspect the link-maps of a process that has triggered the loading of a relocatable object.

   % mdb main
   > ::bp exit
   > :r
   > ::Rt_maps
   Link-map lists (dynlm_list): 0x8045f38
     Lm_list: 0xfeffa204  (LM_ID_BASE)
       Link_map\*  ADDR()     NAME()
       0xfeffc8f4 0x08050000 main
       0xfeffccc4 0xfeef0000 /lib/libc.so.1
     Lm_list: 0xfeffa1c8  (LM_ID_LDSO)
       0xfeffc590 0xfefc8000 /lib/ld.so.1
       0xfeee06e4 0xfee60000 /lib/libld.so.2
       0xfeee0af0 0xfed90000 /lib/libc.so.1
       0xfed80068 0xfed50000 /lib/libelf.so.1

As you can see, the runtime linkers link-map list (LM_ID_LDSO) contains the runtime linker itself, and all the libraries that effectively make up the link-editor.

Besides loading all these support libraries, the shared object image adds additional overhead. Relocatable objects are typically non-pic, and thus require considerable more relocation than their pic (position independent) counterparts. This relocation overhead is passed on to the shared object memory image.

And, because the image is created in memory, each process that uses this technique creates it's own private image. There is no sharing of text pages from the shared object image, as is common for shared objects. This customers application forked off a number of processes that all dlopen()'ed the same relocatable object.

So, this technique may have some use within a development environment, but there are expenses:

Personally, I'd recommend not using this technique in production software.

Surfing with the Linker-Aliens


Robert Charon — Wednesday February 02, 2005
Couple of questions:
</br> 1. What is a relocatable object? Specifically, is a relocatable object different from a shared object that is explicitly loaded using dlopen()?
</br> 2. Why is sharing of text pages from the shared object image not allowed? I understand the part about the image created in memory; however, it's my understanding that the Windows loading of ActiveX components which are explicitly loaded DLL objects is highly efficient and that its text pages are shared among processes. What is it that Solaris is doing differently from Windows in this regard?
</br> 3. If a shared object image is meant to be used for a single process, other than the loading and relocation expense per relocation object, what are the other expenses? In using this technique with multiple processes, shouldn't the runtime linker be loaded only once and its text pages shared across multiple processes since the runtime linker is a shared object? Thanks.
Rod Evans — Wednesday February 02, 2005

A relocatable object is output from the compilers. The link-editor typically combines a number of relocatable objects so as to create a dynamic executable or shared object.

Dynamic executables and shared objects are mapped directly from the file system, thus their read-only text segments can be shared between multiple processes. This shareablity is common to many OS's.

The subject of this posting is the translation of an input relocatable object into a shared object memory image. As the resultant shared object image is wriiten into memory and is not backed by any file, the image is private to the process that created it.

Robert Charon — Wednesday February 02, 2005
Thanks much, Rod.
</br> The mention of a relocatable object in the context of dlopen() threw me off. I've wondered why relocatable files were mentioned in the specification for dlopen() but always attributed this to an unintentional specification error where they meant shared objects since I couldn't see any good reason for it. Learn something new every day.
</br> Thanks again.
Vidya Shankar M. — Wednesday May 11, 2005
this is really very very informative, Thanks, I am facing the similar issue, the system depends on LD_LIBRARY_PATH and the shared objects have lot of relocations. We were thinking of using crle to create directory cache to avoid LD_LIBARARY_PATH issue and alternative objects to reduce relocations. Our tests showed improvement in startup times. However we find very limited documentation on crle with respect to alternative objects, (one in man page, other in linker and libraries guide) Any suggestions on using alternative objects. Thanks
eugenet — Saturday June 18, 2005

To answer your 2nd question (6 months later :), all Windows DLLs (including ActiveX objects) are compiled as non-pic code. DLLs are supposed to be loaded at a base adddress which was specified at their compile time.

If a DLL cannot be loaded at its preferred base address (i.e. if another DLL is already loaded at that address), Windows loader will load the DLL at a different address and will then repatch the text segment as per the new base address (addresses that need to be repatched are specified in a DLL's relocation segment).

This is obviously not very efficient and Rod evaded to this in one of his posts.


P.S. Windows executables do not contain relocation information (to save space) and thus cannot be relocated at load time.
Joey — Tuesday June 26, 2007
Nice explanation, can you add an entry in your blog about possible ways to do shared library interposition, especially methods that do not rely on LD_\* variables.
Rod — Wednesday June 27, 2007

Interposition occurs when the symbol in one library is found before a symbol of the same name in another library, during the runtime linkers symbol search.

You can force interposition with the initial dependencies that are loaded by building the library with <code>-z interpose</code>.

Surfing with the Linker-Aliens

Published Elsewhere


Surfing with the Linker-Aliens

[12] Interface Creation (compilers)
Blog Index (rie)
[14] Relocations Don't Fit