kldd: ldd Style Analysis For Solaris Kernel Modules

Ali Bahrami — Thursday January 11, 2018

Surfing with the Linker-Aliens

Solaris 11.4 comes with a new user level command, /usr/bin/kldd. kldd performs ldd style analysis for kernel modules.

The ldd utility is used to analyze the dependencies of a userland dynamic object. Given an object, it tells you what other objects it depends on, as well as revealing related details. This is very basic, and very useful ability. ldd has been part of our product since it debuted, along with support for dynamic linking in SunOS 4.0 in late 1988. It subsequently became part of AT&T System V Release 4 Unix, and has been adopted by numerous other Unix vendors, and Linux. The reason for this ubiquity is that ldd performs a basic and necessary service.

Until now, there was no similar utility for kernel modules, even though such a utility would be as useful for kernel modules as for any other type of object. There are many possible reasons for this lack of support, including:

Kernel module infrastructure is relatively primitive, and has historically not had a group with the necessary linking background dedicated to improving it.
Kernel modules are created by a small number of people, and those people (kernel programmers) are used to getting by without the sort of hand holding required by programmers working higher in the stack.

In recent years, the Solaris linking group has done work to improve this situation, and bring more of the userland linking abilities we take for granted to the kernel. Early steps have included a recent overhaul of the kernel runtime linker (krtld), and the explicit tagging and finalization of kernel modules.

One can imagine future projects building on this base:

Defining tighter interfaces between modules, by employing ELF versioning.
Having the link-editor (ld) employ its library (-l) option to automatically generate dependency DT_NEEDED records, and eliminate the need to manually specify them with the -N option.
Employ stub kernel modules to enable building the OS with more parallelization, while preventing accidental dependencies from occurring.

These are obvious and time tested (decades in same cases) linking concepts that have already proven themselves in userland. As was the case in userland, projects like these require basic tools for testing and verification. Most of our tools (ld, elfdump, elfedit, etc) work equally well for userland objects and kernel modules. However the lack of a tool that can do ldd style analysis for kernel modules has long been a large and obvious gap in functionality. Such a tool is generally useful, in addition to facilitating projects such as the above.

Background (Rejected Options, and Why)

The obvious way to solve the lack of an ldd-like tool for kernel modules would be to simply extend ldd to handle kernel modules. However, this turns to be more problematic than might appear at first blush.

ldd is really a small wrapper program that causes the runtime linker to run in a special ldd mode. The runtime linker goes through its normal object loading and setup, but in ldd mode:

Prints the informational messages that you see as ldd output.
Does not execute any code, and exits at the point where it would normally pass control to the program main().

ldd has a variety of options for getting ld.so.1 to print differing types of information. These options are conveyed from the ldd wrapper to the runtime linker via private environment variables. In addition, knowledgeable users can use the LD_DEBUG environment variable to obtain information that the ldd options do not address. For instance, the lari utility runs a command similar to the following in order to obtain information about symbol binding:

    % ldd -e LD_DEBUG=bindings,files,detail object

The runtime linker has no useful analysis to offer for objects other than dynamic objects (executables or shared objects). ldd is therefore useless with relocatable objects or kernel modules.

The implementation and features of ldd are all directly related to userland objects and the implementation of the userland runtime linker. I briefly explored the idea of having the runtime linker load kernel modules in a special mode in which it applies the rules employed by the kernel runtime linker (krtld) and disables any non-applicable userland functionality. However, it quickly became evident that this is a bad idea, as there is relatively little overlap between the way the userland ld.so.1 works, and krtld. I believe that the code in ld.so.1 that we could use for analyzing kernel modules is less than the amount of code we would need to add in order to special case ld.so.1 so that it could handle them. Adding such complexity to the already complex ld.so.1 is a move in the wrong direction.

Given that we cannot use the userland ld.so.1 to do this analysis for kernel modules, the next best idea would seem to be to write the necessary code independently of ld.so.1, and to have the ldd wrapper call that code for kernel modules, rather than doing its usual trick in running the runtime linker. This is technically feasible, but I believe that it would be counter productive. There are 2 main reason for this:

If you read the ldd(1) manpage, you will note the large number of options that are only applicable to userland objects. Along with these features comes a large amount of documentation describing the underlying concepts and how to interpret them. Were ldd to support kernel modules, many of these options would have to be disallowed, and/or would work differently. All of these differences in syntax and behavior would have to be documented in one ldd(1) manpage. The result would be a large number of conditional sentences of the form "does XXX for userland objects, but YYY for kernel modules" which would render an already complicated manpage nearly incomprehensible.
In the case of kernel modules, the LD_DEBUG feature (shown above) along with all the other environment variables (e.g. LD_LIBRARY_PATH) that can be used to control ld.so.1, and by extension, ldd, do not exist. I believe that this would be a source of unending confusion for users, most of whom have only passing knowledge of linking internals and could not be expected to understand these implementation details.

The more I considered how to make this work the more I realized that we are talking about 2 similar, but distinct utilities. It will be easier to document, maintain, and understand them as separate things. As such, I settled on providing this functionality with a new command, kldd, specialized to the needs of kernel modules, and free of the baggage that integration with ldd would bring. The ldd and kldd utilities share a basic similarity, but are free to differ when that makes sense. For instance, the LD_DEBUG technique employed by lari is supported by kldd directly via a -b option, something that would make little sense for ldd to also support.

If ldd is implemented as a wrapper on the userland runtime linker, ld.so.1, should kldd be implemented as a wrapper on the kernel runtime linker, krtld? There is an appealing symmetry in that thought, but the two cases are rather different, and the ultimate answer is "no".

All of the work done by ldd occurs within a userland process, which is discarded when ldd exits, leaving no trace on the system, and posing no security threat. Because everything is encapsulated within a single process, ldd does not have to erase its tracks. In contrast, any module loaded into the kernel can have impacts on memory layout that may persist until the system is rebooted, and would have to be coded carefully to prevent security issues. An in kernel implementation would therefore be relatively complex.
The mechanisms by which a userland program named kldd can communicate with the kernel are also relatively complex and awkward.
The behavior of the kernel runtime linker is considerably simpler than that of the userland runtime linker. While anyone should hesitate to attempt to emulate the userland ld.so.1, the reverse is not true. The behavior of the kernel runtime linker relevant to a kldd utility is simple enough to be effectively emulated in userland.

A basic truism of operating system development is that only things that cannot be achieved in userland belong in the kernel. kldd can be implemented as a userland process, and in so doing, benefit from the usual protections and resource management that provides.

Examples:

The following examples are taken directly from the kldd(1) manpage.

Example 1 Displaying module dependencies by file path

The following example displays dependencies of the misc/idmap kernel module, specified by its file path.

$ kldd /kernel/misc/amd64/idmap
        sys/doorfs =>   /kernel/sys/amd64/doorfs
        strmod/rpcmod =>        /kernel/strmod/amd64/rpcmod
        misc/tlimod =>  /kernel/misc/amd64/tlimod
        unix  (parent) =>       /platform/i86pc/kernel/amd64/unix
        genunix  (parent dependency) => /kernel/amd64/genunix

Example 2 Displaying module dependencies by soname

The following example displays dependencies of the misc/idmap kernel module, specified by its module soname.

$ kldd misc/idmap
         sys/doorfs =>   /kernel/sys/amd64/doorfs
         strmod/rpcmod =>        /kernel/strmod/amd64/rpcmod
         misc/tlimod =>  /kernel/misc/amd64/tlimod
         unix  (parent) =>       /platform/i86pc/kernel/amd64/unix
         genunix  (parent dependency) => /kernel/amd64/genunix

Example 3 Viewing search path details

The following example displays dependencies of the misc/idmap kernel module, showing the search details.

$ kldd -s misc/idmap
     search path=/platform/i86pc/kernel:/kernel:/usr/kernel  (default)
     trying path=/platform/i86pc/kernel/misc/amd64/idmap
     trying path=/kernel/misc/amd64/idmap

    find module=sys/doorfs; required by /kernel/misc/amd64/idmap
     trying path=/platform/i86pc/kernel/sys/amd64/doorfs
     trying path=/kernel/sys/amd64/doorfs
         sys/doorfs =>   /kernel/sys/amd64/doorfs

    find module=strmod/rpcmod; required by /kernel/misc/amd64/idmap
     trying path=/platform/i86pc/kernel/strmod/amd64/rpcmod
     trying path=/kernel/strmod/amd64/rpcmod
         strmod/rpcmod =>        /kernel/strmod/amd64/rpcmod

    find module=misc/tlimod; required by /kernel/strmod/amd64/rpcmod
     trying path=/platform/i86pc/kernel/misc/amd64/tlimod
     trying path=/kernel/misc/amd64/tlimod
         misc/tlimod =>  /kernel/misc/amd64/tlimod

    find kernel=unix; implicit
         unix => /platform/i86pc/kernel/amd64/unix

    find module=genunix; required by /platform/i86pc/kernel/amd64/unix
     trying path=/platform/i86pc/kernel/amd64/genunix
     trying path=/kernel/amd64/genunix
         genunix =>      /kernel/amd64/genunix

Example 4 Exposing kernel module deficiencies

The following example employs a pair of kernel modules, demo/mod_a, and demo/mod_b, which have not been installed on the running system, and which are therefore found in a directory named /local/test. The -R option is used to allow kldd to locate them. These modules have intentionally been built with some deficiencies in order to demonstrate the operation of the -p, -r, -U, and -w options.

This output has been formatted for display purposes, and errors not directly related to mod_a or mod_b have been omitted.

$ kldd -prUw -R real demo/mod_b
         demo/mod_a =>    /local/test/demo/amd64/mod_a
         misc/sha2 =>    /kernel/misc/amd64/sha2
         misc/kcf =>     /kernel/misc/amd64/kcf
         unix  (parent) =>       /platform/i86pc/kernel/amd64/unix
         genunix  (parent dependency) => /kernel/amd64/genunix
    symbol not found: extern_sym  (/local/test/demo/amd64/mod_a)
    symbol not found: parent_sym (/local/test/demo/amd64/mod_a)
    symbol not found: weakref_sym   (/local/test/demo/amd64/mod_a)
    unreferenced object=/kernel/misc/amd64/sha2;
        unused dependency of /local/test/demo/amd64/mod_a

Surfing with the Linker-Aliens

Published Elsewhere

https://blogs.oracle.com/ali/kldd/

Surfing with the Linker-Aliens

[37] Core File Enhancements for elfdump Blog Index (ali) [1] Testing...Without...System Into A Brick