What I Did On My CTF Vacation
| What I Did On My CTF Vacation | |
Ali Bahrami — Friday July 17, 2026
My day job is to take care of the Solaris Linkers, and all things that involve the ELF object file format, and its application to Solaris. A nice thing about this job is that the linkers are adjacent to, and intersect with, a wide swath of the OS. If working on something outside my area improves the overall linking experience in a significant way, I can go do that. This adds variety and keeps things interesting. These side trips are common, but are usually short lived, unlike my recent journey through the land of CTF.
CTF (Compact C Type Format) was a Sun invention from a few years before I joined Sun in 2005, created to enhance the abilities of core observability tools such as mdb/kmdb, DTrace, and the proc tools. CTF has been included in the objects delivered with the Solaris OS for many years, and is a linchpin of the Solaris observability story. From the day CTF was introduced to Solaris, there as been a desire to expand its use outside of the confines of the OSnet, at a minimum, to the Open Source software we deliver from the Solaris Userland consolidation, and ideally even to 3rd party and customer built objects. These wider goals went unaddressed for years. CTF intersects with linking and ELF, but for various reasons, I was not pulled into working on it until recently.
Starting in April 2024, I allowed myself to start poking at the Solaris CTF, and things snowballed into a large rolling project to modernize and improve things. Now that the dust is settling, I've written this article to describe the changes made, and provide the details of how we got there. The features should be of interest to anyone who builds code on Solaris, and would like to use mdb/DTrace with that code. The lower level details may be of interest to those who wonder how we threaded the needle through the possibilities. Some of it will not be news, and may retrace work you've already done yourself, but I hope there are also some fresh ideas and useful information.
If you just want to know how to add CTF to your objects, but are not all that interested in reading a very long article about how it works, please see the ctf(7) manpage. If you are interested in some of this, but not all of it, here is a Table Of Contents:
The following is a summary of the CTF improvements that I have made to Solaris over the last couple of years.
#define SHT_SUNW_ctf 0x6fffffeb
The CTF tools, libraries, and the OS have all been modified to use SHT_SUNW_ctf for this purpose:
% elfdump -cN.SUNW_ctf /lib/libc.so.1
Section Header[36]: sh_name: .SUNW_ctf
sh_addr: 0 sh_flags: 0
sh_size: 0x14d5f sh_type: [ SHT_SUNW_ctf ]
sh_offset: 0x2627c8 sh_entsize: 0
sh_link: 30 sh_info: 0
sh_addralign: 0x4Support for .SUNW_ctf sections of type SHT_PROGBITS will remain permanently in place to support existing objects in the field, but are no longer produced for new objects.
The ELF format allows section type to be defined for specific purposes. Doing this, rather than declaring sections as SHT_PROGBITS and relying on strcmp() to distinguish them is considered an best practice, as it is both more efficient, and self documenting. SHT_PROGBITS is intended primarily to label code sections that linkers do not decode or examine, but historically has also been used for debug sections. This is appropriate for the debug sections produced by compilers for use by their own debuggers (e.g. cc and dbx, or gcc and gdb) because as with code, the linkers and similar tools do not decode or examine them directly. I believe that SHT_PROGBITS was originally applied to CTF sections because they also also serve a debug purpose. However, CTF is different than those other sections, because CTF is standardized and manipulated by the operating system, and is a core part of our infrastructure. As a linker alien, I've long been aware that that CTF should probably have its own type, but it was a minor concern, and we left it alone. As I started this large CTF project, it seemed like a good time to warm up with this fix that falls squarely in my usual linkers/ELF box, an easy way to start changing CTF code and get my feet wet.
While the intent was to make libctfgen usable by other callers, it was not initially in a state that would support that. In its original form, it called exit on any error, had its memory allocator hardwired to use malloc(), and printed all errors directly to stderr, rather than allowing the calling application to manage those details. These were all reasonable choices for the context that code was originally written to support, but library code needs to be written to give the calling application control over such details. The link-editor has its own memory allocator and error reporting mechanisms that it would need libctfgen to integrate with, and cannot tolerate having a library function call exit(). Those things needed to change, to continue support the original utilities, while allowing wider use. Once libctfgen was established, I audited and modified its code line by line, to allow the caller control over memory allocation, and error reporting, and to return an error status code indicating one of TRUE, FALSE, or ERROR, which each caller either handles, or passes to its caller, as appropriate. The result of this painstaking overhaul was to bring this code to library standard while preserving the correctness of its basic operation.
At the same time, I modified ctfdump to xlate CTF content on input, allowing it to display CTF content for an object from platform with the opposite byte order (e.g. using ctfdump on x86 to examine a sparc object).
% cc -c -g main.c
% ctfconvert -S main.o
% cc -c -g sub1.c
% ctfconvert -S sub1.o
% cc -c -g sub2.c
% ctfconvert -S sub2.o
% cc -o prog main.o sub1.o sub2.o
% ctfmerge -o prog main.o sub1.o sub2.o
That is still supported, but the ld -zctf option can greatly simplify the process:
% cc -c -g main.c % cc -c -g sub1.c % cc -c -g sub2.c % cc -o prog main.o sub1.o sub2.o -zctf=convert
Or:
% cc -c -g main.c sub1.c sub2.c % cc -o prog main.o sub1.o sub2.o -zctf=convert
Or Even:
cc -g -o prog main.c sub1.c sub2.c -zctf=convert
There is a subtly in my claim of efficiency that might be worth mentioning. When ld -zctf=convert is used, the ctfconvert operation is done for every input object, on every link. In the usual case where everything is compiled and linked once, there is one convert operation per object, and the objects are not rewritten, which is clearly more efficient. However, this may be less efficient overall in the case where you are engaged in an ongoing compile/link development cycle for a program with lots of input objects, only a few of which change. In that case, having ctfconvert run on each input object only when that object is recompiled may turn out to be more efficient in aggregate, than repeating it with each link. I doubt that you would feel the difference, and don't think it's worth worrying about, but mention it in the interest of full disclosure. The convenience of -zctf far outweighs any development time overhead for any but the largest of programs (see the next item).
The largest speedup came from eliminating the use of gelf_getsym(). gelf_getsym() uses a mutex internally to make it thread safe, but in this context, there is only one thread, and the accesses are all readonly, so locking is unnecessary. I replaced the call to gelf_getsym() with a libctfgen-specific version that doesn't lock, and the runtime dropped to 3 hours, which is a dramatic win, while still being much too slow.
The other speedup came from rewriting check_for_weak() to use an AVL tree for these lookups, built only when weak symbols are present. There is a one time setup cost which is O(n), followed by an O(logN) cost per weak symbol. Using this version, the customer link dropped to ~6 minutes, which returns it to the realm of the possible. I would like to think that even this might be improved, but that is left for another day. The impact on more normal links is of course far less dramatic, but still provides a useful speedup for our OS builds. The per-objects savings may be small, but add up across a full build of many objects.
Version 2 remains supported, and is still used by the Solaris OS, which has no issues with the limits of that version, but version 3 is now the default, and will be used unless the version is explicitly specified. The CTF version is a per-object choice. Each object loaded in a process is free to use either version, independently of any other object, and there is no performance penalty for such mixing.
Noting that the C in CTF stands for "Compact", it should be noted that version 2 remains smaller for obvious reasons, but version 3 is also very compact. When doing this work, I built the OS both ways, to quantify the cost. I found that on disk, where compression is used, that the CTF version 2 encoding of the entire OSnet takes 94.5% of the space required for a version 3 encoding, and that the absolute difference amounts to a mere 1M. The numbers in memory are slightly worse: CTF version 2 would require 78.9% as much memory as CTF version 3 for the entire OSnet. This difference is still only 11.2M, but note that getting to that number would require loading every OS object into a single process. That is probably not even possible, and is definitely not realistic. Hence, using version 2 is a harmless micro-optimization for code that can use it, but most users should not be concerned, or even give it a second thought. Our choice to continue using version 2 for the OS was primarily made on the basis of backward compatibility and a desire to see that code regularly exercised. The resulting space savings are nice, but inconsequential.
% gcc -gctf -o prog main.c sub1.c sub2.c -zctf
Thanks to our friend Rainer Orth at Bielefeld University, the latest version of gcc now has a -gsctf option that passes -gctf to the compiler internally, and -zctf to the link-editor, allowing those two options to be replaced by a single one:
% gcc -gsctf -o prog main.c sub1.c sub2.c
The best reference for GNU CTF that I know of is The CTF File Format by Nick Alcock. This document goes into sufficient detail that I was able to determine that while GNU CTF differs significantly from Solaris CTF in its container concept and its version of libctf, their version 3 CTF itself lines up very closely with Solaris CTF version 3, so much so that it was fairly straightforward (though still a pile of work) to have the Solaris libctfgen recognize their .ctf sections, and transparently convert them to native Solaris CTF. The only difference of high level significance is the addition of a single added CTF kind, CTF_K_SLICE. Initially, I did not implement SLICE, translating them instead into CTF_K_INTEGER with an encoding.
typedef int myint_t;
struct st_int {
myint_t i_zero;
myint_t i_one:4;
myint_t i_two:5;
myint_t i_three:9;
myint_t i_four:32;
} x;
enum num { ZERO = 0, ONE, TWO, THREE, FOUR };
typedef enum num num_t;
struct st_enum {
num_t e_zero;
num_t e_one:4;
num_t e_two:5;
num_t e_three:9;
num_t e_four:32;
} y;The fields in x are integers, and can be properly represented in CTF using the CTF_K_INTEGER kind, with an appropriate encoding. However, the fields in y are enum type, represented with CTF_K_ENUM, and unlike CTF_K_INTEGER, CTF_K_ENUM makes no provision for specifying an encoding, causing CTF to represent them all as 32-bit ints. This bad data confuses mdb, which is how we first became aware of the flaw. One workaround for this is to turn such cases into CTF_K_INTEGER, but doing so is very unfortunate, because the nice mnemonic name for the value provided by the enum is lost in the process. SLICE is the answer for this, allowing an encoding to be applied to a kind that does not have that ability itself. As far as I know, the only kind for which this makes sense is ENUM, and Nick confirms for me that this is the main purpose for which GNU CTF uses SLICE.
% file /lib/64/libc.so
/lib/64/libc.so: ELF 64-bit LSB dynamic lib AMD64 Version 1 [SSE2 SSE],
dynamically linked, not stripped, no debugging information available,
CTF presentThis is a small thing, but as people started using the new CTF features, they wanted to confirm that it was working. While they could have used elfdump to look for the resulting .SUNW_ctf section, it became apparent that file should provide an easier way to check, in line with its traditional reporting of debug information.
% elfdump -cN.SUNW_ctf /usr/bin/python
Section Header[31]: sh_name: .SUNW_ctf
sh_addr: 0 sh_flags: 0
sh_size: 0x10f sh_type: [ SHT_SUNW_ctf ]
sh_offset: 0x322c sh_entsize: 0
sh_link: 27 sh_info: 0
sh_addralign: 0x4With this, we've achieved the overall goal of spreading CTF widely into code outside of the OSnet itself.
The result is shorter and sparser output that is easier for the eye to track.
% ctfdump /usr/bin/ls | grep cth_flags cth_flags 0x1 [ COMPRESS ]
When this mechanism is used, the .SUNW_ctf section header will have the SHF_COMPRESSED flag set.PSARC/2013/139 ELF debug section compression PSARC/2015/543 public libelf section compression APIs
There is a simple explanation for this unfortunate state of affairs. The CTF format predates the addition of support for compression to ELF by over a decade. Had ELF support for compression existed in the late 90's, the CTF authors would almost certainly have used it rather than complicating their own work.
Over the years, we have built out our support for ELF section compression throughout the system, in linking utilities, libproc, debuggers, and similar tools. That support has been reasonably complete and functional for years, but our CTF generation code (ctfconvert, ctfdump, ctfmerge, libctfgen) was a notable exception:
This state of affairs has persisted for years, because the problems have been relatively low priority, because there was not much work happening on CTF, and because improvement was not necessary for building the OS/net, which was the only user of CTF. This project changes those calculations, and delivers several improvements:
I should say that the goal of this work is not to encourage or discourage the use of compression, or even to spur a conversion from the original CTF compression to the ELF version. My immediate concern was to ensure that our basic toolchain utilities can handle all the possibilities, and that end users can access those features using simple and documented interfaces. For more information on the options for CTF compression, see ctf(7).
% cc -g -c ~/hello.c % ctfconvert hello.o % cc hello.o % ctfdump a.out ctfdump: a.out: CTF not associated to a symbol table: ctfmerge required
Historically, compilers have produced debug sections to describe the generated code. These sections, which initially were in the Stabs format, and then later DWARF, are used by debuggers such as dbx and gdb to enable debugging of code at the source level.
Low level operating system debuggers, notably adb, were shut out of using this information: It was too large to be included in all OS objects, often complex to access efficiently, and variable between different compilers. That data was intended to serve the needs of specific compilers and to provide a large amount of source level information, information not needed by these low level tools. When mdb was developed to replace adb, the CTF concept was invented to support it. The idea was that a tool could read the stabs/dwarf content from object files, and generate a small description of the basic type information that an assembly level debugger requires. CTF was introduced with Solaris 9 in 2001:
PSARC/2001/021 Kernel Stabs
CTF provides a description of the types used by code, expressed at the level of the C language, small enough to be included by default, and designed for rapid access, meeting the fast and light requirements of low level introspection tools.
In 2003, CTF was extended to userland, first appearing in Solaris 10:
PSARC/2003/283 Userland CTF SupportInitially, that support was used in libc, and then more recently, for all shared objects delivered by the OSnet. When I joined the Solaris OS group in late 2005, that is where things stood. CTF and the tools that enable it were an invaluable part of Solaris, but they were expert tools, usable by their creators in house, but possibly not yet ready for publication, nor easy enough to use for a mass audience.
Discussions about finishing this work, and raising the commitment level would come up periodically. The upside from raising the commitment level of CTF is clear: having CTF in more objects will improve the ability of mdb, DTrace, and other observability tools to extract useful information, in more situations. However, over the years, there have been various reasons why it didn't happen.
The original CTF authors took this post-processing approach by necessity. Ideally, CTF would be created by compilers, and merged during the link-edit, but the project would have been dead on arrival had they taken that approach. There are too many compilers, and they are outside of the operating system's purview. The link-editor is under our control, and a good place for this, but integrating a feature like that is distinctly non-trivial. At the beginning, this was an R&D effort with an uncertain future. The light weight and relatively agile post-processing approach allowed rapid progress, and kept the focus on proving the CTF concept. That was a full success, but the follow on work to deliver that functionality with an easier interface remained, and the best way to do that was unclear.
I recall discussing the idea of moving CTF generation to the link-editor (ld) around 2012 or so. It seemed the logical next move, but as noted, a very large job. We in the linking group did not have a working knowledge of that code, and its authors were no longer around to consult. Of course, the code was there to be read and studied, so doing something was definitely still possible, but it would have been slow going, and this was just one of many projects needing linker alien attention, and not necessarily the most urgent one. However, the larger reason why nothing was done is that there was a debate in the OS group at that time about whether CTF was necessarily the right way forward. CTF had successfully proved the benefit of a small efficient format for this purpose, so that was not in question. However, it was a non-standard addition, unique to Solaris, and systems descended from Solaris. In the same years, the wider world had moved to the DWARF format for representing debug data, which has become as close to a universal standard as we're likely to see. Did it makes sense to figure out a way to use that, rather than promoting a separate niche format just for low level debuggers?
DWARF is not the most efficient format to access, but that could have been managed. The important concern is that unlike CTF, DWARF is decidedly not compact. This is partly because the encoding is verbose, partly because compilers load it with all the information a source level debugger could ever want, and partly because the compilers don't always do a good job of minimizing redundant duplicate data. It is not unusual for DWARF debug data to be 10x the size of the code it describes. This is not a Solaris-specific problem, but rather, something that's felt industry wide. Given the generality of the issue, we had some reason to think that the situation might improve, and that compilers might evolve to offer a "slim DWARF" option that might create data compact enough to replace CTF. Another DWARF concern is that different compilers generate slightly different DWARF dialects, and here again, it was hoped that a slim DWARF might encourage more standardization. In general, we thought that as an industry standard format, a slimmed down DWARF option might emerge that would render CTF redundant.
And so, time passed. Efforts to create a slimmed version of DWARF were made, but there was not much progress, and the status quo persisted. DWARF debug data today is as large as ever, and there is no reason to think that will change any time soon. Meanwhile, CTF has quietly plugged along with minimal change, quietly solving the problem it was invented to solve, but only for core OS system objects.
In April 2024, the question came up again, this time as an expressed desire to add CTF to a selection of key open source packages delivered with Solaris from our Userland consolidation, to enable mdb to do a better job of diagnosing problems in that code. By this time, it had long been evident to me that the slim DWARF idea was a dead end, and that CTF was the obvious way forward. I knew very little about the CTF format, or the existing code, but the things I do know about adjacent areas (compilers, linkers, libproc, etc) made me well positioned to take a deeper look. No one else was presently working in that area, and I had just finished a long term project and was thinking about what my next one should be. So, I stated reading code, and weighing options.
I already knew that CTF had moved into the wider world with OpenSolaris, fueled by ports of DTrace, and existed on darwin (macos), FreeBSD, and Linux. CTF is no longer a Solaris-only concept, and appears to be thriving. Indications are that it will be around for many years, whatever else might also emerge. As I started looking at trying to make the Solaris version public, I learned that some versions of CTF outside of Solaris have evolved in some compelling ways:
The direct relevance of CTF version 3 and GNU CTF, to Solaris was not evident to me at that time, but I could see that CTF was progressing, which seemed a positive sign, and a reason to keep going. It was immediately evident that we should move to clean up and publish the CTF utilities, ctfconvert, ctfmerge, and ctfdump, as supported and documented system utilities. However, it was also clear that this would not be enough to cause a wide uptick in CTF adoption in Userland opensource. The changes necessary to their makefiles would be complicated, require ongoing maintenance, be hard to upstream, and as a Solaris-only feature, relatively uninteresting to those upstream projects. One might do it for a few key projects, but it would be difficult to scale it to a large number. Also evident was that if we could somehow give the Solaris link-editor the ability to do the work of ctfconvert/ctfmerge, turning the necessary upstream change into the simple addition of a linker option, that would be far more likely to work widely, and be sustainable.
The Solaris link-editor has the ability to read command line options from the LD_OPTIONS environment variable, a feature we've long used to pass Solaris-specific options to open source software in our Userland consolidation, behind the back of their build environments. Given an ld option for adding CTF to the output object, LD_OPTIONS could be used to trigger that option for a substantial fraction of the code built from Userland with almost no added effort. Of course, this won't work for everything. Some software has build systems that filter environment variables, and not everything is written in C, the sweet spot for CTF. Even so, a substantial percentage of Userland could be reached, including most of the things we really care about adding CTF to. And so, I came up with the following rough list of high level tasks as a starting point:
Behind each of these high level bullet points lie many subproblems. The detailed TODO list quickly became long, and raised many questions. The overall idea was large, speculative, with uncertain prospects. However, as is often the case in such situations, it was immediately clear that most of the subproblems were tractable and could definitely be solved, and each one would be of value, no matter how the larger project turned out. As each such subproblem was solved, the remaining pile would become smaller, and with luck, simpler, and clarity would emerge. In the worst case, we'd have make some improvements, and moved past the current block. In the best case, we'd make a breakthrough.
And so, it was time to stop studying, start coding, and see where things led. I made a detailed TODO list, a list of unresolved open questions, picked a problem I knew how to tackle, and got started. The result was a 2 year journey through the CTF space that far exceeded expectations, and which is only now winding down. For the first year, my TODO and open questions list grew even as I crossed items off, because the work led to more insights and exposed me to new things to think about. That eventually slowed down as things coalesced into their final shape. The TODO list began to shrink, and is nearly empty now.
This section describes the actual path taken, fix by fix, with additional comments for each describing the thought process at the time, and discussing lower level implementation details. This information is probably not of interest for most readers, but may be interesting to those also working their way through the issues of evolving CTF. It shows how we got from our starting point in 2024 to where we are now in 2026, while continuously maintaining the ability to build, use, and ship the system to customers, throughout that period.
Each subsection below describes one fix, given in the order fixed. Each subsection begins with a list of the bugs and PSARC cases it resolved. In most fixes, the buglist is followed by a "[ High Level Goal ]" link that if pressed will take you back to the high level discussion above, for the goal this fix contributed to. Note that multiple fixes can refer to the same goal. A discussion of the lower level details of the fix then follows. To establish context in this discussion, some information discussed above may be repeated, but I have tried to minimize such repetition.
PSARC/2024/076 SHT_SUNW_ctf: Dedicated ELF section type for CTF 36675528 SHT_SUNW_ctf: Dedicated ELF section type for CTF
An easy warmup, before getting down to business, this change creates an explicit ELF section type for Solaris .SUNW_ctf sections, SHT_SUNW_ctf, and modifies CTF code and the rest of the OS to use it. Support for the older use of SHT_PROGBITS is retained to handle older objects. SHT_SUNW_ctf is part of the Solaris specific ELF ABI, so only applicable to objects tagged as ELFOSABI_SOLARIS, which is the case for most Solaris objects:
% elfdump -e /lib/libc.so.1 | grep ei_osabi ei_osabi: ELFOSABI_SOLARIS ei_abiversion: EAV_SUNW_CURRENT
One reason for this change was just to tidy up something that I had long been aware of. Another was to further distinguish our native CTF .SUNW_ctf sections from the .ctf SHT_PROGBITS sections produced by gcc -gctf. None of this is strictly necessary, but it makes things nice and tidy. Independent of that, it is worth noting that Solaris and GNU ctf sections, by virtue of having different names, can coexist within a given object. However, there is no benefit to doing that, and it is not expected to happen in practice.
PSARC/2024/077 ctf strip class for ld and strip 36675615 ctf strip class for ld and strip
The strip utility, and the ld -zstrip-class option, treated the debug sections produced by compilers along with .SUNW_ctf sections as related generic debug content, and would remove all of it under the 'debug' category. Treating CTF as if it were part of normal debug data was a design error. The debug sections produced by the compilers (used by their associated debuggers), and the .SUNW_ctf sections (used by libproc, libctf, mdb, DTrace, etc) are disjoint, and exist for distinctly different purposes. It follows that users might want to keep one and remove the other. The inability to do that had not been an issue when all CTF use was done internally by the OS itself, but would clearly become one once CTF became committed, documented, and more widespread. This fix introduced a new 'ctf' debug class for strip and ld -zstrip-class, allowing them to be treated as distinct categories.
PSARC/2024/078 making CTF public 36650566 making CTF public
This was the opening move in the effort to make CTF more widely available, and there was a lot to it:
- -a
- Implicitly run ctfconvert on input objects that do not already contain a .SUNW_ctf section. The resulting CTF data is merged, but the object itself is not altered.
- -i
- Used with -a, and serves the same purpose as 'ctfconvert -i', which is to silently ignore non-C objects.
- -f
- Match any global symbols in the output object that were scoped to local during the link-edit process, to the global symbols referenced by the CTF data found in the input objects. Global symbols can be scoped to local during a link-edit with a mapfile, or with link-editor command line options. See ld(1), and the "Oracle Solaris 11.4 Linkers and Libraries Guide".
-f stood for 'fuzzy matching', implying that a heuristic is being applied. However, formerly global symbols which have been scoped to local with a mapfile are unambiguously identified by their position in the symbol table. I think that there must have been some confusion about how scoped globals should be handled back when the -f option was added, but it does the right thing, and there is nothing "fuzzy" or hueristic about it. This should have been the standard non-optional behavior, and with this fix, it now is.
- -S
- Strip compiler generated debug sections from the resulting object.
The -g option will continue to be accepted for backward compatibility with existing makefiles, but has no effect, and is silently ignored. This change is documented in the ctfconvert and ctfmerge manpages, to assist users familiar with the older versions of these tools.
36730716 CTF utilities should handle non-native byte order
In order to be used by the link-editor (ld) when operating as a cross link-editor, libctfgen was enhanced to swap CTF data when it moves from file to memory and back, when the byte order of the running system differs from that of the object.
The same enhancement was made to the ctfdump utility, allowing the x86 version of ctfdump to dump sparc objects, and vice versa, something that I personally find very convenient in my daily work, since I constantly move between such systems.
The ctfconvert process relies on libdwarf, and I had a concern about whether or not libdwarf would be able to operate on objects for a system with the opposite byte order from the running system. I was happy to find that libdwarf transparently handles that case.
PSARC/2024/090 ctfdump support for multiple files and archives 36827583 ctfdump should handle multiple arguments and archives
This change was a general cleanup and modernization of the ctfdump code to allow multiple input arguments, and to prepare it for the upcoming project to overhaul the error handling in libctfgen. It did the following:
36827626 ctfconvert should report unlink errors PSARC/2024/097 ctfconvert and ctfmerge -r option 36847015 ctfconvert/ctfmerge should not unlink input files on error by default
Shortly after integrating the fix to make CTF public, I had a reason to examine the CTF in libc, but accidentally typed 'ctfconvert' rather than 'ctfdump', and got this momentarily heart stopping result:
% ctfconvert /lib/libc.so.1 ctfconvert: /lib/libc.so.1: no type data to convert ctfconvert: removing: /lib/libc.so.1
Deleting libc is a fast way to turn your system into a hot brick, so that definitely got my attention. Thankfully I was running as an unprivileged user when I did that, so the attempt to remove libc failed, and my system was unharmed. This immediately raised 2 questions:
The failure to report that the unlink failed was nothing more than a missing error message, which was easily fixed.
The reason an unlink was attempted in the first place is that this is how ctfconvert was designed to work. ctfconvert and ctfmerge are usually run from a makefile. If the CTF operation failed, the utility would intentionally unlink the input file, ensuring that the next invocation of make will rerun the full build rule, rather than assuming that all is well, and using a half built object. This is undeniably useful behavior, and should be available, but in general, destructive side effects should require an explicit option rather than being the default.
I had already declared ctfconvert to be committed and public with PSARC/2024/078, but since it hadn't yet left our walls and been delivered as shipping product, there was still time to fix this before it would be seen in the wider world. I therefore quickly filed another PSARC case, and reversed the default behavior so that this unlink no longer happens by default. To get the old behavior, one can simply specify the new -r option to ctfconvert and ctfmerge, which I immediately did in the OSnet makefiles.
You can still get the scary message, but now, you have to be intentional about it, turning it into a requested action (like running rm), rather than being a hidden side effect. If you do that, you will now see that the unlink failed, which can be reassuring:
% ctfconvert -r /lib/libc.so.1 ctfconvert: /lib/libc.so.1: no type data to convert ctfconvert: removing: /lib/libc.so.1 ctfconvert: unable to remove file: /lib/libc.so.1: Permission denied
36918319 enable use of SHT_SUNW_ctf for CTF sections in ELF objects
This fix enabled the use of the new SHT_SUNW_ctf section type. Support for SHT_SUNW_ctf was integrated earlier, but was not immediately used. The reason for this is that older versions of the operating system are used to build new ones, so new ELF features can't be used until they have arrived on the oldest supported build servers. We have rules capping how old a build server can be. When new features land, we seed those abilities without using them, delay for the waiting period, and then turn them on with a follow on fix, like this one.
36927818 libctfgen code needs to be brought to library standard
Having moved the common CTF generation code to libctfgen, it was time to overhaul/refactor it to be usable as a general purpose library usable by the link-editor (ld). This meant:
These are only the high points. There are many details involved in turning old code like this into a good library. The code had good bones, but was written for a different context, and had not had a significant revision in decades. There were many smaller things to chase and address.
The most challenging part was reworking how errors are reported, and having the library return a status code to the caller, rather than simply calling exit(). The basic strategy was simple: Declare a standard return type, ctfgen_ret_t, with three possible values, CTFGEN_RET_TRUE, CTFGEN_RET_FALSE, and CTFGERN_RET_ERROR, and then use it throughout the code base. Code that used to return TRUE or FALSE was changed to return CTFGEN_RET_TRUE or CTFGEN_RET_FALSE. Code that used to output an error message and call exit() was changed to output the message and return CTFGEN_RET_ERROR. Code that had never checked a return status, because the called code never returned on failure, had to be modified to check, and handle that case. All routines at every layer had to be modified to return these code back to the caller, and ultimately, to the outside caller of the library.
It was more complicated that it sounds. The amount of code is large, there is a lot of nested indirection involving nested function pointers, and a small mistake could, and more than once did, break everything. The odds of making so much change in one go successfully are essentially zero. This sort of work really can't be done without excellent tests, and constant testing. I was lucky, because both mdb and DTrace have extensive and thorough test suites. The winning strategy was to iterate, being careful to take small bites at each step, to make constant backups at each step, saving all the backups, and then to run both the mdb unit tests and DTrace test suite frequently.
This was not a fast process, and it required a great deal of focus and careful deliberate strategy about what to change, and in what order. Furthermore, the build/test step takes 4-5 hours, futher slowing the pace. This was probably the most painful part of the CTF project, but once I accepted that this was how to do it, I made steady progress and came out the other side with a viable working library.
Once I got the hang of how big a swing to take with each iteration, I had more passes than fails, but there were definitely times where I broke something, and was unable to spot the change that did it. Here, the constant backups saved the day. By keeping track of which backups had been built, and their success/failure status, I was able to quickly binary chop, isolate the first backup with the problem, and then create an incremental webrev between it and the previous backup. By definition, the mistake would be in that webrev, and usually easy to spot and rectify.
37154766 libctfgen should hide implementation details
This was work that could have been done with the previous fix (36927818 libctfgen code needs to be brought to library standard), which I deferred to control code churn in what was already a very large fix.
The libctfgen.h header is used by callers of libctrgen, but it also exposed a lot of internal details of the library. This is bad on general principle, but in addition, introduced a lot of unprefixed generic names into the global namespace that could cause problems for calling code.
This fix split libctfgen.h into 2 versions. libctfgen.h remained as the interface for external callers, and all interfaces defined by it were given a ctfgen_ prefix in order to avoid namespace conflicts. A new implementation header, _libctfgen.h was created to hold the implementation details needed within the library, but not by its external users.
PSARC/2024/123 allow ctfmerge -t to be used with -a 37169748 ctfmerge should allow -t to be used with -a
This change involve 2 ctfmerge options:
- -a
- Automatically run the ctfconvert operation on any input object that does not already have a .SUNW_ctf section, and merge the CTF data produced into the output. Unlike using the ctfconvert command, which adds CTF to the input object, the -a option does not modify the input object.
- -t
- Require each input file to have a CTF section
-a is a new option, added as part of the work to make CTF public (PSARC/2024/078). I had initially made -a and -t mutually exclusive options. In hindsight, as I tried to use it, I realized that this was overly restrictive and unnecessary, and removed the constraint.
37326964 ctfmerge during link is impervious to ctrl-C
This was a dumb error I introduced during the libctfgen work that caused the control-C signal handler to not properly end the process, and required a one-line fix. I mainly include it for completeness, but it serves to highlight the risks that come with touching working code, even when one is very careful, and even when the goal (merging multiple copies of the same thing into a single implementation) is worthwhile.
PSARC/2024/139 link-editor should support CTF generation 37264872 link-editor should support CTF generation 37264912 gelf_getsym() locking is unnecessary and too slow for ctfgen 37264999 ctfconvert and ctfmerge -a mishandle archives 37265039 ctfgen stabs reader needs to xlate 37301852 ctfconvert: enum has too many values 37301886 ctfconvert: unrecognized real type size
Following the massive overhaul of libctfgen, the table was set to use it from the link-editor (ld), allowing ld to perform the operations that previously required post processing input objects with ctfconvert, and the output object with ctfmerge. I added a new option, -zctf, which in turn accepts a list of suboptions corresponding to those of the ctfconvert and ctfmerge utilities. -zctf and its suboptions are documented by the ld(1) manpage.
There were multiple interacting pieces to this:
This went reasonably smoothly, and I soon had a link-editor that could add CTF to small toy programs. As soon as I had that working, I attempted to apply it to larger real programs, which predictably resulted in exposing a variety of new issues to grapple with. The first such experiment was an example link from a customer known for making very large objects. This is a good example of code for which applying the postprocessing approach with ctfconvert and ctfmerge would be very hard, as it has an extremely large number of input objects, many of them in archives. In contrast, applying ld -zctf was quite simple. Doing that revealed 2 issues:
As covered in the high level discussion, the performance of gelf_getsym() in a doubly nested O(n^2) loop found in the merge phase becomes a massive problem for large n.
The ctfconvert process uses libdwarf. I found that libdwarf only understands plain ELF objects, but not objects found in archives. This was never an issue for ctfconvert in the past, because there was no reason to pass it an object from within an archive, but it becomes an issue for ctfmerge -a, or for ld -zctf. This needed to be solved, because an ld -zctf that cannot handle input objects in archives would not be general purpose enough to meet our goals for pervasive CTF support.
The basic problem is that you pass libdwarf an open file descriptor to the file, and it is written to assume a basic ELF object, with the ELF header at file offset 0. This just isn't the case in an archive. I may have missed it, but I did not find a libdwarf option that can handle this, and knowing the wide variation in the archive formats, used by various systems, my advice to them would be to stay the course and give the whole thing a wide berth. I solved this by adding code to libctfgen to recognize the situation, and as efficiently as possible, create a scratch file containing that content for libdwarf to examine. This isn't pretty, but archive support is essential, and this holds up, even in the face of that extremely large customer link, which uses many archives, each of which contains many objects.
All debug sections these days are DWARF, but our CTF code contains support for the older Stabs format, and happily, this customer code contained some stabs. This allowed me to exercise that code, and revealed that our stabs-specific code also needed to support byte swapping, a case that had been missed before. I found this while exercising cross linking of these sparc objects on my x86 system.
And so, I worked my way though Userland, getting CTF added to a good chunk of it, but turning it off for numerous packages. It must be said that CTF is not equally important to all things, and we did cover most of what we really care about. And so, I counted it as a success, and integrated the link-editor work. However, hoping to do better with Userland, I held off on integrating that, and put it temporarily aside.
The issue of our lagging support for gcc DWARF versions was probably the biggest issue preventing wider CTF adoption in Userland. Was it time to try and learn more about their DWARF and try to moderize the CTF code that deals with it? That would obviously be a good thing overall, and probably should be done. However, it points at an inherent problem with the DWARF to CTF conversion approach, in that we are always in the position of having to react to changes from elsewhere, sometimes on short notice. CTF conversion is viable, and should work, but was there a better way?
I knew that other systems were also adding CTF to the same sort of code as what we have in Userland, and must have dealt with these issues. Looking into that, I learned more about 2 significant developments.
The differences between versions 2 and 3 are summarized in section CTF Version 2 and 3 Differences below.
The differences between CTF versions 3 and GNU CTF are summarized in section GNU CTF Differences below.
PSARC/2025/013 CTF version 3 37409530 CTF version 3
This project delivered the following:
The work for this touched both libctf and libctfgen deeply. libctfgen has 2 phases, one that reads CTF as input, and creates an in memory representation of the information it contains, and another that starts with the in memory representation, turns it into CTF, and writes it as output. libctf has its own independent code for reading this content into memory. The changes required to support version 3 essentially come down auditing all of this code line by line, and where the versions differ, adding conditionalized code to handle each version specially.
An important decision revolved around continued support for version 2 CTF. It was necessary to continue supporting it for input, to support existing objects in the field. However, we did have the option to drop the ability to create it in new objects. The argument for this is that the output code can be a bit simpler. The argument against is twofold: (1) The actual amount of code that can be removed is modest, and (2) Unused code rots, and without the ability to create version 2, it becomes hard to test. And so, I decided:
To support both versions, it became necessary to rename many of the definitions in <sys/ctf.h> to include the CTF version in their name, and to provide definitions for the each CTF version. These definitions remain non-public implementation details of the system. There is no exposure to code outside of the OS, and the code in the OS that was affected is the same code that needed updating for version 3. The result is that this renaming was straightforward, and did not impose a widespread cost outside of the CTF code itself.
I should note that libctf supports a "CTF container" concept, used by DTrace, which allows the caller to create CTF descriptions dynamically. These containers live only in memory, do not persist beyond the lifetime of the process containing them, and are not written to disk. As such, there was no benefit to continued support for CTF version 2 in this code, and it was updated to only use version 3.
37302044 ctfconvert: failed to get unsigned (form 0xd) 37302110 ctfconvert: func arg 1 has no name
These are both issues in the CTF convert code, found as I rebuilt Userland with the new version 3 CTF support in place, and was able to push deeper. The first is an attempt to read the value from a DW_FORM_sdata as an unsigned value, and simply required a minor revision. The second results from code built by gcc that wants to represent 16-bit, and 128-bit float types, requiring an adjustment to the CTF floating point encoding. Ultimately, the need for these fixes would be reduced by the adoption of GNU CTF from gcc, but it was still useful to fix them, for correctness, and because we continue to support the postprocessing DWARF approach.
37700283 libctf dynamic containers can corrupt memory if parent is present 37820204 Dtrace scripts fail to compile in st_195 with - cannot find type: struct scsi_address*
Bug 37700283 came in weeks after the CTF version 3 work was delivered. The error looked like this:
# dtrace -n 'fbt::kmem_alloc:entry{exit(1)}'
dtrace: invalid probe specifier fbt::kmem_alloc:entry{exit(1)}:
"/usr/lib/dtrace/etrace.d", line 66: cannot find type: struct
cku_private_s*: File data structure corruption detected
Examining the libctf debug log showed clear signs of memory corruption, and given the switch to CTF v3 in the DTrace-only container code, the fingers were pointing straight at my change. The problem was that I was unable to reproduce it on my own system, but eventually it emerged that this only happened when running IDR bits. IDR builds differ from normal ones in that kernel modules are uniquified against a reference base version of genunix. This exercises some different code paths in libctf to manage duplicate type definitions, where older parent genunix has a definition that has been superseded, and which is therefore also found in the child CTF. In addition, CTF version 3 is a bit larger than version 2, so these definitions can use a bit more memory. All of these differences could be part of the root cause for the observed difficulties.
It emerged that the new kernel module CTF had a structure definition with 47 fields, overriding the same type in the genunix that had 40, and when producing the container's copy, the smaller size was being used, while all the new data was being copied in. This is a simple buffer overrun that predates the version 3 conversion. The difference version 3 brings is that the copied data is larger, and smashes deeper. In this case, it was deep enough to have a visible effect.
The second bug (37820204) was essentially the same issue, but in a different area of the same code.
These bugs were fixed, and lesson learned, I started including IDR builds in my testing to guard against being surprised by others.
PSARC/2025/036 Translate GNU CTF from gcc to Solaris CTF 37776953 ctfgen should translate GNU CTF to native Solaris CTF 37577882 ctfconvert: failed to get mapping for tid 355
With CTF version 3 delivered, the next big project was to take on GNU CTF, translating the variant of CTF produced by the gcc compilers (GNU CTF) to Solaris CTF. In so doing, the need to run ctfconvert is eliminated, and CTF produced by gcc can be sent directly to ctfmerge, or to 'ld -zctf'. This enables us to add CTF to a wide range of software built with gcc which was previously out of reach, improving the ability of tools like mdb and dtrace to analyze such code.
GNU CTF differs significantly from Solaris CTF in its container concept and its version of libctf, and none of that is a good fit for Solaris. However, as already noted, the CTF sections gcc produces in relocatable objects are very close to CTF version 3, and translating it to native Solaris version 3 is relatively straightforward.
The following work was done:
Prior to adding the support for translating GNU CTF, when doing Userland builds, some components built with gcc would report an error like the following:
These "failed to get mapping" issues arise when a CTF_K_UNKNOWN type is encountered within a type section during a merge process. The native Solaris ctfconvert utility does not produce this condition, and our merge code did not have a category for it, leading to this error. However, GNU CTF can produce this state, and part of the fix to support GNU CTF directly was to add an "unknown" category to the CTF merge code to accommodate it.ld: fatal: CTF: failed to get mapping for tid 355
PSARC/2025/044 file utility should identify the presence of CTF 37674598 file command could provide debug type
As noted earlier, with increased use of CTF, we found it useful to enhance the file utility to report the presence of general debug data, and CTF, as separate information:
% file /lib/64/libc.so
/lib/64/libc.so: ELF 64-bit LSB dynamic lib AMD64 Version 1 [SSE2 SSE],
dynamically linked, not stripped, no debugging information available,
CTF present37898560 strip: removing symtab should also take CTF
A simple bug fix. When the strip utility removes the non-dynamic symbol table, .symtab, it is supposed to also remove any other sections that rely on it, since those sections would otherwise be unusable. In the case of .SUNW_ctf, this was not happening, a problem that predated this project.
37911743 ctfmerge: O(N^2) check_for_weak() is severe bottleneck
The second half of the fix for the abysmal O(n^2) performance of check_for_weak() in the merge code. This fix employs an AVL tree, created only when needed, to turn the cost into O(n) + O(logN), greatly lowering the impact.
37577916 Userland should use LD_xxx_OPTIONS to add CTF to delivered components
I returned to my Userland workspace, and modified it to make use of the new ability to use GNU CTF directly from gcc -gctf without the need for a CTF convert operation. The result was greatly improved over what I had been able to do earlier, and I integrated it. A large chunk of Userland is now delivered with Solaris containing CTF from normal builds. After the better part of 2 years, this result was anticlimactic in the best possible way.
CTF remains best used with C, and languages at the same language level as C, and we disabled it for lots of C++ components, either due to issues encountered, or an awareness that it wasn't useful enough to bother with. C++ remains an open issue for CTF generally, but not necessarily due to limitations in our implementation of it.
38192852 <subject redacted>
This bug arrived after the change to deliver CTF with Userland integrated. This is not a CTF bug per se, but one that was caused by our attempt to deliver a package from Userland that was built with the Studio compilers, and which points at a limitation of the traditional approach of using the compiler -g option to generate DWARF, which is then converted to CTF.
In this case, the application of -g changed the code being generated, changes that did not affect our ability to build it, but which did cause problems later for other code that wanted to use it. It seems that when -g is used, Studio retains unused code that would otherwise be optimized away. In particular, static inline functions from headers end up being kept. This is a reasonable behavior for the compiler, or at least, one that it is allowed, but one that turned out to be inconvenient when using -g reasons other than what the compiler authors may have had in mind. The lesson here is not that the compiler was wrong, but rather, that when the compiler supplies the CTF directly, pitfalls like this are far less likely. In this case, we temporarily disabled the CTF generation for these objects, and ultimately, switched them over to use gcc, with the -gctf option for CTF.
PSARC/2025/090 Solaris CTF should adopt GNU CTF_K_SLICE 38226071 add support for CTF_K_SLICE to Solaris CTF implementation 35489361 mdb reports the base types for bitfields 35478862 mdb does not display enum bitfields correctly
As the dust settled on the work to support GNU CTF, and deliver CTF with userland, we were in a position to look at lingering issues in the CTF space. The largest such issue was in our handing of C "bit enums", bitfields in structs or unions applied to an enum type, rather than integers. The native Solaris CTF was unable to represent an enum value that is not stored in a 32-bit field. This was seen in 2 different ways:
The GNU CTF_K_SLICE kind is not particularly lovely, and it implies a kind of generality that I think isn't real, suggesting that it can modify any number of kinds, while really only making sense with CTF_K_ENUM. In discussing it with Nick Alcock, I found that he also was a bit ambivalent about it. So I decided to try and come up with a better answer. If I could do something clean and simple that was specific to ENUM, I thought that might be a better way to go. In such a case, ctfconvert could use that new thing, and the GNU support could translate SLICE to it.
I did several prototypes of different ideas for this, including 2 that I took to the stage of working code. I won't detail those efforts in this already way too long document, but all required the introduction of a new CTF kind of some sort, one of which was a variant of CTF_K_ENUM that had an encoding. In each case, I found that they didn't make things simple, particularly for code that calls libctf, which would need modification. libctf is a private library, and its interfaces can change, but we have a substantial code base in mdb and DTrace sitting on top of it, and API changes have large ripple effects.
These alternative designs and prototypes were valuable, as they convinced me that in comparison, SLICE was no worse, and in many ways, easier to handle. The value of embracing an existing solution widely used elsewhere, rather than going down the "not invented here" path, became obvious. I made a final decision, and leaned into adding support for CTF_K_SLICE to native Solaris CTF as a standard feature, equally applicable to versions 2 and 3. The CTF convert code was modified to issue SLICE when bitfield enums are seen, and the GNU CTF conversion code was simplified to remove the code implementing the CTF_K_INTEGER workaround, and simply passed through as part of Solaris CTF.
The changes to libctfgen and libctf were straightforward, but it was still necessary to change how the ctflib APIs operate when SLICE is present, and would require changes to mdb (fairly widespread) and DTrace (fairly small).
Working this out involved a lot of experimentation in modifying mdb, to use various experiments with libctf, and and observing whether the changes made things harder or easier. Ultimately, I was able to limit the necessary changes to callers of ctf_type_encoding(), which is mainly mdb and libctf, with only minor use elsewhere, and did not have to add or remove any APIs, or add or remove any arguments. The main work involved locating calls to ctf_type_encoding() with cscope, and then applying the advice summarized in Adapting libctf Consumers For CTF_K_SLICE.
This fix added the following definitions to <sys/ctf.h>:
#define CTF_K_SLICE 14 /* variant data is single ctf_slice_t */
typedef struct ctf_slice_v2 {
uint16_t cts_type; /* type ID of type being sliced */
uint16_t cts_offset; /* offset of integral type in bits */
uint16_t cts_bits; /* bit-width of this integral type */
uint16_t cts_pad; /* padding */
} ctf_slice_v2_t;
typedef struct ctf_slice_v3 {
uint32_t cts_type;
uint16_t cts_offset;
uint16_t cts_bits;
} ctf_slice_v3_t;
SLICE operates as a modifier, in the same manner as TYPEDEF, VOLATILE, CONST, and RESTRICT. cts_type is a reference to another CTF type, while cts_offset and cts_bits provide an overriding offset and size.
38242531 warning: CTF: unexpected unnamed INTRINSIC(2) tdesc
As mentioned previously, the initial integration of support for GNU CTF did not support their CTF_K_SLICE, and instead mapped them to CTF_K_INTEGER. In doing so, we sacrificed the mnemonic names provided by the enum, one of the reasons for then going back and adding SLICE support properly.
An unintended consequence of that mapping was this bug that came in during the window between the original GNU work, and the later addition of SLICE. Quoting from my analysis of the bug:
The problem is in the process of turning the GNU CTF_K_SLICE kind, which is not supported, into something that can be represented in Solaris CTF. The basic idea is that we create a narrowed CTF_K_INTEGER of the desired width, and give it the name of the item SLICE points at. This mimics the way integer bitfields have long been supported with CTF produced from objects built with the Studio compilers.I integrated a temporary fix for this, but happily it was all swept away when the support for SLICE replaced it. I mention it here only for the obvious moral, which is that simple expedient hacks to large and complex systems sometimes have surprising side effects.The problem, in this particular case, is that the underlying type we turn into a narrowed INTEGER is an anonymous (no name) ENUM. As such, the INTEGER we create also has no name. This falls afoul of a different part of the CTF merge process, in which an "INTRINSIC" (an INTEGER, or FLOAT) with no name is detected. That's not supposed to happen, so the warning is appropriate, albeit inconvenient for this particular case.
38226138 enable use of CTF_K_SLICE to represent bitfields
As with the earlier fix that switched .SUNW_ctf sections to use the SHT_SUNW_ctf section type, it was necessary to seed support for CTF_K_SLICE and wait for the build servers to catch up before enabling it.
38554022 ctfmerge runs out of stack
This is an interesting case that combines a small bug with an algorithmic problem, and one that is not fully resolved. After I integrated the support for generating CTF for the Userland consolidation, it was reported that a new version of ghostscript was causing the CTF merge process to exhaust the stack after running for some time. When building without CTF, the link, though large, finished quickly and successfully.
The stack trace was full of odd repeating cycles, which we were initially unable to explain, but which were eventually revealed to be caused by variables defined as 'int', used by the traverse process, overflowing and wrapping around. Defining those things as 'size_t' cured the cycles, but the resulting link took an inordinate amount of time.
My colleague Vladimir Marek got to the bottom of this second mystery, saying:
> Do you know how the comparison of two 'types' works in
> ctfmerge? First the structure is stored into hash. The
> hash has 8191 buckets. The types which fall into the same
> bucket are arranged into a list specific to the given bucket.
>
> Problem is that all the anon pointers and anon functions fall
> into the same bucket
>
> My test showed that the list in single bucket had more than
> 40000 'types' in the list
This is a general issue for CTF merge, and worthy of thought going forward. However, for reasons already touched on, the value of CTF for C++ is not high, and C typically doesn't have this issue, or at least not at this scale. A fix for for the overflowing integers was integrated. For now, we simply disabled CTF for ghostscipt, and the issue of how ctfmerge should deal with large numbers of anonymous types is left as a problem for another day.
38673244 ctfdump output could be more polished
As noted in the high level comments, sustained use of ctfdump caused me to be come intimately familiar with ctfdump output, and gave me a desire to streamline and organize it a bit more than had been the case. The result is not too different, and a casual user might not notice, but takes some inspiration from elfdump in terms of layout and labeling to make the output easier to read and understand.
PSARC/2026/015 Enable CTF to use the .SUNW_ldynsym symbol table 38845627 libctf should support SUNW_ldynsym symbol table 38870260 mdb should support SUNW_ldynsym symbol table 38912882 Solaris CTF should include type information for SUNW_ldynsym symbols
As the CTF project wound down, the knowledge gained about CTF and mdb internals allowed me to finish work to fully make use of the .SUNW_ldynsym symbol tables that were added to Solaris in 2007.
39048226 unmerged CTF should be detected and diagnosed
When using the postprocessing approach to adding CTF to objects, it is possible to apply ctfconvert to the input objects, but forget to run ctfmerge on the resulting output object. Such objects appear to have CTF, but it is incomplete and invalid. Historically, libctf, or ctfdump, did not detect this case, and would blindly try to interpret the bad CTF. ctfdump in particular would produce plausible looking, but incorrect, output.
Over the years, I have thought about this whenever I came across makefile rules to add CTF. Our make rules were correct, and so this was never more than a theoretical observation. "Doesn't Happen" can be an acceptable answer for an internal tool, but is not good enough for one that is public and supported. This fix adds missing checks to detect and reject unmerged CTF.
39147002 krtld should be hardened against SHF_COMPRESSED ELF sections 39147030 CTF generation tools should support SHF_COMPRESSED ELF sections
As mentioned above, CTF predates the introduction of ELF Section Compression, and as such, has its own built in compression mechanism. For historical reasons, the CTF version is still generally used today for .SUNW_ctf sections, rather than the standard ELF form. However, other sections (symbol tables, etc), may be compressed using the ELF format, and the ELF form can even be applied to .SUNW_ctf, with some effort. As such, our CTF code needs to handle ELF section compression.
The support for ELF section compression in our CTF code was incomplete. I've addressed that with this fix. The issue of using it for .SUNW_ctf sections is left to another day, as I mention in What's Left?.
39693462 ctfmerge and ctfdump should support ancillary objects 39665830 ctfconvert may mishandle gelf_update_shdr() return code 39685356 mcs/strip/elfcompress can botch e_shstrtab from extended ELF header 39692966 elfdump misdiagnoses symbol table with ABSENT string table 39693296 private libelf _elf_zdata() incompatible with GElf APIs 39693395 libelf can set sh_offset/sh_addralign incorrectly for ABSENT sections 39717393 ancillary object checksums are not always unique as required 39739198 ctfconvert can deference NULL pointer for missing type id
The primary fix in this cluster of changes is to properly support ancillary objects from the libctfgen code. The other bugs are things that were discovered while testing that work.
Support for ELF Ancillary objects was added to Solaris in Solaris 11 Update 1:
PSARC/2012/064 ELF Ancillary Objects (Separate Debug Files) 15487884 SUNBT6715267 ld: support separate debug file
Support for ancillary objects was not added to our CTF utilities at that time. This was primarily because the OS itself does not use ancillary objects, and as CTF was an OS-only mechanism, such support was not needed. However, complexity of the CTF code, and a lack of familiarity, were also reasons to defer that work. Later, in 2013, minimal support was added to ctfmerge to support the case of a single ancillary object, intended to work around a specific issue with the Studio Performance Analyzer, and in no way really qualifying as "support". And then, that work was obsoleted by more recent work that allows more than one ancillary to be produced:
PSARC/2022/009 Support for Multiple ELF Ancillary Objects 33756071 link-editor should support multiple ancillary objects
The recently added support to the link-editor (ld -zctf) does support adding CTF to ancillary objects, and libctf does support them, so that gave us some minimal support. However, without the ability to look at it with ctfdump, and with support missing from the ctfmerge utility, this support was badly incomplete. As was the case with ELF section compression, my recent work on CTF put me in a good position to finish off this incomplete corner of support. With this fix:
PSARC/2026/069 Revised placement of .SUNW_ctf section data in ELF ancillary objects 39760475 ld -zancillary -zstrip-class=ctf should work like -zstrip-class=symbol
As I worked on the CTF support for ancillary objects, I noticed that when the ld -zctf option was used in conjunction with -zancillary, that the CTF data was written only to the ancillary object, and was marked as ABSENT in the primary object. This is generally how ancillary objects are intended to work, but it is the wrong default behavior for CTF. CTF is designed to be compact enough to always include in the primary object, and there is a large benefit to always having it available for use by libproc, mdb, and DTrace without also requiring the ancillary objects to be present. The nature of in the field debugging is that one rarely has the option to install additional software first. The tools have to work with what is already present.
There are two broad categories of non-allocable section produced by the link-editor:
Category (2) sections were anticipated in the initial implementation of ancillary objects (PSARC/2012/064), and these sections are given special treatment, as documented by the ld(1) manpage for the ld -zstrip-class option :
Stripped sections are completely removed from the output object. The use of the -z ancillary option alters this behavior with regard to the non-dynamic symbol table .symtab, and the sections related to it. By default, the symbol table is written to both the primary and ancillary objects. If stripped, the symbol table is written to the ancillary object only, and is identified as absent in the pri- mary object. If .symtab is stripped from an object without the use of -z ancillary, the section is completely removed in the usual manner.CTF sections should be treated in the same way, and doing so requires only that the right flags are set for the .SUNW_ctf output section descriptor. The fact that this was not originally done when the -zctf option was introduced (PSARC/2024/139) was a simple oversight, which this fix corrects.
PSARC/2026/071 Controlling CTF compression 39806653 ctfconvert and ctfmerge should give user control over compression 39806766 ld and elfcompress lack a ctf compress class
A previous fix added basic support for standard ELF compression, in CTF, as well as for other sections. This fix builds on that work by adding a new -z option to ctfconvert and ctfmerge that allows the user to control whether compression is done, and whether it is the original CTF style, or ELF. In addition, it adds an ELF 'ctf' compression class to the ELF compression support provided by ld -zcompress-class, and elfcompress.
Over the last 2 years, we've accomplished most of the goals we had for CTF, and then some. The experience gained from working on that code also opened the door to finishing off related work in other areas, notably ELF compression and ancillary objects. Today, the TODO list is short. However, software is never done, and I do have some thoughts about future work that might eventually be worth thinking about.
CTF could still shift to using ELF section compression rather than its own, and perhaps we will eventually do that. However, the old CTF compression support would have to remain to provide backward compatibility for existing objects, so there can be no net simplification from doing so. It may still be worth it, for the conceptual simplicity, and to gain access to Zstd compression. But these are small concerns: ZLIB does an excellent job, and the resulting CTF is small enough that further reduction is not an urgent priority.
Before that can be done, a prerequisite would be to add support for ELF section compression into the kernel runtime linker (krtld). Currently, the kernel only supports the original CTF style of compression.
This section provides a detailed list of the format differences between CTF version 2 and version 3.
The primary difference between the versions is that in version 3, the integers representing types, and some metadata, are widened from 16-bit to 32-bit.
Many CTF data structures are unchanged between versions 2 and 3. Where the structures differ, fields and their purposes remain unchanged, and the differences are all due to the widening of integer types.
The differences between Solaris CTF versions 2 and 3 are:
+--------------------+ | kind | root | vlen | +--------------------+ 15 11 10 9 0
In version 3, it becomes:
+--------+--------+----------------------------+ | kind | isroot | vlen | +--------+--------+----------------------------+ 31 26 25 24 0
The kind field is increased from 5 bits to 6, and vlen grows from 10 bits to 25. The widened vlen addresses most of the limit issue seen when trying to add CTF to objects in the Userland consolidation.
The max size for the short form is raised from 0xfffe to 0xfffffffe, making the need for the large form considerably less likely.
In both forms, the ctt_info, ctt_size, and ctt_type fields are all widened from 16 to 32-bit.
- ctf_member_t
- The ctm_type and ctm_offset fields are widened from 16 to 32-bits.
- ctf_lmember_t
- The ctm_type field is widened from 16 to 32-bits.
Prior to the introduction of support for CTF version 3, the gap between GNU CTF and Solaris CTF was considerably wider than it is today. With the version 3 differences out of the way, what's left is considerably easier to deal with. As noted earlier, the best available reference for GNU CTF is The CTF File Format by Nick Alcock.
The following is a summary of the differences between Solaris version 3 CTF and GNU CTF.
This section provides notes on the Solaris implementation of ctf_type_resolve() and ctf_type_encoding(), and their interaction with each other.
A typical scenario for a CTF consumer is to determine the data type of a given variable, and then, its encoding. The encoding specifies type details, offset, and width (bits), and is part of the INTEGER and FLOAT types. In addition, the SLICE type can be used to add or modify the offset/bits details. SLICE is used to describe INTEGER or ENUM bitfields in struct or union members, adapting the base encoding to fit the bitfield.
Consider the example of a bitfield integer in a structure. The CTF for that field it might look like:
SLICE -> CONST -> VOLATILE -> TYPEDEF -> INTEGER
There can be an arbitrary number of attribute type IDs before the underlying base type. In this example, SLICE, CONST, VOLATILE, and TYPEDEF are all attributes of a base INTEGER type. ctf_type_resolve() and ctf_type_encoding() are used to manage such type chains:
- ctf_type_resolve()
- Given the type ID for the starting type, ctf_type_resolve() returns the type ID of the "base" underlying type, which in this case is INTEGER.
- ctf_type_encoding()
- Given the type ID for the starting type, ctf_type_encoding() returns the encoding for the underlying base type, incorporating the information from SLICE if a SLICE is present.
It should be noted that in both cases, the same starting ID is passed. Both functions employ the same resolution algorithm for moving through the type chain to locate the information they return. Historically, this was not the case. Prior to the introduction of SLICE, standard practice was to resolve the base type, and then call ctf_type_encoding() on that base type to extract the encoding. This worked reliably because the encoding was always found in the base type. The introduction of SLICE changes this dynamic: Calling ctf_type_encoding() on the resolved type, rather than the start type, will provide an encoding that ignores any SLICE that might be present. In this case, the encoding will be too large, and accessing that memory will yield bad results.
Code written before the introduction of SLICE may need to be adapted in order to operate properly when SLICE is present. Existing code that does something like this:
if (((rid = ctf_type_resolve(fp, id)) != CTF_ERR) &&
(ctf_type_encoding(fp, rid, &e) == 0))
should be modified to pass 'id' to ctf_type_encoding(), rather than 'rid':
if ((rid = ctf_type_resolve(fp, id) != CTF_ERR) &&
(ctf_type_encoding(fp, id, &e) == 0))
If rid is not needed elsewhere, the call to ctf_type_resolve() can be entirely dropped, leaving simply:
if (ctf_type_encoding(fp, id, &e) == 0)
That said, there are some situations in which one can safely call ctf_type_encoding() on the resolved type, rather than the start type:
Current manpages for the utilities affected by this project, as they existed when I wrote this blog, are available:
- Text (html)
- ctfconvert(1), ctfdump(1), ctfmerge(1), elfcompress(1), ld(1), strip(1), ctf(5), ctf(7).
- Text (plain)
- ctfconvert(1), ctfdump(1), ctfmerge(1), elfcompress(1), ld(1), strip(1), ctf(5), ctf(7).
- ctfconvert(1), ctfdump(1), ctfmerge(1), elfcompress(1), ld(1), strip(1), ctf(5), ctf(7).
- roff
- ctfconvert(1), ctfdump(1), ctfmerge(1), elfcompress(1), ld(1), strip(1), ctf(5), ctf(7).
FreeBSD ctf(5) manpages covering CTF version 2 and 3.
- Version 2
- ctf(5)
- Version 3
- ctf(5)
The CTF File Format by Nick Alcock
The current Solaris <sys/ctf.h> header, providing definitions for CTF versions 1 - 3.
Robert David was responsible for CTF and DTrace on Solaris in the years before I took on this work. His response to my proposal to walk into his area without really knowing what I was doing and do a massive restructuring of it in pursuit of something that might or might not work out, was to welcome me in and hold my hand throughout the first year as I got going. This included all of the churn to the libctfgen code base to make it library ready, and multiple in depth code reviews for all of it.
Chris Gerhard, our mdb expert, was my constant companion in this work. Chris made me aware of CTF issues I had formerly not known about, and helped me work my way through the massive mdb code base, with its many layers of abstraction. He did that repeatedly as I moved through each stage of the project. Chris acted as a sounding board for all the various attempted schemes, helped me debug and crack issues I was stuck on, helped modify the vital mdb unit tests to keep up, and took on considerable follow on work to take advantage and/or support it. He also code reviewed nearly every fix. He did this for each phase, over the entire 2 year span.
Jan Zaloha took over the CTF/DTrace space from Robert about a year into things, but his evident depth of knowledge betrays a deep preexisting familiarity, and he jumped right into the deep end. Jan has also helped me substantially, as a source of advice, and for code reviews. In addition, he was the front line for any bugs I created (notably 37700283). Fortunately, there was not a lot of this, but when there was, Jan never pointed the finger of blame, and instead was unfailingly helpful, going above and beyond to help diagnose the issues and set me up to fix them.
Alan Coopersmith, Darren Moffat, and the engineers and leadership of the Solaris OS were unfailing sources of support for this work, willing to take the necessary calculated risks to move us forward, and to be ongoing sounding boards for ideas, even the harebrained ones. Their deep knowledge of Solaris history and implementation were essential.
Vladimir Marek, our Userland Gatekeeper (which deeply undersells the scope of what he does to keep the Userland trains running). Though I worked to minimize the impact, my changes to add CTF to Userland still created some churn. Vlad was unfailingly cheerful and encouraging in the face of it, taking the breaks in stride, and some cases even doing a root cause analysis to identify the problem in the CTF tools.
Rainer Orth, for his enthusiasm and support for this project (and so many others), and for rushing to provide gcc -sctf as soon as ld -zctf had stabilized.
FreeBSD, your CTF version 3 was the right thing, and you did it right.
Nick Alcock, and his colleagues in the Oracle Linux Tools Group, the developers of the GNU CTF work, and the gcc -ctf feature we leveraged. Thank you for your help, and for the bug fixes. My discovery of Nick's spec for GNU CTF was a key turning point in this project. Once I made contact, he was unfailingly generous in discussing the hows and whys of what they had done, and in helping me understand how I might make use of it. I asked way too many questions, and Nick took the time to answer them all. Thank you for doing this good work, and thank you for doing it as you did. You were solving your own problems, but in preserving the essential shape and spirit of the original CTF, you made it possible for others with unrelated problems of their own to benefit as well. I'm sure you didn't set out to kick Solaris CTF into wider use for open source, but as the saying goes, a rising tide floats all boats.
Thank You.
| [38] kldd: ldd Style Analysis For Solaris Kernel Modules | [1] Testing...Without...System Into A Brick |