AV-Portal 3.23.3 (4dfb8a34932102951b25870966c61d06d6b97156)

Everything You Always Wanted to Know About "Hello, World"*

Video in TIB AV-Portal: Everything You Always Wanted to Know About "Hello, World"*

Formal Metadata

Everything You Always Wanted to Know About "Hello, World"*
(*But Were Afraid To Ask)
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
I've been working on a new system call ABI and the required runtime support for a C variant with spacial memory safety. Along the way I've encountered lots of interesting bits and pieces required to implement a simple C "Hello, World" program. I found the process fascinating so and this talk brings all that knowledge together in one place. The first example in the classic "The C Programming Language" by Kernighan and Ritchie is in fact a remarkably complete test of the C programming language. This talk provides a guided tour a slightly more complex program where printf() is called with multiple arguments. Along the way from the initial processes' call to exec() to the final exit(), we'll tour the program loading code in the kernel and the dynamic linker, the basics of system call implementation, the implementation of the memory allocator, and of course printf(). We'll also touch on localization and a little on threading support. Where appropriate, I'll discuss portions of the system that need changing to accommodate memory safe versions of C like the version we are developing for our CHERI CPU. This talk will assume some knowledge of a language with C-like syntax (C, C++, Java, and PHP should all be fine).
Web page Suite (music) Slide rule Programming language Dialect Run time (program lifecycle phase) Programming language Interior (topology) Bit Complete metric space Mereology Computer programming Revision control Goodness of fit Oval Oval Statement (computer science) Software testing Conditional-access module
Programming language Suite (music) Computer virus System call Assembly language Length Ferry Corsten Assembly language Maxima and minima Parameter (computer programming) System call Product (business) Number 2 (number) Revision control Writing Kernel (computing) Oval String (computer science) Revision control Software testing Right angle Functional programming Address space
Email Standard deviation Link (knot theory) Computer file Assembly language Computer program Memory management Maxima and minima Computer programming Revision control Programmer (hardware) Fluid statics Object-oriented programming Functional programming Pairwise comparison Internationalization and localization Library (computing)
Randomization Computer file Constructor (object-oriented programming) State of matter Sheaf (mathematics) Set (mathematics) Compiler Mereology Computer programming Number Revision control Array data structure Linker (computing) Different (Kate Ryan album) Object-oriented programming Integrated development environment Process (computing) Functional programming Addition Computer virus Process (computing) Assembly language Run time (program lifecycle phase) Point (geometry) Constructor (object-oriented programming) Computer program Electronic mailing list Bit Machine code Line (geometry) Compiler Pointer (computer programming) Integrated development environment Personal digital assistant Function (mathematics) Order (biology) Funktionspunktmethode Quicksort Fingerprint
Computer icon Slide rule Process (computing) Touchscreen Graph (mathematics) Computer-generated imagery Content (media) System call Machine code Computer programming Revision control Uniform resource locator Kernel (computing) Spherical cap Visualization (computer graphics) Personal digital assistant
Ocean current Implementation Process (computing) Semiconductor memory Binary code System programming Bit Parameter (computer programming) System call Computer programming
Web page Email Email Computer file Mapping Sound effect Mereology Medical imaging Data mining Process (computing) Read-only memory Kernel (computing) Functional programming
Process (computing) Namespace State of matter Structural load Virtual machine Bit Computer programming Medical imaging Process (computing) Virtual memory Integrated development environment Semiconductor memory Linear programming Energy level Functional programming Quicksort Spacetime
Web page Interface (computing) Content (media) Basis <Mathematik> Stack (abstract data type) System call Process (computing) Virtual memory Different (Kate Ryan album) System programming Computer worm Address space Computer architecture
Web page Standard deviation Computer file File format Multiplication sign Sheaf (mathematics) Bit Machine code Instance (computer science) Parameter (computer programming) Variable (mathematics) Computer programming Process (computing) Integrated development environment Vector space Oval Semiconductor memory String (computer science) Initial value problem Address space Spacetime
Point (geometry) Standard deviation Process (computing) State of matter Computer program Set (mathematics) Stack (abstract data type) Parameter (computer programming) Regulärer Ausdruck <Textverarbeitung> System call Frame problem Computer programming Process (computing) String (computer science) String (computer science) Spacetime Right angle Functional programming Address space Spacetime Address space
Greatest element Implementation Overhead (computing) Multiplication sign Set (mathematics) Parameter (computer programming) Mereology Stack (abstract data type) Computer programming Area Workload Facebook Array data structure Mathematics Semiconductor memory String (computer science) Computer programming Computer hardware Software testing Functional programming Mathematical optimization Window Information management Standard deviation Interface (computing) Binary code Debugger Counting Machine code Instance (computer science) Cartesian coordinate system Number Process (computing) Pointer (computer programming) Integrated development environment Personal digital assistant Network topology Computer cluster Order (biology) Hill differential equation Cycle (graph theory)
Slide rule Dynamical system Process (computing) Transport Layer Security Moment (mathematics) Bit Parameter (computer programming) Variable (mathematics) Computer programming Pointer (computer programming) Integrated development environment Vector space Oval Personal digital assistant Computer programming Functional programming Condensation Library (computing) Condition number
Thread (computing) Vector space Data storage device Personal digital assistant Multiplication sign Vector space Right angle Parameter (computer programming) Variable (mathematics) Computer programming Local ring Library (computing)
Web page Email Beat (acoustics) Thread (computing) INTEGRAL Transport Layer Security Sheaf (mathematics) Control flow Parameter (computer programming) Computer programming Pointer (computer programming) Goodness of fit Semiconductor memory Computer programming Vector space Computer hardware Spacetime Extension (kinesiology) Resource allocation Mathematical optimization Condition number Default (computer science) Addition Email Process (computing) Information Mapping Interface (computing) Transport Layer Security Bound state Bit Basis <Mathematik> Machine code Variable (mathematics) System call Pointer (computer programming) Resource allocation Integrated development environment Vector space Data storage device Sheaf (mathematics) Local ring Library (computing) Spacetime
Greatest element Quantum state Ferry Corsten Interface (computing) Multiplication sign Constructor (object-oriented programming) Sheaf (mathematics) Machine code Mereology Pointer (computer programming) Process (computing) Linker (computing) Different (Kate Ryan album) Personal digital assistant Functional programming Library (computing)
Gibbs-sampling Inclusion map Execution unit Polygon mesh Cache (computing) Ferry Corsten MiniDisc System call Local ring
Point (geometry) Default (computer science) Server (computing) Thread (computing) Process (computing) Transport Layer Security Bit Machine code Cartesian coordinate system Flow separation Inclusion map Frequency Integrated development environment String (computer science) System programming Functional programming Local ring
MUD Computer file String (computer science) Multiplication sign Buffer solution Object-oriented programming Machine code Dynamic random-access memory Mereology Fingerprint Local ring Area
Point (geometry) Implementation Run time (program lifecycle phase) Computer file Length Multiplication sign Zoom lens Letterpress printing Function (mathematics) Number Web 2.0 Linker (computing) Deadlock String (computer science) Representation (politics) Integer Functional programming Booting Computer virus Process (computing) Information File format Block (periodic table) Digitizing Floating point Bit Line (geometry) Machine code System call Kernel (computing) Personal digital assistant Network topology Buffer solution Right angle Writing Spacetime
Run time (program lifecycle phase) Software Linker (computing) Ferry Corsten String (computer science) Multiplication sign Projective plane Bit Window
Point (geometry) Run time (program lifecycle phase) Overhead (computing) Computer file Ferry Corsten Mereology Disk read-and-write head Computer programming Programmer (hardware) Mathematics Graphical user interface Linker (computing) Semiconductor memory Different (Kate Ryan album) Core dump Object-oriented programming Aerodynamics Functional programming Gamma function Resource allocation Position operator Exception handling Window Process (computing) Mapping Assembly language Structural load Moment (mathematics) Independence (probability theory) Bit Hecke operator Machine code Binary file Variable (mathematics) System call Symbol table Uniform resource locator Kernel (computing) Process (computing) output Right angle Quicksort Table (information) Library (computing) Row (database)
Run time (program lifecycle phase) Ferry Corsten Multiplication sign Image resolution MIDI Maxima and minima Modulare Programmierung Mereology Computer programming Revision control Mechanism design Linker (computing) Computer programming Utility software Default (computer science) Process (computing) Assembly language Aliasing Keyboard shortcut Bit Machine code System call Symbol table Personal digital assistant Sinc function Library (computing)
Inclusion map Computer programming Multiplication sign Maxima and minima Table (information) Simulation System call Symbol table Computer programming Address space Surjective function
Standard deviation Email Link (knot theory) Feedback QR code Website Online help Bit rate Electronic visual display Rotation Default (computer science)
Inheritance (object-oriented programming) View (database) Graph (mathematics) Computer file Price index Electronic mailing list Parameter (computer programming) Bookmark (World Wide Web) Kernel (computing) Process (computing) Internetworking Internetworking Function (mathematics) Maß <Mathematik> Window Flag
Web page Inheritance (object-oriented programming) System call Graph (mathematics) 1 (number) Set (mathematics) Price index Primitive (album) Parameter (computer programming) Mereology Mechanism design Mathematics Latent heat Structured programming Different (Kate Ryan album) String (computer science) Flag Maize Address space Mathematical optimization Computer architecture Window Default (computer science) Raw image format Zoom lens Process (computing) Inheritance (object-oriented programming) View (database) Online help Computer file Electronic mailing list Bit Bookmark (World Wide Web) Kernel (computing) Process (computing) Sample (statistics) Data storage device Function (mathematics) Kerr-Lösung Convex hull Quicksort
Run time (program lifecycle phase) Ferry Corsten Strut Time zone Grass (card game) Parameter (computer programming) Mereology Computer programming Mathematics Type theory Linker (computing) Kernel (computing) Number theory Core dump Website Source code Logical constant Theory of relativity View (database) Fatou-Menge Computer file Web page Menu (computing) Bit Instance (computer science) Bookmark (World Wide Web) Virtual machine Oval System programming Data structure Spacetime Asynchronous Transfer Mode Software developer Computer-generated imagery Architecture Structured programming String (computer science) Software Computer hardware Navigation Summierbarkeit Booting Window Multiplication sign Rule of inference Interface (computing) Lemma (mathematics) Debugger Memory management Machine code Limit (category theory) System call Inclusion map Pointer (computer programming) Kernel (computing) Error message Integrated development environment String (computer science) Revision control Address space
Zoom lens Multiplication System call Computer file View (database) Ferry Corsten Graph (mathematics) Computer file Coroutine Price index Function (mathematics) Bookmark (World Wide Web) Area Sample (statistics) Personal digital assistant Function (mathematics) Buffer solution output Window
good morning I'm gonna talk to you today about hello world which is a deceptively simple program um just you know it is as
this slide shows the first example in Kerrigan and Ritchie's the C programming language it's on the first page you know it's about the simplest thing you could do or so you might think it turns out actually that talking to several people who have implemented languages and runtimes that printf is usually the last part of the language you implement because it requires all of the rest of the language and in fact a slightly more complicated version of hello world that I'll show you shortly is essentially a complete test suite for C so here it here again is the first example so well to start with we have to correct it a little bit because it turns out the example in knr is not valid C in any modern C dialect so we had to add a couple of void statements but so so that so that's it that's a bit of an interesting thing to get all the way to the to the example many use today I'm gonna add a couple
more things and slightly complicate matters so what what I've done here is I've added a static string rather than just having a product so that we're actually using the varatec arguments functions of printf which is one of those horrible features that no language should ever have at least not one with a register based calling convention but you know all modern architectures do this and this is one of the reasons why printf is really a test suite before that though let's talk about a the minimal version of printf this are of hello world this is the smallest thing you could write in C that admits that omits the string it uses the right system call and just splats it out by length and then it calls exit which is also a system call it's very simple very straightforward and you might think that it would turn
into a piece of assembly code much like this don't worry if you don't know assembly this is the last you'll see in this talk but it is very very simple so we load the one argument into the first argument register we look up the address of the string hello we load it into the second argument register we load the length incorrectly into the third argument register we load the system call number into the system call argument register and we trigger a system call trap this goes into the kernel the kernel does its thing it writes reads the data it writes it out magic happens and then we return and we do basically the same thing again except that we load the return value of zero and the system call number for exit and that's it this is in fact look at the compiled or DLA as a pseudo instruction so it actually does two whole instructions but it's expanded but really this is it so you might think
that's what happens but in fact the minimal C version is about a half megabyte stripped it turns out that this is mostly malloc and localisation despite the fact that you're not using either of those features so you might wonder why that happened you know how did we get all this stuff in a program that doesn't use any memory management functionality at all well first some
hints at that let's look at program linkage so here's how you might link a simple program where you have an object file and you want to turn it into an executable most programmers probably know that this links in the C standard library that's expected but what you might not know is here's the actual link
command quite simplified that's generated by the compiler I've removed all the quoting and pramook I've simplified the paths and I've chopped a few things out that weren't interesting but you notice all these little object files these these are linked in this order because they contain initialization code so here's
the list of them again in a more sorted order so CRT one this is this is the beginning of all things it contains this the underscore underscore start function which is the thing that actually gets run first in any sort of conventional POSIX C program the name doesn't matter but by convention that's always the name and its job is to set up the environment that C expects and then call main in following that there is the there there's CRT I CRT is job is to ensure is to call a number of initializers and set also set up some destructors the things that it sets up are horrible sure just a genuinely terrible idea so there's there's these an underscore in it and underscore Fein a functions what appears in CRT eye is some assembly for a function entry for each of these in a different section then any random object file that's linked later on could contain some other machine code which is concatenated together hopefully everything goes right and no one missile lines the stack on the way through no one corrupts the state of the stack in other ways and all that and then way at the end CRT and add some return from that function and now again hopefully everything went well all that's just sort of mushed together by the linker to create some functions it's lovely then there's CRT begin there's three different versions at FreeBSD they're just differ in what kind of linkage you're doing and they're compiled slightly differently but they all do the same thing they allow they support yet another kind of constructor these are c++ seat or detour constructors there are another set of sections these are at least half way saying in that they are in fact a section which contains an array of function pointers so you just called the functions and that makes that makes sense and it's fairly reasonable and in addition to declaring those arrays they declare some functions to actually call what's in them so that's so the the layout of the array is is not part of the ABI in this case so again CR Tien terminates those they need to be null terminated so each program might contain a bunch of function point crease objects might contain a bunch of function pointers that get put into these arrays and then they're terminated so that that's fine and there's actually two more kinds of constructor and destructor that we'll get to later and these are where all this stuff gets pulled in we're gonna do a walk through the code or through the execution a little bit of a code mostly just looking at the stack and I'll talk about what various things do and and along the way I'll show you where we hit all these things and why stuff gets pulled in that you wouldn't expect a quick note most of the rest of
my slides are going to be screen caps of zoomable SVG's if you look at this URL either one they both go to the same place you can see the actual SPG's and so you can explore on your own if you want to these were generated by taking an instruction trace out of queue and move doing a bunch of processing with awk and then feeding that into a slightly hacked version of Brendan Greg's flame graph program so that you can see stack so I think it's kind of a neat visualization I wouldn't want to try it on a really complicated program but you see it's kind of a neat way to
look at things so we're gonna start not in printf itself but in the kernel at the assists exec ve call exactly easy AAB is to take an existing process and replace all of its contents with whatever program you want to run in this case hello world so let's take a look here the first thing we get is in exact V E is exact V copy in args its job used to allocate
some memory and then copy in the program path and the other arguments that were passed as passed by reference to the exec ve system call it's not very exciting but it is one of the one of the first things that's involved in setting up the the exact call the next thing
we're going to look at is current exec fee this is the actual implementation of exec ve and it's used by all the different system call a vis in freebsd so if you call a Linux binary it does a bit of work to deal with Linux enos and then calls current exec Vee to actually execute the process so the first thing
that happens in current exactly is it looks up the path that you've passed I'm not gonna go into details because I don't understand name i but it's magic you give it a path it finds it it's all good and as George has been telling me
all week names are an illusion so we don't we don't want to know anyway next part is a little function which checks the permissions of the file make sure it's executable by the user who owns the process it seems like a good thing and as a side effect opens the file then
exact map first page does what you might expect it Maps the first page of the program so that it can examine the header and other parts and other things other image activators can look at that header and say is this mine is this is this the something I can run then we get
into the meat of things so this is this trace is on a 64-bit mips machine basically because that sort of been doing all my work and because I have nice instruction level tracing so on there the first image activator we find is the one we want which is the 64 bit elf execution environment image activator and it's so it's it's going to try and load load the program get memory set up for it and then set up register so it can return into the program so the first function we get is exec new namespace
it's job is to soar a new VM space sorry its job is to set up the memory mapping of the process in the initial State
first thing it does is it delete ultimate all the pages from the memory mapping because it doesn't care about whatever the contents were from the previous process that worked the next
thing is that it Maps a stack into the address space this was something that actually surprised me a little I I had expected that stacks might be might be placed on a per per system call interface basis but in fact they're always called they're always set up here they can move around um so when X could put the system could put the stack at a different place but in practice looking at the system it's always the same on every architecture and FreeBSD which is a little surprising so now we have a
stack and an otherwise empty address space so the next thing we're gonna do is we're going to load some sections in the memory into memory so a typical simple statically linked program which is what we're talking about has two sections in the file there's the text section which is the executable bit of code so it's all the instructions and some other non instruction things but it's basically all the instructions that will be executed and then there's the data section the data section is all the static all the static data all the dynamic values or all the global variables all the initialization values for various things and then there's BSS which is all the global and static variables that default to zero and so those aren't in the file that saves space but those pages are mapped at the same time so back into current exec ve the next
step is exact copy out strings this is a bit of an oddity it's actually despite the fact that we're on MIPS it does a bunch of things according to this Co I 3 d6 ABI which happens to define how every elf executable which is the standard file format on every UNIX anyone cares about except Mac OS so it defines how the process is laid out and where to find things so for instance where to find the argument vector where to find the environment director and I'll get get to the how the how this is laid out but what's interesting is it's actually just plopped on top of the stack and why historical reasons but it's a it's an
interesting thing there and then finally there is exact set reg exact set regs its job is to set up the registers in the in the trap frame so that when the system call returns it's as though it returns to where the start process is with the right arguments and I'll talk about what those arguments are when I get to a get to start so here we are
we've returned back from current exec V we go to assist exec V and we get the standard system call return path which is out of scope today and we return to
user space so the recap at this point the stack is mapped into the address space the program is mapped into the address space and a bunch and a bunch of strings and whatnot are set up and then finally the register state is set up to call the start function so now to talk a
bit about what's on the stack there's a whole bunch of things there order is very strict is in is in part strictly defined in part loosely defined the top is PS strings it is used by the debugger and it points by programs like PS to find program names that are too long if not doesn't appear to actually be necessary in modern software but we're stuck with it for existing a B is another oddity cig code is where signals returned where signal hand rollers return to so that they can jump back into the kernel and for some reason we decided to put it here long long ago and then there's a bunch of strings there's the stack canary there's some arrays of things those are accessed by the elf axillary arguments by the environments array or via RV so and the finally the last thing at the bottom of the top set at the stack is the argument count so this is all the things that you need to pass to main in order to run a program with with standard arguments when oddity here is Argosy in c is defined as an integer but in the ABI it's a long so that it's always pointer aligned and so that stack alignment is preserved so lots of weird history here and so I would say what I forgot to tell you in the introduction is the reason that I'm here is that I and looking at all these things and finding all this history is that I'm in the process of writing a new API for a memory safe variant of C with Hardware enforcement so it turns out all this history and all these odd things that you know you you just know you can find them at the top of the stack for instance that's not okay if you want memory safety so we have to take these all these assumptions find them tear them apart so here we are so you end up with the stack pointer at the top of the stack additionally the first argument to start is also that stack pointer and it's used for things I go into this again but it's used to magically find things on the stack so this layout is all part of the application binary interface and it's part of how the process works so now we're now let's get into start
which one thing you might observe looking at this is that most of the most of the arguments have these je or most of the functions have je something under them that's because in fact this program spends almost all of its time in Mallik like something like 90% 95% of cycles in hello world are spent in Mallik so part of that so most of that is because je Malik is a Malik designed to support giant multi-threaded programs you know one of the optimization test cases in the first case was Firefox these days it is whatever Facebook's PHP workload looks like so it is a big heavyweight implementation designed for real problems so you know it doesn't really matter hello world's not a performant thing you know you don't really care how long it takes but it's interesting that there's all this overhead and in fact for some program if you wanted a trivial program like this to be super fast you would definitely want to use its simple malloc and actually one of the things we found in our research is we were using a trivial malloc in some of our benchmarking realized we were doing a lot better than we should be there should have been more overhead compared to the unmodified environment it turned out oh right we hadn't turned jml lock on yet so we made that change in our tree recently so let's take a look at
start I've condensed start onto two slides by stripping out all the bits about dinah all the the conditional cases for dynamic programming and or dynamic libraries and all the conditionals for profiling so it's just the minimal thing so first up we're we're setting up the argument vector so we have AP which is you recall is that pointer to the top of the stack we know that that is in fact a pointer to Argosy so we we get Argosy out of there and then we say Oh we know the next thing that's RV and then we know that the environment is just past that plus a null pointer and we and we buy Arg C and we'll get later you know in a moment to an even more bizarre example of finding things on the stack the next thing that we do is
there's this function handle art v its job is to set the environment the the environ variable so it sets it to the pointer to the argument array and it sets the program name variable it's kind of a minimal thing next up is a knit TLS
so this is this is in fact why that example program with just right was so bloated turns out that we need to set up thread local stories and we do this unconditionally we need to do it in all cases because we don't know whether or not we'll be threaded so in a static program we probably could avoid it but in a dynamic program we must always set up TLS because we might load a library later that uses that uses threads so we need to have thread-local storage setup so for those who don't know thread-local storage is global or static variables which are per thread so each thread has its own copy of that variable you know again unsurprisingly most of the time here is spent in malloc so let's let's
talk about what an it TLS does so the first thing it does is it finds the elf exhilarate arguments vector it does that
why this cute bit of code which is it takes the environment array and it knows that the environment array is null terminated and it knows the elf axillary arguments vector has just passed it so it walks right off the end of the array on purpose and then it says oh that's the elf axillary arguments vector good I'll just cast it use it it's great so
it uses that to find the program headers so one of the many things in the elf excel arguments in addition to things like the stack canary and the array of page sizes and whatnot he's a pointer to the map seen programs program headers so that's the thing that tells you what all the sections are in the program in the program's binary it uses that to find the PL TTLs section this section contains the default values for all of the tls variables that aren't 0 by default so you might have a you might have a TLS variable that's one because you want to do something you know you want a condition to be true that will be stored in here and that's the beginning of the TLS section is all those things so that can just be splattered right over the top once it has that information and it's stored it so each thread can use it in the future it calls this Lib C allocate TLS which allocates space and initializes values is necessary and this is where je malloc comes in so i needs to allocate space and that space needs to be Fria bellator by calling free because you might start another thread and killed the original one so you can't just leak that and you you don't want to have to record that and maybe I'll replace you know you don't want to record that this one's special in some way so what happens is there's a special interface into je malloc which doesn't use thread local storage and which does all the initialization bits but it avoids all the things that haven't been set up yet because as I said J malloc is designed for a threaded environment so of course it needs per thread pools of storage and whatnot so it makes use of TLS and finally the TLS pointer is set on a per thread basis on MIPS this is calling a special ciserek syscall which says set the TLS pointer to this value and then when you want to use it you call this this call again usually via a instruction that traps that generates a trap because it's typically unimplemented and you you get the value back in cherry bsd we actually have implemented an optimization so we so we call this this call to set the value but then we set a user local register which is a special register readable by the by the program or readable from users base yes oh sure so so cherry beat cherry is our is our processor with memory safe extensions and cherry BSD is our fork of FreeBSD I talked about it last year so I think there's probably a talk online if you're interested it is basic so it adds bounds checked pointers so we can do all the things that a SLR and and control flow integrity except we can do them enforced in hardware and fast so yeah so we've implemented some optimizations that one we could probably merge back to FreeBSD I think it works on the edge rather light so we probably should so back to
back to start again we've we've initialized TLS and now we get into
static a handle static and it I'm not gonna go dive deep into the code because it is a chain of boring and hard to read code but and it basically this is where all the initial all the constructors are called and where disruptors are registered so as I said earlier there are four different kinds because people added them at different times and wanted different things so admittin fina were the first things historically and i believe a knitter ray was added next so that's an array of function pointers that's the bottom one that runs last and it's called it's a sensible interface more or less and then I assumed that people just had oh but we need to run some things early you know some libraries set up stuff that might be called by other initializers so we got pre init array just you know because why why not why have a consistent and sensible interface when you can have something dumb actually what's really going on here is it's hard to get all the world to change their linker and to add new features to all the linkers in the world so therefore you have to write something that works with the old cruddy linker you have not the new shiny linker you would like to have and then you encode it into the ABI as is done here you make it part of all the standard interfaces and then you're stuck with it forever and ever I think we can kill an it in Phoenix we're doing it in cherry BSD we're gonna see what happens and there's also this the C++ constructor is the seaters section and that's weirdly in FreeBSD I think actually due to a new thing called through in it and that's the only thing that gets called through in it that's a trivial fix but I haven't done it in any case Oh what all these do is they call a bunch of is essentially they call a bunch of functions which which set various things up the thing that actually matters here is that in the process the destructor the finalized destructor which calls Fein a and which calls there's a Fein a array at exit get set up and then finally we're ready to
call main in FreeBSD is start that is main and whatever main returns gets passed to exit and the exit system call gets run so we'll get into main now okay
so here it is let's see and I can barely read it so again you know it is mostly malloc so and in fact although it is
also mostly VF printf BF printf is called by printf and down here and then the F printf starts doing some work it's
the first thing it does is it has to get the locale which is how it can look up
things like which decimal point is used and we'll get we'll get to the that is in fact the reason it's used in this case but get low-cal is a bit of has a bit of interesting machinery to it so locales are per thread because you might want to write a server where one person connected is you know country where the the decimal separator is a comma and another one in a period I don't understand exactly why you would want to structure your application this way but this is in fact a something you do so locales are per thread so we look up the locale then we call this once function which is a interesting interesting bit of machinery its job is as you might expect to do something once either once per thread or once per execution so you can look up a value and then you can save it if and and so what it's doing here is it's trying to find the locale path it looks at the environment tries to find the path in the end it's not set in the environment so it uses the default so all that work to copy a string but it's very general and that's
that's one of the the amusing things about looking through printf is printf is trying to solve is is you as you walk through all this code is trying to solve the whole big problem and that's why it's such a big beast to make to system calls so back into VF printf now let's
look at underscore VF printf underscore VF printf we get here bypassing the locale that we got before to VF printf L which takes an explicit locale and then we call VF printf which does all the actual work of printing a string so I'm
going to ignore all the stuff on this side because what this is is it calls into the file object for step four standard out and it looks at it and says oh you're an out you're a buffered file and you don't have a buffer yet I guess I better allocate one and so that's what all this code is is is basically allocating an array so and I don't have time to get into J malloc so we won't do that now this is the part
that's kind of that that's the actual meet of printing first this was a bit of a surprise and I think needs to be optimized out printf no matter what it's printing what's passed to it always spends a bit of time here finding out what the decimal point is on entry because you know you might print a floating-point number so you should look that up next is a very fairly simple thing as you recall the string that we were parsing was a was a percent s space % d /n so a string a number and a and a bit of white space so the wavy F printf works is for each block of data up to a format code you know it assembles a a buffer of things to print out and then flushes those out at the end so in this case it first finds the percent s string format so it puts that on the string in it in here it looks up the the length of the strings that's that in the buffer here it adds a little bit that says okay there's gonna be a space and then it says oh there's gonna be a number and it calls this function whose name I can't quite remember whose job it is to write to us to write out the representation of the integer string one thing that's that I found was interesting when I was reading this code in modifying it for some other reasons is it actually works backwards so it starts at the end of the buffer of the temporary buffer and writes so it writes the first digit or the least significant digit the next the next the next which is an easy way to do it and I thought that makes a lot of sense and then I stumbled across three other implementations of printf in the tree all of which do it the other way in fact we have at least four implementations of printf in the tree the runtime linker has its own web c has its own there's one in the kernel there's I think one in the boot and I'm pretty sure there's some more all right J Malik has one too because printf is you as you've noticed uses Malik and J Malik wanted to have a printf so that they could print debugging information and then you know deadlock is bad so so there's a copy of for a deaf in there and then we finally get to the last bit so there's the the new line at the end and we'll zoom in on that because that's where the actual
write system call goes so we have this single single buffer to be written out at the beginning when it's scanning the buffers it first copies them all into one place and then it looks at them and says it scans them all and says oh is there a new line anywhere in this output because if there's a new line and it's a buffered file descriptor then we need to actually call right when we hit that new line so it finds the new line which means that this fflush call gets made later that flush in turn does a bunch of
things digs down digs down and digs down and finally we write the string so there
we are now that's the basic static hello world I said in my abstract that I would do dynamic hello world so I will talk a little bit about dynamic hello world
unfortunately I was assuming that I would have finished modifying the runtime linker for our our project to support our new our new ABI and it being a software project I didn't get it done in time so I don't actually understand runtime linking yet but I understand the basics so I'll give you a quick tour of it oops wait I'm sorry I am not there yet I have one more tiny bit we're back in just we've returned from Maine we're back into start we get to exit and exit
isn't actually called the system call directly it's a little more complicated than that inside exit is a function it's job is to call destructor so this this C T X or this CX a finalize is a func function whose job it is to call all the all the destructors that were registered through the ad exit function so finalized is one such one such destructor so that gets called that calls Fein a cleans up all the does a bunch of D allocation and clean up in practice it doesn't really do much the next part is there is this cleanup function which could use a more descriptive name it's what it actually is is it's a function that's called only if a a C file object was used during during the course of the program and it's used to flush out any remaining i/o and then finally we call the exit system call so okay now dynamic libraries so
here's a different trace what you might notice is that you can't see where main is main is this little bit on the far far right so for those in the recording it's a little bit over here so dynamic library dynamic linking has quite a lot of overhead there's also a few changes in the process layout so the first so we'll start here at the beginning this is the same memory layout that you saw before you see the program on the left you see the stack on the right and that's the kernel loads of the program just the same way except that it also loads the runtime linker the runtime linker and the other the other difference in the in the execs in exec ve is that it jumps into the runtime linkers entry point rather than into the program the first thing the runtime linker does is it calls this long name which as you can tell it's just if you look at it you'll notice it includes a relocate itself that's because the runtime linker is position independent code and it needs to be have its variables updated and be relocated to it it has a bit of slightly horrible code that make sure that wherever the heck the kernel decided to put it it can run huh all right let's see it's not it's at least see walks in array it does some things it's not you know it's not a big pile of assembly or something so once that's run and and the programmer is relocated it starts parsing the program itself it notices that the program depends on Lib C so it loads Lib C and it does an initial relocation of Lib C sets up all the global variables and then sets up a table of of entry points for each for the program to call into the symbols it needs and we'll get to that in just a moment so we have let's see we have Lib C late yes so live C is now now loaded in one one thing about the about our Tod is the kernel loads it as though it were loaded by by the EM map call which Maps a file and so it it picked attempts to pick the same location in the kernel not that seems sort of gratuitously pedantic but that's what the kernel currently does one thing I did notice when I was doing my initial tracing is that the location it chooses is very poorly aligned and so it made it very hard to understand what the runtime linker was doing when I was look so looking at an object dump file and looking at disassembly because everything was shifted in fairly low bits so in cherry BSD I've moved it so it's aligned at 16 Meg's or something and it's much easier to do math in my head and then Lib C gets mapped by ax and map the normal way at the end so then so once Lib C has been initially relocated we call into start again so you can see way way way off to the right
and this I think looks fairly familiar you have main here the stack trace is a little screwed up because I didn't get my too fixed in time so at Exit does exit and it's not part of the stack forever but what you might what you'll notice here is that there are these are TLD bind starts after things like printf which has this funny name so let's let's take
a look at that so you see here is that printf has this this long name which is / def and then at at this FBS do9 that's says that this is it's looking for version 1.0 of the printf symbol it's looking for somebody to supply that in this case lib see that's in the symbol versioning mechanism which is why we haven't bumped had to bump the version of Lib C since freebsd 7 which means your programs keep working unless they do something like use lib KVM or a Lib util which are hard to version so there's the symbol and then when calls are made between modules in a dynamically linked program they defy default what gets called is this this little bit of runtime linker code rather than so that so the symbols not result not pre resolved this is because Lib C has a gazillion symbols and they take a fair bit of time to resolve and and to look up so it'd be it'd be very expensive on program start if we took all several thousand symbols and started them up you know as you you know you see here the whole runtime you know there's maybe room for 20 of them in that total runtime so you wouldn't want to make do all these resolutions so you call this little assembly stub here and it's job is to say oh you're calling you're looking for printf so it goes in and it
wanders around in the Lindsey symbol table does a bunch of lookups finally finds the symbol and then it returns and
it sets up the table so now the table includes the correct address of the symbol so future calls not that anything calls anything twice in this program would will go direct rather than having this lookup occur it's called lazy binding so that's basically the same
otherwise the programs are the same it's just that any time there's a cross module call there's one of these look at otherwise dynamic dynamic for a death is our dynamic hello world is is pretty much the same so that is the end of my talk as I yeah I'd
happy to take questions I would like to request that you give me feedback either by email or this is a link to the conference website that's anonymous as far as I know and the QR codes for that too so let me know was the talk helpful interesting did it make sense what would you like to learn more about maybe you wanted to learn about jay malik maybe there was something you wanted to learn less about something that was dreary and boring this is a big messy thing some of its kind of that way so I'd be happy to take any questions you might have
so let's see so that the question is big how does the how does the argument copying from the initial process into the kernel and then back out work so yeah so that that allocate arguments let me pull up to the zoomable F SVG whoops
I've turned the internet back on they were sending me messages please
we want the colonel one okay so it so
exactly copy in args it allocates some kernel storage so it looks at the argument list and allocate some storage in the kernel and copies in all the strings you know using it using the usual copy and primitive and then after the address space is reset the
so after the address space is reset the
there is a stack and that's that's already mapped that's just anonymous pages with the stack flag which says they grow down rather than up this uses optimization and then exact copy out strings this job is to take those things we've copied in and if destroyed in the parent process and to copy them out and it so it knows that whole layout so it knows PS strings is at the top and it sort of works its way down just copying out through the usual copy out mechanism yeah that's true so I think it could I don't know yeah so yeah you probably could in fact do that it would make the setup more complicated well yeah I mean this is quite a bit yeah I don't I don't think the copy outs very expensive you don't lose very often but I mean I think you certainly could do it the problem is that you would end up change the structure of copy out strings too you would have to compute how much it needed so you'd have to move who is responsible for creating the stack which would be fine but you'd have to change that and then you'd have to have something that determined how big that set of strings was going to be and then it's more than that though it's it's a bunch of ABI specific things so you could I think if you reordered everything a fair bit you could do it and I think part of the problem right now is there's quite a bit of plug ability so so exact copy out strings is the default for the architecture but different ABI is have different ones so you could do it but you'd have to refactor all of the compat layers so be an interesting experiment to see if you've got better performance or enough better performance to to deal with the overhead oh sure I got one over here I'll take it what is in PS strings um well let's go take a look I think I remember the name
yeah so it is it's a pointer to one or more argument strings and the end to the environment it's used to update so it's it's primarily used for instance when you call set proc title to change the name of the program you then need to point to a potentially completely new Malick's set of arguments it's a little gratuitous in that at least for for the arguments because you also called a curdle and say please set this historically at least this was it was done this way so that there wouldn't so you wouldn't have to allow the kernel to the kernel wouldn't have to accept an arbitrary arbitrarily large title in practice there doesn't seem to be an actual limit on the syscall that's our the on the sissy TL that's used to set it so in practice I don't think it matters so in Cherry avi I think I'm getting rid of this because it the interface has a bunch of other problems but it's yeah so it's it's just pointers to these once they've changed so the debugger can find them part of why I say it's a bit gratuitous is that you could simply pass this same stuff to the kernel because you only really care about it if you're writing out a core dump or doing some debugging so if the colonel knows where these things are it's fine it can write an elf note in the core dump and you can look it up there rather than sticking it at the top of the stack and hoping no one steps on it you had another question so what am I going to kick off the stack in cherry BSD in in Cherry a bi which is the the the alternate mode where all pointers are our hardware pointer hardware protected capability pointers so what I am planning to evict everything from the stack none of this stuff has any relation to the program stack so I'm gonna kick it all off right now I'm currently writing it to that memory allocation because I guess to the further reason that I said you know it'd be harder to check there would be a bunch of work to change how things are done it's easier to just splat it to the stack and then I actually restrict the stack pointer so the stack doesn't include that stuff the pointer that's actually accessible to the program doesn't include any of those things so I plan to kick all that stuff off basically and eventually it'll just go somewhere else so yeah exhilerated go and I don't want I don't really want them to be accessible from the rest of the program so I don't want there to be a global to access them I want you to have to get them through the runtime linker which will be able to compartmentalize so the runtime linker can become a trusted linker and loader I thought about changing the way we found two PS strings but caustic really didn't like it so we're not doing that what else are we doing um not a whole lot of the API stuff I guess we are I think so on cherry API the way we find all those auxilary arguments and stuff is there is a structure that's the interface between the kernel and user space so we pass a pointer instead of passing a pointer to the top of the stack we pass a pointer to a structure that has an Arg V pointer and an Arg C and an NV pointer did we end up doing that forearm v8 okay sorry yeah so I think some of that disassociation should just be done and there's there's actually no reason any of this should be at the top of the stack it's just where it is so but for the most part the big the big change was cherry API is going to be that you don't have things next to each other that you can access access one from the other that's not valid C and is something we can we can prevent so we need to actually actually pass proper pointers to things rather than just knowing that things are next to each other then it occurs in the exit code yeah so let's go
back yeah it's the cleaner the cleanup
routine does the flush in that case so
so yeah way way way off at the end the end of exit because you have ever done any i/o on that file descriptor or attempt to do any i/o on a file descriptor or perhaps a buffered file descriptor that cleanup routine is registered and therefore it does the work so that's why you get this silly interview questions with you know multiple outputs and stuff is it walks the descriptor array and I think flushes things that are that have have data buffered yes so yeah it calls it says is there anything to do well thank you all for coming