Unikraft: Debugging and Monitoring
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 287 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/56915 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 202298 / 287
2
4
6
8
12
17
21
23
31
35
37
41
44
45
46
47
50
62
65
66
67
68
71
73
81
84
85
86
90
92
94
100
102
105
111
114
115
116
117
118
121
122
124
127
131
133
135
137
139
140
141
142
145
149
150
156
164
165
167
169
170
171
172
174
176
178
180
183
184
189
190
192
194
198
205
206
207
208
210
218
220
224
225
229
230
232
235
236
238
239
240
242
243
244
245
246
249
250
253
260
262
264
267
273
274
277
282
283
287
00:00
Projective planeDisk read-and-write headOpen sourceImplementationInformationServer (computing)Web pageInstance (computer science)Point cloudOnline helpExecution unitDiagramEngineering drawingComputer animation
01:08
Address spaceCurvatureSingle-precision floating-point formatKernel (computing)Binary fileDevice driverInterface (computing)Resource allocationStack (abstract data type)Computer networkRead-only memoryIntegrated development environmentMeta elementPhysical systemPrinciple of maximum entropyLibrary (computing)Block (periodic table)System callBroadcast programmingServer (computing)Directory serviceKeyboard shortcutLevel (video gaming)Process (computing)CASE <Informatik>Connectivity (graph theory)Computing platformInterface (computing)Device driverComputer architectureRun time (program lifecycle phase)Mechanism designPrimitive (album)SoftwareCartesian coordinate systemLibrary (computing)Computer fileModal logicBinary codeAddress spaceSemiconductor memoryKernel (computing)Mathematical analysisNeuroinformatikService (economics)BitSlide ruleCuboidIntegrated development environmentFormal languageVirtualizationSingle-precision floating-point formatMikroarchitekturOperating systemUnit testingLetterpress printingÜbertragungsfunktionDirectory serviceServer (computing)Functional (mathematics)Network socketBuildingMathematical optimizationPosition operatorInstance (computer science)Network topologyHard disk driveSpecial unitary groupComa BerenicesPRINCE2Schmelze <Betrieb>Observational studyProduct (business)Bit error rateForm (programming)PlotterMultiplication signPhysical systemDampingView (database)Group actionComputer animation
06:36
Library (computing)Interface (computing)CompilerString (computer science)Visualization (computer graphics)System programmingDisintegrationOverhead (computing)ComputerParsingArchitectureDirectory serviceFront and back endsInternet service providerInstance (computer science)Product (business)Data typeStack (abstract data type)Physical systemFront and back endsRun time (program lifecycle phase)Interface (computing)Directory serviceBuildingSet (mathematics)ImplementationMultiplication signNeuroinformatikLibrary (computing)Latent heatContext awarenessINTEGRALOverhead (computing)Gastropod shellCommunications protocolVisualization (computer graphics)SoftwareWorkstation <Musikinstrument>System callPhysical lawComputer architectureOrder (biology)Point (geometry)Computer animation
08:50
Witt algebraCommutatorSubsetLibrary (computing)Set (mathematics)Instance (computer science)Process (computing)Metric systemMacro (computer science)Range (statistics)Front and back endsSoftwareINTEGRALRepresentational state transferBuildingComputer animation
09:28
State of matterInternet service providerFront and back endsGastropod shellBefehlsprozessorData storage deviceRead-only memoryComputer networkModal logicRepository (publishing)Utility softwareLibrary (computing)Front and back endsSet (mathematics)Macro (computer science)Metric systemProcess (computing)Decision theoryData storage deviceSoftwareComputer animation
09:58
Crash (computing)Uniform convergenceInformationNetwork socketEmulationDebuggerOpen sourceLevel (video gaming)Single-precision floating-point formatComputing platformMeta elementPoint cloudHypercubeThread (computing)DisintegrationKernel (computing)TelecommunicationEvent horizonPhase transitionBootingProcess (computing)Library (computing)ImplementationMessage passingComputer networkPhysical systemRead-only memoryResource allocationVideo game consoleSerial portAddress spaceCommunications protocolParameter (computer programming)BefehlsprozessorComputer wormNumerical digitMilitary operationServer (computing)Front and back endsComputer configurationArithmetic meanDebuggerError messageCondition numberSingle-precision floating-point formatSerial portMessage passingSpeicheradresseMemory managementVirtual machineBootingProper mapVideo game consoleOperating systemData structureConnected spaceProcess (computing)Level (video gaming)Regular graphTask (computing)Set (mathematics)Physical systemInformationGame controllerOnline helpComputing platformShared memoryParameter (computer programming)Function (mathematics)Tracing (software)Point (geometry)Heat transferTouchscreenSoftwareTelecommunicationCommunications protocolINTEGRALMultiplication signCrash (computing)Loop (music)Client (computing)Disk read-and-write headSummierbarkeitState of matterVirtualizationKernel (computing)Sign (mathematics)CASE <Informatik>Control flowFile formatInternet service providerNetwork socketOpen sourceSemantics (computer science)Functional (mathematics)BefehlsprozessorException handlingSystem callEvent horizonBuildingSoftware bugSoftware developerLibrary (computing)Execution unitFamilyBus (computing)Symbol tableNoise (electronics)Uniqueness quantificationTransport Layer SecurityKey (cryptography)Medical imagingMetreSound effectElectronic mailing listOntologySemiconductor memoryComputer animation
16:19
TelecommunicationRead-only memoryAddress spaceCommunications protocolParameter (computer programming)BefehlsprozessorComputer wormNumerical digitMilitary operationString (computer science)Client (computing)DisintegrationArchitectureImplementationCodeDebuggerIndependence (probability theory)Library (computing)Event horizonInterface (computing)Link (knot theory)Image registrationCrash (computing)Physical systemIntegerAbstract state machinesPointer (computer programming)Asynchronous Transfer ModeException handlingSystem callAnalytic continuationCodeComputer architectureDebuggerSemiconductor memoryInterpreter (computing)ImplementationFunctional (mathematics)Computing platformTable (information)Type theoryWeb pageInterface (computing)Revision controlLibrary (computing)Crash (computing)Connectivity (graph theory)Connected spaceFehlererkennungData structurePhysical systemNumberSpeicheradresseGeneric programmingEvent horizonException handlingComputer configurationIndependence (probability theory)Communications protocolAddress spaceClient (computing)String (computer science)Parameter (computer programming)Analytic continuationOcean currentPointer (computer programming)Link (knot theory)LengthImage registrationDependent and independent variablesElectronic mailing listKernel (computing)Operator (mathematics)BefehlsprozessorLine (geometry)MereologyWritingPoint (geometry)Control flowReading (process)IntegerAssembly languageMultiplication signFlagHexagonEnvelope (mathematics)Musical ensembleMultiplicationDesign by contractMoment (mathematics)Right angleState of matterUniform resource locatorTracing (software)Phase transitionUniformer RaumArrow of timeAsynchronous Transfer ModeMixed realityData storage deviceContext awarenessState observerConstructor (object-oriented programming)Software bugInsertion lossDampingFerry CorstenDressing (medical)Sound effectCoefficient of determinationGraph (mathematics)TrailComputer animation
22:26
Scheduling (computing)ArchitectureCodeUniform convergencePhysical systemCrash (computing)Symbol tableImage resolutionLink (knot theory)Run time (program lifecycle phase)Interface (computing)State of matterThread (computing)Computer hardwareComputer networkStack (abstract data type)TouchscreenArmInformationComputer architectureMultiplication signCrash (computing)Symbol tablePoint (geometry)Tracing (software)Data storage deviceUniformer RaumThread (computing)Link (knot theory)2 (number)Address spaceDemo (music)SpeicheradresseDifferent (Kate Ryan album)Process (computing)Table (information)Binary codeSpacetimePointer (computer programming)System callAdditionImage resolutionMathematicsPoint cloudSoftwareIntegrated development environmentComputer hardwareConnected spaceCodeExecution unitStatisticsDressing (medical)WhiteboardCausalityWave packetVideo gameCuboidComputer animation
25:05
DemosceneDebuggerInformationThread (computing)Type theoryInterface (computing)Pointer (computer programming)Directory serviceOvalEvent horizonSymbol tableConnected spaceServer (computing)Group actionControl flowState observerEvent horizonVapor barrierProcess (computing)Video game consoleConfiguration spacePoint (geometry)MetreException handlingFunctional (mathematics)Virtual machineUniqueness quantificationSingle-precision floating-point formatMedical imagingFunction (mathematics)Expert systemCuboidVideo gameVariable (mathematics)Web 2.0BootingSubsetOpen sourceDebuggerParameter (computer programming)Kernel (computing)Computing platformSet (mathematics)Line (geometry)Computer animation
27:27
RippingSoftwareInformationDirectory serviceFreewareGEDCOMBootingHost Identity ProtocolGamma functionConstructor (object-oriented programming)BlogComputer fileReduction of orderGame theoryData typeRevision controlColor managementTwin primeCrash (computing)Data storage deviceMetreAddress spaceGreen's functionPhysical systemWeb pageComputer architectureRevision controlCore dumpExpert systemSemiconductor memoryException handlingDressing (medical)Game controllerFunctional (mathematics)Right angleSoftware bugCondition numberInformationCASE <Informatik>TouchscreenDebuggerSystem callThread (computing)Parameter (computer programming)Web 2.0Local ringServer (computing)Error messageSource code
29:11
Video GenieCodeTwitterNetwork topologyOffice suiteProjective planeOpen sourceWeb pagePoint cloudAddress spaceOnline helpSlide ruleHome pageComputer chessDecision theoryComputer animation
29:55
Slide ruleWeb pageVisualization (computer graphics)INTEGRALComputer animation
30:21
INTEGRALVisualization (computer graphics)Kernel (computing)Metric systemPhysical systemComputer animationMeeting/Interview
30:50
File systemStack (abstract data type)Software developerGraph (mathematics)Grass (card game)Standard deviationLine (geometry)CodeMeeting/Interview
31:21
Computer architectureLatent heatContext awarenessPlastikkarteMereologyHypermediaFunctional (mathematics)Single-precision floating-point formatMeeting/Interview
31:51
Game controllerInformationBefehlsprozessorBlock (periodic table)State of matterCore dumpMeeting/Interview
32:27
Electronic mailing listMiniDiscAlpha (investment)Binary codeStress (mechanics)Game controllerBlock (periodic table)Scheduling (computing)Medical imagingComputing platformKernel (computing)BootingStandard deviationForm (programming)Meeting/Interview
33:25
Computer animation
Transcript: English(auto-generated)
00:12
Hello and welcome everybody. My name is Simon Kunze and I'm the CTO of Unicraft UT. It's a commercial open source company that we're currently launching
00:21
and it utilizes the Unicraft open source project. With me, there's also Mark Vittinghaus who joins us as the head of engineering. Because I'm not sure if everyone in the audience is familiar with what we are doing, I will first give you a short and brief overview of the Unicraft project. I will then directly jump into our monitoring debugging features
00:42
and Mark will present a deep dive into our GDB server implementation. There's one thing I also want to mention here. In case if you're interested in trying out Unicraft for your company and want to understand the benefits, we'd like to ask you to contact us so that we can offer help.
01:00
We have also more information online on our company webpage on unicraft.io. What we do with Unicraft is building unikernels and we call ours even highly specialized unikernels. But what's that? So let's take a look at this picture. So if you have a, let's say, a service running somewhere,
01:23
let's say in the cloud, for instance, you will run it on a common shared platform, maybe a hypervisor environment where you have a kernel deployed and then on top, there's your application with your application libraries. What we do is taking that application
01:46
and simplify the case that we want just that service running and nothing else, no other applications, no other tools, whatever that comes with a standard OS.
02:00
So with that, we can build a flat and single address space, which allows us to take kernel components directly to the application. So we get a single monolithic binary with only the necessary kernel components. And then we get advantages from specialization.
02:22
So by optimizing the kernel components to the applications and optimizing the kernel components to the platform you're executing, you get performance and efficiency benefits. You get also a small trusted compute base and smaller memory footprints and fast boost times,
02:43
which can be really interesting for some use cases. We have the following design principles in Unicraft. Specialization is by far the most important principle that we follow in Unicraft, which means that everything that we build should be highly customizable,
03:01
that allows you a KPI driven specialization approach. And KPI could be for instance, high performance or memory efficiency. Our philosophy is that everything is a micro library. This includes even OS primitives like schedulers,
03:21
memory locators, virtual file systems, network stacks, et cetera, et cetera. Then also architectures and platform support and drivers are also selectable libraries that you can choose from and also application interfaces. The use case targets are really widespread that we see.
03:42
One of them is obviously microservices, but you can also build function as a service where you have really just the language environment or the language runtime actually running. Network function virtualization because of high performance, edge computing and IoT including industrial IT and automotive
04:02
because of its efficiency. To give you a brief impression, what we mean with our micro libraries and how they're organized, you can see this picture. So here you have the assumption, you have an application that's written for Linux or any other POSIX operating system. And then you have the following layers underneath.
04:22
The lowest layer is the platform layer, which basically implements drivers and CPU architecture things. So the API it provides is really low level and more mechanics driven. Then you have this OS primitives layer,
04:41
which then implements things like schedulers, file systems, networks, et cetera, et cetera. But also here you can choose only the components you need and then on the higher layer, you have things like the POSIX compatibility layer, which mimics things like file descriptors
05:01
or how a POSIX process is organized or the socket interface, et cetera, and the libc layer that most applications use. One benefit here is that when you develop, let's say an application from scratch, you're totally free to choose to which layer you wanna bind your application to.
05:23
So meaning, and then let's say most extreme case, you would write your application directly to the platform layer, and then you have the shortest access to the platform features and hardware drivers or virtual hardware drivers provided by the platform library.
05:42
Our monitoring and debugging features are implemented with micro libraries that you can select and choose for your build. So you have, for instance, libuk-debug, which provides you a logging facility and a print system, provides you a search that you can enable or disable for your build,
06:01
trace-pointing, and the GDB server that Mark is presenting in a bit. We have also uktest, which we use for unit testing. We have ukstore, which is basically a directory of runtime library getters and setters. I will tell a bit more on the next slide about that.
06:22
And we have ubsun for detect runtime memory box and a tool that we call uniprof, which we use for performance analysis. We originally started to implement and design ukstore
06:43
because we wanted to be able to monitor Unicraft instances in production deployments. So we came up with the following requirements. Any instrumentation that we need to do to our micro libraries, let's say, the network stack or the VFS,
07:01
should be reusable as much as possible for different scenarios. Ukstore itself should be optional. This means that any instrumentation of micro libraries should also get removed at compile time when ukstore is out of the build. We wanted to be able to retrieve data,
07:21
but we also wanted to be able to set values at runtime to libraries, which actually leads us to a getter-setter interface. But the data types and naming and so forth should be defined by the library because there you have the context
07:41
what you're actually implementing. We wanted to have a pull-oriented design because we prefer minimal overhead. Computing and parsing should actually be done only when requested. And the whole thing should also enable us integration to common, monitoring, visualization and alerting systems
08:02
like you know it with Prometheus or even Grafana. We then came up with the following architecture. So in the middle you have a directory of entries or probes or let's say actually an entry is a getter or setter.
08:21
The micro libraries, they populate entries into that directory. And on the other side, we have access backend implementations that implement an exporting to a specific protocol like REST or pseudo FS or shell command,
08:42
which will then actually look up the entries and then directly call these getters and setters provided by the libraries. By having the instrumentation, you can then build dashboards like this. Together with the Prometheus access backend library that opens the REST API port on a defined network interface,
09:03
you can then visualize with Grafana all the metrics you're interested to. A nice thing of Prometheus is actually that you can set up alarms when values go into a bad range that you define beforehand. And so you can actually seamlessly integrate
09:22
Unicraft instances to your Prometheus deployment. Yuki Store is currently in the upstreaming process to the Unicraft base repository so that we have all the necessary APIs and macro definitions available to implement the library instrumentations,
09:43
but also the access backends. As next step, we will provide an initial set of important utilization metrics like memory, CPU, network, and storage, and we'll continue working on access backends. And with this, I would like to hand over to Mark
10:00
so that he can present recent improvements to UK debug, including the announced GDB server backend. Thank you, Simon. My name is Mark Ellinghaus and I'm head of engineering at Unicraft and I'd like to give you an overview of the new debugging features we are currently working on.
10:21
Previously, when we were debugging issues in Unicraft, we used printf debugging or resort to the debugging support in the underlying platform. For example, the GDB stuff in XAN and KVM. One of the new features in Unicraft is an integrated GDB stuff so that you can directly connect your GDB to Unicraft
10:40
and start debugging right away. Another request that we had and we also heavily missed internally is a proper crash screen that would display information about the register state, a stack dump, and a call trace at the time of the crash. Especially the call trace with simple information is of great help because you don't have to manually look up symbols
11:02
over and over again. In the following, I'd like to give you some details about how we integrated these features in Unicraft and kind of share our experience on how to build a GDB stuff for an own operating system. When it comes to implementing an OS integrated GDB debugger stuff,
11:21
you may ask why to bother in the first place. For example, KVM provides full source level guest debugging support. You simply start your virtual machine with a special parameter and the QML process will open a TCP socket for GDB to connect to. And then you can do regular debugging with single stepping, setting breakpoints,
11:40
inspecting the system state, et cetera. However, relying on such debugging support in the platform comes with some major disadvantages. For example, not all platforms you may want your operating system to run on come with such debugging support. For example, Hyper-V does rely on the operating system to have a debugger stop.
12:01
Similarly, you won't find such capabilities on bare metal. And also cloud providers in most cases won't give you the required level of control. So it makes it hard to debug the operating system when it's deployed in its target infrastructure. Another issue is the semantic gap between the platform and the guest.
12:23
And because the platform does not know the data structures of the operating system that's running in the virtual machine, certain capabilities such as threat level debugging aren't provided by the platform debugger. And closely related to this is that usually in error conditions,
12:41
we want to get the chance to connect the debugger and inspect the situation. However, since the platform does not know when the system in the virtual machine is in an error state, you usually have to set breakpoints beforehand manually so that in the situation of a crash, you don't end up deep down in a halt loop
13:01
when you connect the debugger. Considering all these aspects, having an OS integrated debugger becomes an attractive solution. So what we want to do is to move the debugger stop to which we connect from the platform to the guest itself so that we can have debugger support on all platforms
13:23
with no semantic gap and with proper state inspection in error situations. To have an OS integrated debugger stop, we need a couple of things in place. First of all, we need to have a communication channel over which the GDB client and the GDB debugger stop communicate over.
13:42
Then of course, we need to implement the GDB stop itself, which is mostly parsing of GDB packets and processing of GDB commands. Then we need to establish the means to react to debugging events. For example, every time we hit a breakpoint, the CPU raises the CPU exception and we need to be able to catch this exception
14:01
in the debugger. And then of course, we also want the debugger to be invoked in software-based error conditions like a failed assertion or a kernel crash. And since this is unique craft, it was important to us to build as much of this functionality as optional and replaceable micro libraries. That means that we can remove the support for debuggers
14:22
if we don't need it. And at some point in the future, add support for other debuggers besides GDB. When we think about communication channels, we have to consider that unique craft has a very short boot phase. That means the system is usually far into the boot when network's available. However, we want to be able to start debugging
14:40
as early as possible during boot. This means we have to use a very simple device, which comes to mind our serial device because they don't have any requirements on subsystems like for example, the memory allocator. They are available on most platforms, even bare metal. They're quick to set up and simple to use. However, in unique craft, we already use a serial device
15:02
for outputting kernel messages. And this left us with two options. First, we could share the serial device between the serial console and the debugger and transfer ownership to the debugger as soon as we encounter an error condition. The second option is to dedicate the serial device to debugger and output kernel messages in the debugger instead,
15:22
as long as we only have a single serial device available. In the end, we settled with option two. That means we dedicate the serial device to the debugger. And this is because we also want to be able to interrupt execution at any point in time using control C in the GDB debugger.
15:40
And for this to work, you need to have an already established debugger connection. The main task of the actual GDB stop is to communicate with the GDB client. That means reacting to commands like reading a certain memory location or setting a break point. The communication works using a packet based protocol
16:01
where packets are encoded in a simple format with packet data enclosed with a dollar sign and a hashtag followed by a simple checksum which is only the sum of the characters in the packet data. An example request could look like this. We have at the beginning a short string
16:21
identifying the kind of command. In this example, it's reading a certain memory location followed by a comma separated list of parameters where the first parameter is the address to read and the second parameter at the length of the memory access. Another example is retrieving current snapshot
16:43
of the register state, which is encoded by just using the lower letter G and the GDB client then expects a response which simply consists of the register values encoded as hex string. Now, as I said, the GDB stop is mostly parsing of packets
17:01
and that means for our implementation that around 70% is dedicated to protocol handling which is around a thousand lines of code. And to get basic debugging operation working, you need to implement around 10 comments. On the architectural side, the GDB stop is responsible for saving and restoring the CPU context,
17:21
reading and writing memory, setting up single stepping which means for x86, it's just setting the trace flags in the eFlex register and of course implementing trap handlers, for example, for the breakpoint trap. Another thing is that setting and unsetting of breakpoints can be entirely done by the GDB client. The client does this by replacing instructions
17:41
with the debug break instruction, just by issuing a memory read and write commands. This advantage, however, of this approach is that you cannot set watch points. That means you cannot observe memory reads and writes to certain memory locations. At the moment, Unicraft supports x86 for the 64
18:04
which is around 450 lines of code for the implementation and RM64 which is around 250 lines of code. One part that's interesting on the architecture side is the trap handling. As we said, the debugger must be able to react
18:20
to various traps, for example, the breakpoint trap. Now in Unicraft, we have for each trap provided by the architecture dedicated handler function in this platform support code. Now an option would have been to manually invoke the debugger in the various traps, either because the trap is directed to the debugger or because no one else in the code handled the trap
18:43
and we want to fall into the debugger. However, that would have created a dependency between the platform and the debugger library. And in Unicraft, we want to keep the library independent of the platform so that we can easily disable the functionality
19:02
or replace it with a new implementation. So what we wanted is a more extensible and generic trap interface. And what we came up with is an event-based interface where the platform defines a number of events that any library can register handlers for.
19:21
For example, we have a trap debug event or a trap page fault event. Then in the trap function itself, the platform raises the corresponding event, which eventually will call the handler function. To keep initializing overhead, low handler registration is done at the link time
19:41
by building corresponding handler tables. If multiple handlers exist for a single event, we can specify priorities in the event handler registration to give precedence to a certain handler. An unexpected problem that we encountered when doing first experiments with our GDB stuff
20:01
is that the GDB client might issue invalid memory accesses. And that can, for example, happen when GDB tries to interpret integers as pointers, for example, in the assembler mode, when it tries to read from invalid pointers, for example, during a backtrace.
20:20
And also the user might unintentionally issue an invalid memory access, for example, when interpreting data structures. So this needs special handling, otherwise the GDB stuff will crash the system with an unhandled page fault exception. Ideally, we want to have a function that transparently catches illegal memory accesses and just returns an error code
20:41
if the access cannot be done. For example, we have here a version of the memcpy function that returns the copied length of success and minus one on error. Internally, the function just continues execution at the specified go to label whenever a page fault is encountered during the copy operation.
21:01
This is similar to the no fault type of memory access functions we know from the Linux kernel. With the new trap interface we have, it was pretty easy to implement this. We just register a low priority page fault handler. And now for every location, in the code that should be able to try a memory access, we register an entry in an exception table.
21:22
And this exception table also stores the continuation labels. And then in the page fault handler, we just scan this table for the current instruction pointer and then continue execution at the fault label when we return from the trap. Bringing all components together,
21:41
we have the GDB client over here that is connected over serial connection with the GDB stop. And the GDB stop then performs every memory access using the no fault library so that memory accesses don't lead to a system crash. Then we have the trap interface
22:00
with the debug trap events and events like the page faults. And we have the libraries that register handlers for these events. All events that are not handled lead to a system crash. And this crash is also handled by the debug library. And over this way,
22:21
we also get a chance in the debug stop to invoke the debugger. The second debugging feature we recently added to Unicraft is a uniform crash experience. Previously, we had every architecture with its own crash handling code. That means dumping registers, doing stack traces, halting the system, et cetera.
22:42
What we wanted to have is a uniform experience on all architectures and also have the chance to invoke the debugger if available. And what we came up with is this screen where we dump the registers, stack trace, and the call trace. And we implemented this screen for the x86
23:03
and ARM architecture. And as you can see, the layout and the kind of information that is presented is similar in both cases. A particularly useful feature in the call trace is the symbol resolution. And we do this by embedding a symbol table
23:20
in the final binary. And this table contains entries that map instruction pointers to symbol names. We generate this table by linking the unikernel three times. The first time we link the unikernel without any table and then extract the debug symbols. And then in the next step, we generate a table
23:40
and then link the binary with the table. But this might change symbol addresses because the table occupies additional storage space. So we extract the debug symbols again and then update the table and link the unikernel a third time. To access the symbol information at runtime,
24:00
we added a convenient API where we can specify an address and get a symbol information in return if there's a symbol stored for the specified address. Both of these features, the GDB stop and the crash screen are currently in the process of upstreaming in Unicraft. But there's still a lot to do.
24:21
For example, we are currently working on adding thread support in the GDB stop so that you can get information on different threads and switch between threads. And also we need to visit some of the code as soon as the SMP support is there. And another point is that we also want
24:41
to add hardware watch points so that you can monitor memory addresses for modifications. And even debugging over network connection is an interesting feature when you think of a cloud environment where you only have a network connection available. I'd like to close the discussion
25:01
of the debugging features with a live demo. For the live demo, we prepared a Unicraft image incorporating the Nginx web server and the debugging features we just discussed. On the left side, we will start a new virtual machine running this unique kernel on KVM.
25:20
And we also supply as one parameter the Nginx configuration that the web server is using. Now the Unicernel is configured to wait very early in the boot process for a new debugger connection. And we will establish this debugger connection with GDB on the right side. We are using a dedicated serial device
25:41
for this connection. And we also specify the debug image that comes out of the regular build process. So GDB has all the debug symbols available. We are now connected to the internal Unicraft GDB stub. And when we switch to source mode,
26:00
we see that we are in the libkvmplot entry function, which is the first C function in the KVM platform. And we are now able to do regular debugging. For example, do single stepping. And we see with each step that the corresponding output is presented on the console. And of course, we can also set breakpoints,
26:22
for example, here in line 283 and continue execution up to this point. And of course, we can also inspect variables, for example, the libkvmplotcfg variable, and see all the values that are currently set. Now, since this is a Unicernel,
26:40
we can also debug the Nginx web server just like we debug the Unicernel itself. So for example, to set a breakpoint when a new connection is established to the web server, we can just break on the Nginx event accept function
27:01
and then continue execution. And as soon as we establish a new connection to the web server, we will break at this point. Then we stopped in the Nginx event accept function and can continue regular debugging
27:22
and also output variables just as we did before. One advantage of having an integrated debugger stop is that you can easily break into the debugger as soon as the system encounters some kind of error condition. For example, we appear in the main function right before entering the Nginx web server
27:41
and assertion that checks if there are enough parameters supplied to the web server. And as soon as this assertion fails, we would get the chance in the debugger to inspect the situation. The same is true for other kinds of exceptions. For example, architectural traps, just like a page fault. So if we run the Unicernel
28:01
and supply my crash as the parameter, we perform this illegal memory access and we see that we are indeed crashing the system. And now we are having the chance to connect the debugger and inspect the situation and see that indeed we supplied the crash parameter.
28:21
On the left side, we also see the other new feature in Unicraft, which is the crash screen. And we see that the crash screen supplies all kinds of information, even if there's no debugger available. For example, we have here the information on which Unicraft version this is,
28:42
what threads, the name of this thread, the address of the thread control block, the base address of the stack and the thread local storage, as well as a dump of all the registers and the stack, as well as the call trace when the crash occurred. And here at the bottom,
29:00
we have the reason that caused the crash. In this case, it was an unhandled trap 13, which is a page fault at the given address. Thank you, Mark. Now I would like to give you these pointers if you're interested in trying out our open source project or even start contributing to it.
29:22
I should also mention that we have many more things available like our companion tool called Craft. Unicraft itself is a Linux Foundation project and we originally started as a Zen project incubator. And if you would like to try out Unicraft for your company and want to understand the benefits,
29:43
like saving the costs on your cloud build up to 50%, we can offer help. You can just contact us on one of these main addresses that I listed here on this page. And now thank you for your attention. The chat should actually be open now
30:00
for questions and answers. And we will also provide this slide set on the Forstam page of this talk. Thank you.
30:24
To get visualization and integration with alerting systems when you run it on deployments like people have out there with Prometheus and Grafana. So there's only a little piece that needs to be implemented on the kernel side
30:40
to export all these metrics. And that was really, it's a bit of fascinating to see how quick this can be done. Then, and also for the Uniprof tool, generating the flame graphs is very helpful. For example, we had an issue, a performance issue in the file system and the IO stack.
31:01
And we could quickly identify where the issues were and correct them. So these tools are really helpful in the development. So for the GTP part, you said you implemented 250 lines of code for x86. Did I understand this correctly? 450, yes.
31:21
Just 450 lines, that's it. Yeah, the actual architecture specific part, this is not much because you only have to write small functions that read the registers and return them to the generic part, let's say. So it's mostly context saving and restoring
31:44
and also implementing things like setting the correct flags for single stepping, et cetera. And if I got it right, there is threat inspection support, right? Not yet, but I'm working on it. And it's not that difficult because there are commands
32:01
from a GDB that you get where it's, for example, asks you to switch to a certain stack, a threat, and you then just return the state of this threat. And it's very easy if you have internally the threat control block, then you just take all information
32:20
from that control block instead of just returning the state of the current CPU. So where does the threat control block come from? So in Unicraft, we have, for example, the scheduler that has a list of all active threats. And from this list, we can pick the control block.
32:45
And what I also wondered, so this Unicraft binary in the end, is this an ELF or what is it? It depends where you run it. Basically, it's a monolithic kernel image for the platforms where you boot from ELF.
33:00
It is an ELF where you have multi-boot support. It's a multi-boot binary, right? That's following the different standards because the idea is to specialize also the resulting binary to the target platform where you're executing. Okay, so the QA is over now. I hope some more questions follow in the chat room.
33:22
I thank you very much and see you later.