Device driver gardening
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61508 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Device driverDifferent (Kate Ryan album)WritingSoftware frameworkArmGoodness of fitSoftware developerDevice driverComputer animation
00:39
Device driverKolmogorov complexityComputer hardwareDecision theoryComplex (psychology)CodeEmulatorMaxima and minimaRun time (program lifecycle phase)Electronic visual displayGame controllerWhiteboardBridging (networking)Revision controlOrdinary differential equationTouchscreenIntegrated development environmentRevision controlCodeSoftwareFreewareMereologyDevice driverSoftware bugPhysical systemSoftware frameworkNumberDecision theorySingle-precision floating-point formatDampingComputer hardwareImplementationMikrokernelComplex (psychology)Integrated development environmentKernel (computing)Bridging (networking)Electronic visual display1 (number)Flow separationMilitary baseSystem on a chipShared memorySummierbarkeitDifferent (Kate Ryan album)Maxima and minimaFunctional (mathematics)Heat transferCASE <Informatik>Complex analysisOpen sourceConnected spaceSoftware development kitPerformance appraisalGroup actionPoint (geometry)Multiplication signEmulatorLatent heatEqualiser (mathematics)TouchscreenTouch typingRun time (program lifecycle phase)Data managementEvoluteOpen setPhysical lawSystem callContext awarenessMenu (computing)Control flowForm (programming)Graph coloringCross-correlationArmDeterminismNetwork topologyCuboidOptical disc driveMetropolitan area networkGoodness of fitComputer animation
08:08
Integrated development environmentDevice driverCross-correlationDemo (music)Computer fontKernel (computing)Wide area networkScripting languageNetwork topologyBridging (networking)Maß <Mathematik>Compilation albumPhysicsScale (map)EmailInclusion mapSoftware frameworkScripting languageGastropod shellNetwork topologyConfiguration spaceKernel (computing)BitInformationIntegrated development environmentArmMereologyCompilation albumDevice driverPhysical systemException handlingString (computer science)Point (geometry)1 (number)Execution unitoutputComputing platformAlgebraic closureFunctional (mathematics)Order (biology)EmailMaxima and minimaSystem callEmulatorRun time (program lifecycle phase)Source codeInterrupt <Informatik>CodeOnline helpCompilerComputer hardwareDifferent (Kate Ryan album)Error messageDevice driverSymbol tableMultiplication signFreewareBit rateCore dumpSpring (hydrology)Complete metric spaceVideo game consoleRevision controlAreaLevel (video gaming)Computer fileData conversionCausalityLie groupTheory of relativityComputer configurationOpticsAnalytic continuationComputer architectureComputer animation
15:37
Maß <Mathematik>Compilation albumSystem identificationDevice driverScale (map)Data bufferExecution unitCompilation albumFunctional (mathematics)ChainArmMultiplication signBit2 (number)Virtual machineDevice driverSymbol tableComplete metric spaceCountingInclusion mapEmulatorPoint (geometry)ProgrammschleifeDeclarative programmingCodeAsynchronous Transfer ModeDigital rights managementComputer fileComputer animation
18:24
Kernel (computing)CodeMemory managementCore dumpComputing platformService (economics)Right angleDevice driverCodeEmulatorAbstractionLimit (category theory)1 (number)Acoustic shadowKernel (computing)ArmSocial classEmailAssembly languageInclusion mapComputer animation
20:00
Computer hardwareMultiplication signPresentation of a groupInstance (computer science)ArmGroup actionDevice driverDemo (music)Virtual machineGoodness of fitComputer animation
21:06
IntelElectronic visual displayDevice driverUniverse (mathematics)System on a chipGame controllerGraphics processing unitArchitectureIndependence (probability theory)Cross-correlationRevision controlInstance (computer science)MathematicsData structureMultiplication signOpen setWebsiteElectronic mailing listDifferent (Kate Ryan album)String (computer science)CASE <Informatik>Stress (mechanics)Device driverNumberMixed realityRevision controlPoint (geometry)CodeCross-correlationIndependence (probability theory)CausalityArmComputer architectureResultantMereologyExpert systemForcing (mathematics)Functional (mathematics)Parameter (computer programming)Computing platformBlock (periodic table)2 (number)Computer hardwareContext awarenessLink (knot theory)Moment (mathematics)Traffic reportingCommunications protocolArithmetic meanPhysical systemStability theorySoftware developerKernel (computing)Stack (abstract data type)Staff (military)Ext functorTrailBitLaptopCountingExplosionFile systemSingle-precision floating-point formatElectronic visual display
27:46
Point cloudProgram flowchart
Transcript: English(auto-generated)
00:08
Can you hear me? Okay, fine. Stefan, please. Yes, good morning. Hello and welcome to my talk. I'm Stefan Karkowski. I'm a Genode developer since 2009.
00:24
And today I want to present you how to transplant Linux kernel drivers into the Genode OS framework much more faster than before and hopefully precisely. So let me start with the motivation behind this.
00:45
Of course, you might ask why we use monolithic kernel drivers when we talk in a microkernel dev room. Of course, there are good reasons to implement drivers from scratch and we also have several drivers
01:01
which are written from scratch, but the ever-increasing complexity of modern hardware for single devices, but also for the pure number of devices inside a system on ship is not easy to handle for a small team. That's the one reason.
01:21
And often it's poorly documented even if at all. And we also have hardware bugs inside and you have to find those bugs because they typically are not documented. And then you have to find workarounds for it. And all of this is mostly part of the Linux kernel.
01:41
And yeah, you can reuse it because it's free and open source software. And so to sum it up, it's simply an economic decision. So if you want to enable a modern device and you have limited time, then this is the way to go. Okay, we have collected a lot of experiences
02:03
in the last decades to port drivers from Linux to the Geno.OS framework. And you have in general two extreme approaches and the reality is somewhere in between always.
02:20
So either you use just the pure driver code, what I mean by this, I mean code which directly interacts with the hardware by writing to some IO registers or by setting up a DMA transfer or something like this. And in that case,
02:41
you of course have to implement each Linux kernel function that is called by this driver code. But the good thing is you don't have to implement the whole semantic which the original function is implementing. You only have to match the semantic this single driver needs.
03:01
This leads to a more low complex function than the original ones and in sum to a more minimal Linux kernel driver. But of course you cannot share this emulated code in between different code bases.
03:21
So if you have not only one driver but several, they will have slightly different semantic needs. And so reusing the same emulation code might be a problem and therefore the whole effort if you don't only port one driver but several ones can increase if you ever and ever again
03:40
have to implement the whole emulation code base for the driver. And of course you need the actual needs of that driver. So we need a deep knowledge of the driver itself. On the one hand, this is the one approach. On the other hand you can use as much as possible from the original code base.
04:02
Thereby you might gain more or less the same runtime behavior than the original one. And you can of course then better share resulting emulation code because it's already stressed by this whole bunch of code running on top. Thereby you get less manual work to do
04:23
for having more than one driver. But of course the code base for the single driver increases because you have much more of the original Linux kernel. And if a problem arises then you have to know a lot of the whole Linux kernel itself because it might be in the timing subsystem
04:43
and whatever, you can name it. So in the recent past we were more on this side on the taking the pure driver approach but the high effort for each driver was also leading to the situation
05:01
that you keep your old code base that you are not that good and maintain the code and getting a new kernel version and driver updates. And at some point there was a need for action. For me this was at the beginning
05:20
of the pandemic situation when I was trying to enable the display engine of this device which is a MNT Reform 2 from MNT-RE, a small company from Berlin. So a completely open hardware. And yeah, I tried to enable the display engine
05:40
and it includes an NXP iMX8 SoC. And we already had a driver for this because a colleague of me, he enabled in three months on the early evaluation kit the HDMI connected display. So this was one part
06:01
and then another colleague of me wanted to have a touch screen which is connected via BSI connector and not via HDMI. And again, he had to spend three months into this work because on the one hand
06:21
there are more devices involved now. On the other hand, you had all this bureaucracy for device pre-management and it was all hard coded for this first use case of using HDMI for the specific port. So there was a lot of manual tweaking to do to enable the touch screen.
06:40
And then I wanted to, I thought, yeah, I don't have to do this, someone else did it. I can now reuse it for the MNT Reform for the panel because it's also connected via BSI. But actually there's another device in between an EDP bridge in between the BSI connector and the panel. So yeah, I had to do work again.
07:02
And then I recognize, oh no, the code base we used for porting is a different one than the one of MNT Reform and it's a totally different kernel version. You have to back port stuff, you cannot correlate it. No, I give up. So that was the turning point for me
07:20
to start a new way of porting. And of course it was not only me, but we had a lot of discussions formally in the kitchen, here in coffee breaks, what we want to change. And so number one requirement for the new approach was to reduce the manual work for tailoring a driver specific environment.
07:43
And we wanted to meet as close as possible the original semantic offset driver so that whenever you changed the context, like it was this display engine, it just works. You don't have to do much more. And because formally we all,
08:03
at some points we have the impression you cannot be deterministic in knowing when you will finish your porting work because when some problem arises, you could not correlate it to the original runtime often.
08:23
So it was somehow hard and we wanted to change this. So there should be an easy way to correlate it to the original runtime. And last but not least, we wanted to share more of this resulting emulation code, which is more semantic complete so that we can maintain the code better.
08:42
Okay, so this is the story beforehand, and now I come to the actual work. So I want to introduce you to this approach for those of you who like to port drivers to G node or like doing the same approach somewhere else.
09:03
So we typically start now by configuring a minimal executable Linux kernel. Let me just call it tiny kernel, so to say. So you have to do some manual work here. You have to use the Linux kernel built system itself.
09:22
It has a tiny config, some small configuration, which is at least compilable for your architecture, but it won't run any device. And then you just enable certain configuration objects, config options, and of course you have to find them
09:41
by looking at the configurations. And in the end, this might take some time, but in the end you will have something which you can correlate laterally. If you run the driver in your ported environment and you want to look why doesn't it work, then you really have a minimal Linux kernel
10:02
which just drives this device. And this is the first thing to do. And another aspect of this is that you gain a minimal kernel configuration for your code base which just calls those kernel functions that you really actually need to drive that device.
10:22
So you don't have to emulate that much. Okay, kernel configuration is only one part. If you take an ARM device today, then you have of course these device trees which name what kind of devices you actually have in hardware and which also contain additional driver information.
10:44
So it's a bit of configuration is also inside of these device trees. And this is the device tree for the MNT reform. You see it's quite complex. So you have to identify what kind of devices are interesting for my tiny kernel to execute those.
11:03
And this is again, some work to do, some manual work to do, but at least you start to know more about the dependencies of your hardware. And we have developed some tooling for it.
11:21
So you can, this is a small TCL shell script which parses device tree sources and then you can name device nodes that should be extracted and it will take them and the transitive closure to give you something like this. And then you can take that device tree of course
11:41
with your tiny kernel and start it and it will just drive that device. And we also take that for our own ported drivers as input value. Of course, you won't implement everything which is seen here. So powering, reset, pins, IQ, stuff like GPIO
12:04
or something like this would be part of other drivers in the system, like the platform driver or some dedicated GPIO driver in the Genome OS framework. So those highlighted ones are the ones that we actually need for porting.
12:21
And this is the starting point for you to identify the first compilation set that you need. So each of those device drivers has some compatibility string and those are used in Linux to identify the concrete driver of the Linux kernel. And so you can take those strings
12:41
and grab in the Linux kernel sources and then you get something like this. So you have your first compilation set units and you can put them into a make file, into a build environment. And then we combine it with the unmodified Linux kernel headers.
13:03
So we take the original include part of the Linux kernel. Formally, we always defined the whole definitions you needed by hand. So this was a lot of work to do. I would say initially the most work you had to do.
13:26
And now we just take the original Linux headers and then you can just compile those compilation units you already have seen. So it's really a work which is done by this. But of course, there are some exceptions.
13:43
So we had to tweak some headers, we shadow some few headers to prevent the system from trying to enable-disable interrupts or something like this. And especially to define init calls in the Linux kernel.
14:01
So each subsystem in the kernel, including any driver, has some init call definition and those are the order of the init calls is, yeah, important. Even if you have one init call priority level, there are dependencies in between the different compilation units
14:21
and they are solved by linking order. So the Linux kernel uses some weird linking magic to put them all into one order and laterally, when starting the kernel, it takes that order. So we didn't want it to infect our linking script with this. Thereby, we have built some tooling again
14:42
which uses this tiny kernel that you built in the very beginning and just extracts the order of the init calls, put it into a header and you can just include it in this build environment and then you run and it will, yeah, the emulation code environment of us
15:01
will just call the init calls by the correct order. So when we do all of this, then you of course get a lot of undefined references for all the functions which are not implemented yet and this is a lot of error messages from the compiler so we made a small tool
15:20
to identify those undefined symbols, help you to identify the original compilation unit which implements them and then you can try to find a correct setup for this and I want to show you this shortly.
15:44
So I've prepared some make file like here, so here you see the compilation units we identified, there's some inclusion of the general emulation code base
16:03
and yeah, if you now use this tool, it will build the target which you name, so it will try to build the driver and it will collect all the undefined symbols and here it just shows you the symbols
16:22
and the overall count of the undefined symbols. Typically, you can also have this, what I said, the compilation unit which is responsible but I've skipped this here because on this machine, it's a bit slow.
16:42
So we can now identify, okay, there are symbols for DRM mode which we want to solve and we see, okay, let's try to add the original one. Oh, sorry, and yeah, you just run the tool again
17:06
and it will show you in a few seconds. So on the PC, this is quite quick but this is just one gigahertz or whatever, I don't know. So it's a bit lame
17:21
and it has to recompile the driver, of course, in the background. Okay, and now you see it's seven symbols less and in the end, I think because of the time we will skip this, in the end, you can generate with the tool the missing symbols and it will give you per function
17:43
the correct declaration of the function, of course, and it calls a function which gives you the backtrace, fill them and just loops endlessly so you have a no returning function. Therefore, you don't have to get a valid value back
18:02
or something like this. So if you now take the driver, it will link, you can start to execute it and you will always get the point where something is not implemented yet. Okay, so let me just switch back.
18:24
So okay, so this is the overview of the APIs involved. I don't want to explain them in detail now but what you should take from that is we have a very strict layering.
18:44
There is this layer where there's only a C and assembly code which is actually the Linux kernel code and the shadow copies of the Linux kernel code. Those are the only ones which can include Linux kernel headers and then you have this emulation code base
19:02
which is just a C abstraction for the Linux kernel code above and then you have all this C++ stuff from us which abstracts the G node services and the concrete driver services. And the good thing is you have those abstractions here
19:23
from the device services and you have their pondon here and then if there's one, let's say for an ethernet class, you can just reuse it. So if you already have this in our emulation code base,
19:40
you just need to implement or port the concrete driver but you have all this glue code which connects with the actual APIs and services. It's all always there. Okay, so now let's see this in practice.
20:03
Okay, I just skip this here. Once you shut down the virtual machine monitor. Okay, so you actually see the whole time G node in action
20:24
and what you see here on this mount reform is actually everything the device brings with it except audio. So all other drivers are already in place and this also is valid for the GPU for instance.
20:44
So here you can see the GLMark demo again. Those of you who have seen Norm's presentation yesterday already knows it. So I think I run out of time. No, you have five minutes. Okay, good. If you haven't, we'll give you two questions
21:00
but two more minutes, it's okay. Anyway, I think this is enough. As I said, more or less everything runs on this device. Yeah, so what are the results? So this is the list of drivers we have ported within one year now, besides of course doing other stuff. So we don't just port drivers all the time
21:23
but this is really a significant change. So we have taken new drivers for our whole x86 code base for instance. So, and you see a lot of ARM drivers for the PinePhone and for the MNT reform are also,
21:42
were also added like the Mali and the Vantel GPU for instance. And this was done by a very small team. And we also had some architecture independent porting of WireGuard. So something which doesn't even use any device at all. Okay, so in numbers, the initial driver porting,
22:06
so nowadays to compare to the initial approach, we have something like 15% of the time that it takes to do this. Of course, it's a bit hard to measure because we don't track all times.
22:21
We do in spending porting work but approximately this is the number. And this is especially because of this tooling which reduces the manual work. Of course, you have to find semantic backs
22:40
but here this tiny kernel correlation helps a lot. So you can instrument the original code then just run it on original Linux and on your ported code and you can see the difference. Driver updating, we also did this within that year because the first port was done for this display engine
23:03
and then there was a new version available, two kernel versions later and we made an update. So it's significantly faster than the initial driver port of course. And the drivers meet a better all purpose. This is what I meant with,
23:22
for instance, it took one day to enable the HDMI connector for the MNT reform once the panel worked. So it's much better matching the different contexts. And of course, there's something bad on the other hand. So the code base for a single driver
23:42
explodes like two or three times more than before. Yeah, but on the other hand, the code to maintain by ourself decreased to the count of 20% than before.
24:00
Okay, so I think that's it. If you want to read more about this, I can, yeah, reference this book, the second G notebook about how to enable a platform. There is a lot of this stuff already written by Norman and we have also much details
24:22
in different Genoians block articles. So thank you for your attention and I'm open for questions. Thanks, Stefan, any questions? Yeah, please. Yeah, first of all, awesome work, by the way.
24:41
Thank you. Linux is known for not having a stable driver API. I think there's a Linux developer from Red Hat who once says we do not do hardware extraction layers in Linux. You did say that the initial port is the hardest and then there's a lot less work maintaining back ports going forward,
25:00
but there's still some work involved. So I was wondering, wouldn't it be less painful to, for instance, support drivers from BSD? Because I'm not mistaken, they have a more stable hardware extraction layer. That might be probably the case, but actually we want to have this first argument, Linux runs on all kind of hardware and all kind of different situations.
25:23
You have, for instance, we have a BSD ported driver for audio, but for today's Intel HD audio devices, it's somehow, yeah, that device might work, but on that device, the microphone doesn't work, on that device, this and this.
25:40
So it's more about we just want to have the functionality and therefore we need to look at this. And of course we were not, yeah, we didn't like to get kernel experts, Linux kernel experts, but now we had to do it. And yeah, if you want dived into it,
26:01
then maybe we just take that advantage. Thank you. Another question here from Alex. Hi, great talk. And so I was just wondering, I wonder if I might be able to use your tooling to introduce NVMe driver into OSV, but I also wonder if maybe similar approach
26:22
could be used to also port file system drivers into operating some like OSV is missing a EXT tool driver, and I wonder if I could do something like that. I'm pretty sure you can. I mean, we already used this RAM kernel approach from Antti Kante,
26:42
who was also in the staff room in the past years, and the support BSD port of the protocol stack. And we also used the Linux IP stack in the past from Linux, and we will of course use this approach again to renew that version.
27:00
And as I said, we already used WireGuard for this, so something which is not at all connected to any device driver code. Yeah, it's possible. Okay, thank you. I have a question on licensing. Is it okay because GPL, BSD? It's all under GPL then only, of course.
27:24
Each driver report has to be under GPL, so. And there's no problem with having them linked together, I'm not sure about the license of G node. No, it's not a problem because this code then is only GPL code and G node itself is also under GPL, so it's possible.
27:41
Thank you very much. Okay, thank you so much, Stefan. Let's thank him again. And we. Thank you.