Buildroot : Using embedded tools to build container images
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 44 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46108 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
All Systems Go! 201933 / 44
5
10
17
20
21
30
32
36
44
00:00
System programmingComputer-generated imageryOpen setMultiplication signRoutingData centerQuicksortComputer animationLecture/ConferenceXMLUML
00:23
Embedded systemConfiguration spaceData centerGroup actionComputer networkGraphics tabletProduct (business)SoftwareInformation securityExponential functionRevision controlSource codeSystem programmingComplete metric spaceCodeIndependence (probability theory)Patch (Unix)Scripting languageBuildingComputer-generated imageryBuildingSpacetimeComputer hardwareQuicksortClosed setData centerSoftwareCASE <Informatik>Library (computing)Physical systemWeb serviceRevision controlPatch (Unix)Process (computing)Heegaard splittingDatabaseSource codePoint (geometry)File systemRollback (data management)CodeProduct (business)Dynamical systemGame controllerWordGroup actionRootHuman migrationCartesian coordinate systemDifferent (Kate Ryan album)Compilation albumNumberScripting languageCompilerComputer fileElectric generatorFile archiverDampingNamespaceLine (geometry)Term (mathematics)Information securityOverlay-NetzInternet der DingeVulnerability (computing)BitConnectivity (graph theory)Data managementMoving averageMultiplicationMechanism designComputer animation
05:58
Process (computing)Computer-generated imageryCompilation albumPatch (Unix)Source codeBuildingMiniDiscContent (media)Source codeElectric generatorCompilation albumPatch (Unix)Type theoryArmConfiguration spaceRootSoftwareFile systemBuildingComputer filePhysical systemFile archiverInstallation artMiniDiscProduct (business)Projective planeComputer animation
07:18
Configuration spaceComputer-generated imageryDefault (computer science)RootChainCloningFunction (mathematics)Kernel (computing)Total S.A.ArchitectureConfiguration spaceMenu (computing)Computer architecturePhysical systemLaptopLatent heatHoaxGastropod shellElectric generatorCuboidBuildingPoint (geometry)Kernel (computing)Type theoryRoutingRootChainFile systemUtility softwareComputer animation
08:30
Default (computer science)RootComputer-generated imageryConfiguration spaceChainCloningFunction (mathematics)Kernel (computing)Total S.A.ArchitectureCASE <Informatik>Different (Kate Ryan album)ChainPairwise comparisonTurtle graphicsLibrary (computing)GNU CPoint (geometry)Computer animation
09:02
Computer configurationPhysical systemGastropod shellComputer-generated imageryScripting languageScripting languageCuboidQuicksortRoutingGastropod shellUtility softwareChainService (economics)Default (computer science)Process (computing)Binary fileServer (computing)BitPhysical systemGame controllerPrincipal ideal domainPoint (geometry)BootingComputer configurationWeb pageMessage passingPatch (Unix)Computer animation
10:58
Directory servicePortable communications deviceSinguläres IntegralRootPhysical systemConfiguration spaceOverlay-NetzFile systemService (economics)Computer-generated imageryComputer fileBinary codeDirectory serviceRootOverlay-NetzService (economics)File systemPoint (geometry)Profil (magazine)File archiverConfiguration spacePortable communications devicePhysical systemRoutingMultitier architectureComputer animation
13:02
Portable communications deviceSinguläres IntegralRootConfiguration spaceDirectory serviceOverlay-NetzFile systemPhysical systemService (economics)Computer-generated imageryPatch (Unix)Scripting languageMiniDiscMetropolitan area networkRootSystem callLibrary (computing)Multiplication signComputer filePoint (geometry)Instance (computer science)HoaxPatch (Unix)Scripting languageMathematicsSource codeCASE <Informatik>RoutingLevel (video gaming)Data storage deviceWeb pageFile systemDirectory serviceMiniDiscBitPartition (number theory)Process (computing)Universe (mathematics)Block (periodic table)ChainFile archiverGastropod shellComputer animation
16:05
Gastropod shellNP-hardBuildingSource codeRootComputer-generated imageryPhysical systemIndependence (probability theory)SurfacePortable communications deviceService (economics)RootComplete metric spaceRevision controlBinary codeRepository (publishing)File archiverSource codeHash functionBuildingElectronic signatureProjective planePhysical systemSign (mathematics)Natural numberRoutingComputer animation
17:15
NP-hardGastropod shellBuildingSource codeRootComputer-generated imageryPhysical systemIndependence (probability theory)SurfaceService (economics)Portable communications deviceBuildingProcess (computing)AreaComputer hardwareSource codeVirtual machine1 (number)Multiplication signSymbol tableKernel (computing)Directory serviceSoftwareGastropod shellCodeBitTwitterQuicksortData managementPortable communications deviceSurfaceRevision controlService (economics)Goodness of fitIndependence (probability theory)Reduction of orderDifferent (Kate Ryan album)Computer configurationDampingTracing (software)Physical systemCompilation albumMathematicsDistribution (mathematics)Patch (Unix)Open sourceRootMeasurementText editorInsertion lossComputer animation
20:57
System programmingLogic gateSet (mathematics)Computer hardwareBuildingServer (computing)RootMultiplication signArmRepository (publishing)RoutingScripting languageCompilerDirectory serviceCombinational logicProcess (computing)Water vaporCodeComputing platformRemote procedure callFile systemSource codeSoftwareCross-platformSlide ruleLibrary (computing)Term (mathematics)Computer configurationFirmwareTurbo-CodeBinary codeComputer fileDistribution (mathematics)Electric generatorLink (knot theory)WordPhysical systemDifferent (Kate Ryan album)Level (video gaming)MathematicsDirected graphComputer architecturePlanningProjective planeSoftware developerElectronic signatureDataflowOperator (mathematics)Endliche ModelltheorieCompilation albumInformation securityExpert systemHoaxModule (mathematics)Revision controlSynchronizationBootingComplete metric spaceCommitment schemeTrailQuicksort3 (number)Arithmetic meanState of matterGoodness of fitGroup actionFlow separationNormal (geometry)Natural numberOverlay-NetzTable (information)Lecture/Conference
29:15
WebsiteSystem programmingLattice (order)Computer animation
Transcript: English(auto-generated)
00:05
Hello, everybody, so I'm here to talk about a tool called build route That they have existed for quite a long time in the embedded world But I think could be interesting for people that are working on containers and in data centers and that sort of stuff
00:23
So what for start? I would like to give a little warning. I'm a person from the embedded world and More precisely from the embedded industrial world so that's totally different from mobile Phones are totally different, and that's a bit different from consumer
00:41
Embedded stuff we don't exactly have the same problems. It's pretty similar, but still a bit different and It's totally different from the data center so again I think I have a tool that might be interesting for other people and that's why I wanted to talk about it But I may be wrong so if it does not solve your problems That's fine. That's why why we are here the second thing is we're going to talk about root file system
01:06
Generation and only root file system generations. There is nothing about namespace control groups that are that sort of stuff I love that stuff. It's pretty cool, but it's not today's subject So before we get started a little word on why I'm interested in containers
01:24
And what they mean for the embedded world so? Containers in the embedded world are kind of a great technical tool. That's still looking for a problem to solve Because we don't have anything like thin provisioning or dynamic deployment or live migrations doesn't exist in the embedded world
01:43
Our applications tend to be highly hardware dependent so isolating from the hardware makes no sense We do have some use cases for Multiversioning and different libraries or that sort of stuff But it's mainly linked to old or closed source software that absolutely want that particular version of that particular
02:05
Library, it's usually more like hack when we need it There is also limited use cases for containers as a sort of Weak package manager, so I won't go deep into Why but in the embedded world in general we try to avoid packages like at any cost because it breaks devices
02:25
so we usually reinstall from scratch when we upgrade, but it's a different subject and More and more are people that try to push the DevOps approach into the embedded slash iot world But it's not that easy. It's not as easy because rapid deployment does not work
02:43
Typical upgrade of Embedded device can take months or years because they stay offline because people don't upgrade them because Various reasons which means that whenever you do an upgrade you have one more version of the software That's floating around and that you need to deal with and that will have a different upgrade path
03:03
So we try to limit the number of version and upgrading in one package in an image means a new version So we tend to do big upgrades We need to archive all source code for legal reasons I mean if you're in the data center you're providing a web service usually and
03:24
You don't need to redistribute your code. You should do it, but you're not legally forced to do it in the embedded world We are which means that whenever we do an upgrade. It's one more version. We need to archive and distribute So that's a lot of work and the last point rollbacks are really frightening
03:43
I mean we do have rollbacks mechanism, and there are very robust because they are not allowed to fail But you need to remember that we have no access to our devices once they are sold We might have remote access depending on the use case and what's actually implemented But that's very limited and if things go wrong. We're toast so we tend to be very very
04:06
Cautious so the last point that stays is containers as a security feature Which again is not that simple? Why mainly because it's hard to split an embedded system into components? There is usually one user space software that does the grant of the job
04:25
And it needs network access because it's basically getting stuff from the network all hardware access because it's driving the hardware and All data access because the data is what we need to drive the hardware So splitting in in front and back and database and stuff like that doesn't really work for the embedded work
04:41
but still we are trying to look into it trying to see how to use it and Seeing what's going on so What do we want because of those well legal requirements and also because we need to manage software in the very long term like 20 years we need complete traceability to the line of code everything needs to be checksum
05:03
And we need to make absolutely sure that every line of code that ends up in our product is archived Okay, so complete archival of the source code all source tarballs all patches when we have to add a patch to upstream We need to keep that patch all Overlays because we add files just like everybody to our images we need to archive and all
05:25
Scripts that we run during debugging need to also be archived and we also need to be very independent from our host Why is that because in 20 years? Urban 226 will not work with packages. We've been building now So we need to archive also all our tools compilers
05:44
So the source of the compilers the compiler the auto tools the source of auto tools all that sort of stuff So build root so build root is an image building system It's basically an automated Linux from scratch it has big meg files that will handle dependencies for you
06:01
So if you ask it to be build Apache it will find the dependencies And it will build all the dependencies for you Then it will download everything all checksum is verified by build root directly if you provide patches It will patch your software run configure build step install It's also an image generation tool so once we have installed we all have to transform this into something that is usable
06:26
So that's tweaking permission and adding a suid or a root Permissions on the different files adding customized files and content creating file system images packet packing file system Images into disk images because we also do host system. It's not just containers
06:44
Collecting all the licenses and making sure they haven't changed and collecting all the source code for archival That's all the things it does it's cross compilation friendly because well most embedded ARM device are way too Small to compile their own software, so we need to cross compile, and it's a very old project since 2001
07:05
The overall philosophy we have in the embedded world is one command to do everything we type make Builders make base so you type make and at the end you have an image and everything is automated so Quick example, how does it work you clone build it you use menu config to customize your builder here?
07:27
I just change the architecture to x86 64 mainly as an example you type make you have an image At this point you start needing to be rude because we're going to Unpack the our root file system and just launch it with end spawn I use end spawn because it's the one I know
07:45
But there is nothing and spawn specific in built route So what we have in this image, it's already a working image. It has a busy box so basic shell and utilities Micro leap C No kernel we did not enable kernel building it did not only build micro leap C and busy box
08:04
It did the whole tool chain generation Auto tools which are needed for busy box make dev fake root and a few other tools that are needed to build the image It's technically a heavy container because it has an init system It's a system 5 type in its system that comes with busy box, but it's in there
08:22
It took me on this laptop 12 minute to build with a everything pre downloaded and it's 1.6 megabytes total, so it's very small As a comparison I also used a precompiled Gnu C library toolchain and in that in that same case it took it takes only one minute to compile
08:44
But it's six points three megabytes large, so As a gnu C library is quite larger. Is it a lot? It depends on your use case, but it's like three time as large, so it's it's an important difference
09:03
Something a little bit more realistic We're going to do a light container with apache so light container means no in its system And we'll try to remove everything we can remove so what did I do I did I changed a little few more options? I was tired of recompiling my to chain, so I took a pre
09:23
Compiled to chain from from butlin which provides us with a sort of service I've added apache, and I've removed anything I could remove so I removed busy box entirely Which means that we have no shell no standard utility know nothing I? Told the build route that this new system would have no in its system, so it removed any in it dot D or
09:45
R-System D services that might be lying around and I also told him that there was no simling for slash bin slash sh because there is no shell We do the same thing and we rebuild and we have a functional light container, so no shell no in its system
10:01
No, customization whatsoever you will have the apache it works page, and that's it It took me three minutes to build and it's 11 mega size, so Apache is a bit big, but it's still reasonable and then you need to Learn to use it because it's not that obvious
10:21
Apache control which is a standard way to start an Apache server doesn't work because it's a shell script and we have no shell So you have to launch it manually so again? I use and spawn which is the one I know and so you just give it as PID to and you tell HTTPd to run in the foreground because by default it will fork and
10:42
When it forks the main process dies and when the main process dies in the containers container containers terminate So you need to keep it around so yeah? Very easy to use that's the whole point very small and very easy So now how would you customize because right now?
11:02
We just took a generic Apache, and we have nothing for us the Easiest way and the most common way to do it is with overlay, so I guess most of you are familiar with overlays I mean that's more or less how you do it with docker files too, so I guess it should be okay basically you tell
11:21
Build root You give it a directory And you tell it this is stuff that you have to put on top of your file system And it will just put it on top of the file system after everything else So you tell it where it is the top dear slash overlay you create the directory and? Subdirectories as you want them in your target, and you simply put the file in there still
11:43
I am a normal user. I do nothing as root you run your make you unpack your file system and you Run it so in this example. I did a portable service, so I don't know most of you. You're familiar with portable service Okay, so portable service allows you to package in an image a binary its dependencies and run it as a system
12:05
D service and it's basically a Light container plus a service file so it can be used out of the container itself weird way to define it, but it'll do So the only thing I add is the apache dot service file, and then I run it so again
12:21
I don't care about container configuration so I just used the trusted profile from system D and did not go into the details of configuration and It works, that's it, so the the whole point is that using overly directories is Is great for archival because it's trivial to use with git
12:45
Because you see when your file of are changed you can trivially archive them you can trivially work with them and overall it's It's always the same thing. I mean the whole point was Was build root is that anybody who wants to play with it you just clone the stuff make when you can feed built
13:06
and it works, it's Just play with it, and you'll see how easy it is to use There we go with our customization tools We have a patch directory, so there is a directory you can you create a subdirectory for any package you want
13:22
And you put a passion there and build root will automatically apply the patch to the sources So you have this chain where a build root will download the source check the checksum for you Apply any page you give you you give it and build the whole thing and the whole thing is checked and the patches are again In directories that are is easy to store in git
13:43
So it's pretty easy to have everything you put in your image stored in a single place and trivial to find post root FS scripts are Scripts which are run within fake fruit do people here know a little bit how fake root works
14:00
Yes, so fake fruit. Okay. I'll do a quick reminder fake fruit basically allows you to run Any process and it use LD preload to go? Under well between the C library and the process and it will emulate any call that would require root permissions
14:20
So for instance if you try to change the UID is a ownership of a file to root it will write down somewhere that you try to do that and tell you that it worked and Whenever you ask him about the permission again later It will give you its fruit, but it won't do the actual change and just simulate them for you But it's good enough to build an image as a normal user
14:44
Because you create your image like that so pretending to be a root and still be in this same fake root universe You will run tar and you will create a tar files with fake permissions But inside the tar files you will actually have the right UID so that's how you can create a whole tar file containing
15:03
Block devices or containing files owned by root without being root yourself Linux won't won't allow you to untar the file because you will need the right permissions to entire But you can create the tar file in the same way after that you can within fake root You can then take the tar file and create a disk image
15:23
containing root owned files without being root So the whole thing so this post root FS script allows you to run Shell scripts within fake root to do the kind of adjustments where you actually need to do to be a route to do And then you have less use case which is a post image script
15:42
Which is more for the stage where you assemble various partitions and file system images into a disk image So that's less useful for containers. I guess but when we're doing actual embedded stuff. We use that a lot So yeah this whole thing has been around for a long time and have solved the problem of archival and of
16:02
reliability which is the whole point So Why use build root? What does it bring so we're source-based you have all the source of everything around Signatures are included in the recipe. I mean build root upstream whenever it
16:22
Upgrades the version of a package available in build root it will include in the build root get the hash Of the source so when you build your system will go and download say on github a git repository And then we'll compare that with the hash comings from the build root project so everything is checked automatically
16:43
No need to be root to build images so this morning I've learned about all the different versions of a rootless container building this one is also completely rootless It does not have any suid binaries anywhere. We just do everything as a normal user You don't need any root only call to build an image
17:03
Complete archival complete traceability complete license compliance these are really really needed in the embedded world so we Kind of can't work without them Highly independent from the host system, so it's kind of obvious when you sing think cross compilation
17:22
But we do need to rebuild every tool we need to rebuild them as a o2 tools We need to rebuild make because different version of make Produce different builds, so if you want your build to be reproducible you have to have the same version of make and more importantly in 30-years time who knows what version of make we will have and if you have to rebuild an old software in 30 years
17:47
It's very important to save all the source code I mean if you want a good source of stories where open-source saves a day go and find find some Some very early Linux users Because those are the ones that had serious problems with hardware and that could solve it because it could patch
18:05
kernel the kernels that were 20 year old and It works, and it's awesome, and you can save your customers lives this way. I did it a couple of time so All patches are visible and easy to manage same idea we have all the source code around
18:22
Reduced attack surface if you have binary based Distributions you cannot easily or not at all change compilation options with this system if you if it doesn't find an optional Dependency for a software it will compile the software without the support for that optional dependency so reduced attack surface
18:44
Reproducible builds they are marked as experimental, but In practice they work really well Build root and yokto the embedded world in general We have been pushing for a predictable build for a long time And we're really glad that people like red hat and Debian have started pushing to because it because it helps a lot
19:05
So very easy to build portable services and like containers because again you can have everything from scratch And you have all the dependencies managed. You don't even need to know what your dependency are so you'll just build it It's easy to debug and hard to debug It's easy to debug because you have everything available you have all the source all the debug symbols
19:25
It will keep all the build directory so all the build artifacts everywhere so in a way. It's easy to debug It's hard to debug Because you don't have a shell well usually you have a shell But if you start removing every tool in your image, you won't have any debugging tool and sometimes
19:42
It's very disturbing to discover that Nothing works So you have to know about remote debugging or you have to? Tweet your image to have a shell to debug ours. There's all sorts of stuff which makes it a bit tricky You need to understand Linux, that's That's pretty surprising, but there's all sorts of stuff in Linux nobody knows about so it's a very frightening Pam for example
20:07
Our TTY management or all those sort of subject your distro does a great job of handling that for you Except that when you're really working with empty Containers like very minimal you won't have that so you will have to tweak some areas where nobody is very
20:23
comfortable so know about it and Build time yes, and no build time nowadays on mother machines are very short Especially when you have already everything downloaded of course you don't download multiple time. It just downloads the first time so
20:41
build time is Might be an issue. It's for you to measure. That's the big idea with builders It's easy, and it traces everything and you have all the source code Thank you, so any questions
21:06
I've been working also in the embedded work more specifically we have medical devices yes So the development flow was a bit different because you wouldn't want to flash The device each time you did a change with the code
21:22
So you will just have a NFS mount of the file system usually locally and then you will maybe use everything We are staging folder where you will just install your artifacts to to mount it How is your development flow by using build root? Do you need to flash each time or?
21:41
Build root is just about making the files that will go on the target so you can in this example. I've only generated tarballs, but you can Generate file system images when I use it with a remote root on NFS like you do I Usually have some well this time they need to be through ID scripts that will just untar the tarball in whatever
22:04
Directory is mounted remotely you reboot the device and you're good. It's It's a process to learn how to make it build root doesn't care about that State it's released a step before and back then we weren't really able to use fake root for
22:21
Generating the tarballs we use pseudo tar at the very end Because we had some binaries with capabilities set like a cap natural or so I think the newest version of fake root handle capabilities, but just retest don't take my word for it. Okay. Thanks
22:42
follow-up question on that So Yocto, I think uses or gives you the ability to Wrap the resulting artifacts in depths and RPMs so you have an option to service the image That is other than flashing whether it's actually flashing or what you're describing With you know individual kind of package upgrade operations any plans to do that in build root
23:03
No, that's pretty much the opposite of the builder philosophy so build root is a Linux as a firmware Yocto builds you complete distribution which includes images to install your distribution But it's a distribution before being an image generation system, so they're really the opposite philosophically and so
23:26
No, it would be against what? Build root is trying to do if you want packages you should go with yocto It's the right tool for the job now. I can explain Build root in 20 minutes with a couple of slides. I cannot do it with yocto
23:49
Nowadays this containers and other things are available multiple platform like you can run container on arm or other things So I think currently if you want to build an arm
24:01
Container it will be a feed of it the from image or whatever will be again a arm image so So in a OCA normal if you a docker you take you can run a docker in Your Raspberry Pi or any other arm platform or any other platform or so so to build that you have to be in arm
24:23
Oh you mean you when you're doing docker for arm you usually build on arm Yeah, otherwise you have to have a some kind of a that's what my understanding is. I never believe you all yeah So my question is here can build root can is if I this sitting using its cross come Baylor and build
24:42
To an another target for OCA compliance targets well there multiple answers There's a question basically is why cross compile when you can directly compile on the arm systems? No, that's the problem is it will be low hardware, so it will take more time That's one of the reason especially well you you have server arms, and you have embedded arms and the embedded arms
25:05
Nowadays could compile that sort of stuff, but it will take hours. It will be incredibly Yeah, that's what so what I'm asking asking is By sitting in x86 architecture. I can I use build root and build and I
25:21
Yes, yeah, so is build root support some kind of packaging for OCA compliant and Images it does not handle OCI
25:41
Nobody did it as far as I know Yeah, basically I Think you can just take the table and put it in the container and you're probably again I I don't know CI and then containers. I don't know well, so if you can answer it
26:01
It's a good question for me. I'm glad I think that's People who doing this on Choctaw, that's how they do it, okay? For economy
26:21
Then then you need to do the light container like you just described And then you just get the stuff you need and then you unpack that using docker or ever Use yeah, you can probably repackage the root file system into docker I I don't know enough of docker to do it I do it with and spawn because that's the one I know but it's the only one I know
26:47
Just the question because of Licensing and version Tracability do you check in your whole like for example build root? Repository with all the settings Sudan into git and that's it or how do you are so?
27:05
How there are various? Philosophy depending on who you're asking how we do it. We have a git repository where we have build root as a sub module okay, so we clone build group and we only We work upstream for packages that are in build root and for everything that is specific
27:24
And we think cannot be upstream we keep it in a separate git repository You're using some modules and some make file tricks basically so that means our sub modules We have over all the overlays all the scripts that are specific to our build and in the build route we have everything Upstream and that need to be upstreamed which is directly there everything is the
27:44
Check some by the git commit of the under one which also includes a git commit of build root Which includes all the files and build root which are all the recipes and each recipe contain a checksum for the source it downloads so you have
28:01
recursive checking of all the signatures I Have not checked the complete security model But there is no trivial way to go and inject you would have to inject it in the project then you would have to Trick all the automated build systems the project uses to check which are spread around the world. I don't know where they are to
28:22
miss whatever checksum you've put in there and and everything recursively, so it's Seems kind of solid to me, but I'm not a security expert, so I can't say much more than last question Not sure if that's the right question, but can you also use? Tactically linked binaries to reduce the size of the image so rather than the the light container yes
28:47
Build root has somewhere in the compilation of options something to have aesthetically linked libraries cave at Most software are not tested with with status linked libraries So sometimes they break
29:02
All right, but build root knows how to do it alright Thank you very much, Jeremy