Bootstrapping the Debian/Ubuntu arm64 ports

Video in TIB AV-Portal: Bootstrapping the Debian/Ubuntu arm64 ports

Formal Metadata

Title
Bootstrapping the Debian/Ubuntu arm64 ports
Title of Series
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2014
Language
English

Content Metadata

Subject Area
Abstract
Bootstrapping a new Debian port of 18,000 source packages is always a massive job, and on average happens every year. This year it's arm64. This talk covers how such a port is done, and the work of the last two years (on multiarch, cross-tools, dependency-anaylsis tools and packaging infrastructure) to make this a repeatable and largely automated process, rather than an epic labour of hackery and bodging. It covers specific issues of the arm64 port, such as no hardware existing yet, and the current state of play.
Loading...
Point (geometry) Server (computing) Arm Information Multiplication sign View (database) Bit Mereology Software Addressing mode Different (Kate Ryan album) Personal digital assistant Network topology Freeware Booting Computer architecture
Degree (graph theory) Mobile Web Server (computing) Arm Real number Virtual machine Computer Arm
Laptop Server (computing) Netbook Arm Real number Keyboard shortcut Virtual machine 1 (number) Line (geometry) Solid geometry Computer Arm Software Cube Hard disk drive Cuboid Game theory Laptop
Laptop Mobile Web Trail Arm Causality Structural load Real number Computer hardware Operating system Computer Host Identity Protocol
Point (geometry) Asynchronous Transfer Mode Arm View (database) Weight Source code Binary code Device driver 32-bit Arm Revision control Floating-point unit Bootstrap aggregating Befehlsprozessor Operating system Computer architecture Fingerprint
Point (geometry) Building Arm Code Length View (database) Multiplication sign Virtual machine Bit Online help Arm Virtual machine Number Kernel (computing) Bootstrap aggregating Different (Kate Ryan album) Normed vector space Video game Alpha (investment) Compilation album Computer architecture
Wechselseitige Information Arm Source code 1 (number) Linear programming Arm Equivalence relation Bootstrap aggregating Loop (music) Programmschleife Bootstrap aggregating Green's function Alpha (investment)
Binary code Library (computing)
Point (geometry) Source code Distribution (mathematics) Building Multiplication sign Structural load Source code Keyboard shortcut Interior (topology) Formal language Bootstrap aggregating Root Quicksort
Software engineering Arm Building Multiplication sign Virtual machine Bit Medical imaging Process (computing) Personal digital assistant Hacker (term) Computer configuration Computer hardware Computer hardware Order (biology) Chain Quicksort Endliche Modelltheorie
Installation art Standard deviation Building Email Multiplication Multiplication sign Forcing (mathematics) Bit Hand fan Revision control Mathematics Different (Kate Ryan album) Personal digital assistant Normal (geometry) Utility software Office suite Library (computing) Computer architecture
Building Angle Profil (magazine) Personal digital assistant Building Poisson-Klammer Configuration space Bit Quicksort Rule of inference Communications protocol
Installation art Multiplication Arm Computer file Multiplication sign Combinational logic Revision control Emulator Mathematics Bootstrap aggregating Process (computing) Different (Kate Ryan album) File system Quicksort Computer architecture Library (computing)
Installation art Point (geometry) Multiplication Building View (database) Moment (mathematics) Revision control Inclusion map Latent heat Routing Fingerprint Multiplication Computer architecture Library (computing) Exception handling
Installation art Building Arm Building Electronic mailing list Set (mathematics) Subset Revision control Process (computing) Personal digital assistant Normal (geometry) Right angle Fingerprint Library (computing) Exception handling Computer architecture
Installation art Point (geometry) Scripting language Multiplication Arm Proper map Arm Category of being Film editing Bootstrap aggregating Root Profil (magazine) Chain Order (biology) File system Imaginary number Library (computing)
Point (geometry) Building Arm Code Building View (database) Patch (Unix) Closed set Bit FLOPS Line (geometry) Arm Entire function Revision control Video game Pattern language Data type
Point (geometry) Standard deviation Multiplication Repository (publishing) Patch (Unix) Multiplication sign View (database) Quantum Streaming media
Scripting language Standard deviation Slide rule Standard deviation Building Link (knot theory) Patch (Unix) Multiplication sign Set (mathematics) Bit Computer programming Number Revision control Cache (computing) Repository (publishing) Configuration space Library (computing) Computer architecture
Distribution (mathematics) Mapping Computer file Source code Mereology Variable (mathematics) Number Bootstrap aggregating Process (computing) Repository (publishing) Software repository Chain Configuration space Quicksort Table (information) Fingerprint Local ring Computer architecture
Medical imaging Parity (mathematics) Binary code Source code Order (biology) Chain Set (mathematics) Hill differential equation Computer architecture
Revision control Email Mechanism design Bootstrap aggregating Process (computing) Profil (magazine) Set (mathematics) Game theory Mereology Computer architecture Compiler Library (computing)
Email Arm Multiplication sign Patch (Unix) Source code Binary code Bit Revision control Differenz <Mathematik> Bootstrap aggregating Personal digital assistant Software repository Kernel (computing) Multiplication Computer architecture Physical system
Web page Point (geometry) Building Common Language Infrastructure Multiplication sign Source code Bit Type theory Bootstrap aggregating Root Personal digital assistant Repository (publishing) Chain Flag Pattern language Routing Computer architecture Exception handling Library (computing) Physical system
Installation art Building Multiplication Installation art Building Multiplication sign Bit Number Inclusion map Mathematics Integrated development environment Computer configuration Profil (magazine) Software testing Computer architecture
Demon Scripting language Mobile app Building Installation art Code Building Patch (Unix) Multiplication sign Source code Bit Complete metric space Revision control Bootstrap aggregating Process (computing) Loop (music) Computer configuration Repository (publishing) Profil (magazine) God Computer architecture Exception handling
Demon Installation art Mobile app Multiplication Computer file Source code Binary code Electronic mailing list Mathematical analysis Type theory Repository (publishing) Function (mathematics) Fingerprint Computer architecture
Data Encryption Standard Mathematical analysis Electronic mailing list 1 (number) Function (mathematics) Window
Bootstrap aggregating Uniformer Raum Profil (magazine) Source code Binary code Order (biology) Keyboard shortcut 1 (number) Set (mathematics) Limit (category theory) Library (computing) Formal language
Source code Building Multiplication Arm Information Java applet Structural load Computer-generated imagery Keyboard shortcut Set (mathematics) Bit Canonical ensemble Software bug Process (computing) Personal digital assistant Repository (publishing) Profil (magazine) Chain Bus (computing) Software testing Fingerprint Library (computing)
Point (geometry) Scripting language Source code Email Multiplication Thread (computing) Arm Trail Computer file Haar measure View (database) Computer-generated imagery Source code Electronic mailing list Planning Parameter (computer programming) Translation (relic) Mathematics Personal digital assistant Configuration space Right angle Fingerprint Computer architecture
Laptop Type theory Medical imaging Bootstrap aggregating Trail Computer configuration Profil (magazine) Order (biology) Tangible user interface Disk read-and-write head Booting Multiplication
Email Building Matching (graph theory) Arm Trail Multiplication sign Patch (Unix) Planning Numbering scheme Bit Translation (relic) Tangible user interface Revision control Mechanism design Kernel (computing) Network topology Chain Fiber bundle Object (grammar) Computer architecture Library (computing) Physical system
Point (geometry) State observer Building Multiplication Arm Trail Virtual machine Tangible user interface Medical imaging Bootstrap aggregating Network topology Order (biology) Computer hardware Hill differential equation Object (grammar)
Medical imaging Uniform resource locator Arm Multiplication sign Electronic mailing list Quicksort Canonical ensemble Disk read-and-write head
Revision control Execution unit Arm Computer file Hash function Different (Kate Ryan album) Multiplication sign Order (biology) Electronic mailing list Right angle Computer architecture
hello you will hear me is that about right okay good good so yes this is the arm 64 port for debian and ubuntu sadly it's a little bit of bun tree at the moment but we'll come to that in a minute from my point of view sound difference josh has covered the theoretical side of an important part of this I shall now cover the practical part and try not to repeat too much wow so just in case you don't know who I am my suspect coil people here do I've been doing free software stuff for a long time and I'm debian for a fair while now and since i joined linaro I've been forced to do a bun two things as well which actually turns out to have some advantages there's some things that are quite a lot easier to get done in a bun too if canonical actually want it doing or leas quicker than in debian so especially when Debbie ins in fries so I'm 64 how many of you went to the fedora I'm 64 talked earlier hardly any right yeah we don't talk to those people terrible if you'd been there you would have had a lot of information about what arms 64-bit architecture actually is and why you might care and how it's all for servers and is very cool I'm not going to cover any of that stuff you're just supposed to know there's a new architecture it's shiny and we need to make it work a little bit about what's involved with getting that done the initial boot strap that was done inside arm and the work when they're doing in public and how far we've got with that
so um 64-bit ARM that sounds like crazy talk why would you want that I guess again Andrew spoke about that earlier and you know to a large degree it's about servers and desktops and real computers Army's most people if anything to do with it in last 10 years it's kind of been about phones and Reuters and mobile stuff but in fact that's not
always been true there have always been
when I started getting involved with arm my desktop machine was one of these wrist pcs back in 994 it went surprisingly fast for a 16 megahertz arm it's depressing how much your desktop hasn't really got any faster since then despite astonishing increases in the numbers and yeah these
days you can buy a nice little but much less missing isn't it nevermind so solid run cube which is a cool little box that came out last year which is a desktop machine it's about this big you plug a keyboard and monitor and a hard drive in got real desktop computer you can run debian ubuntu on it and similarly for servers much of the early arm porting work was done by the rebel people for the net winder which was a network arm box based on strong arm and you know 10 years 12 years down the line we now have proper server looking boxes to stick things in that's the 32-bit one 64-bit ones will be along any minute now I'm assured nobody tells me anything about exactly when hardware's available but I think it's fair to say that by the middle of this year there should actually be something you can run things on probably won't work very well and maybe by the end of this year there'll be real stuff again desktops laptops
have been on actually as new as you think they are armed ones the Scion were a long way ahead of the game back in 2003 with the original netbook I even tried to claim the netbook recently after everyone else had used it a lot and they obviously haven't sold any so we have linux running on that back in 2004 and sadly it never got released but it was very cool bit slow I think that
was the main reason I never got released and you know now we have shiny show any laptops like the Chromebooks just come out which is very nice hardware handlers you can't cause notice the genesee hardware so yeah it's not all about
mobiles it never has been obviously Debian particularly has never really been the world's finest mobile operating system so we're quite interested in what you can do with real computers and the whole load of arm-based real computers are coming any minute now so this is the history of arm ports just to keep track of all this the original port again basically what
the net wonder people did is the old ABI for v3 architectures which has since been discontinued we don't care anymore nobody cares anymore there was we tried to mention it a very short-lived big-endian version of that which was only because the linksys slug had a big endian binary blob ethernet driver it was much easier to build the whole the rest of the operating system big-endian then it then it was to solve the blob and to be fair they fixed the blob a year later so he could throw all that away it now exists exists in source to confuse people because they naturally think the army be must be the opposite of army l but no in fact it's completely different API so since then we had the new API which was bootstrapped especially booze draffen about two thousand six it was until two thousand nine it got released more recently everything's moved on to v7 architectures where you actually expect to have a floating-point unit which is what I'm HF is about to amend the audience who don't work here over there no the joy of bootstrapping and there we have I'm 64 coming along any minute now 64-bit a yet another instruction set architecture although to be honest apart from the 64-bit pneus and the entirely new instruction set from the 32-bit point of view it looks extremely similar to the v7 stuff of this stupid name so
your point of view I'm 64 a art 64 and V are all the same thing you don't care and arm care and they really don't want to click on 64 and but the colonel people notice that the name they did want to call it a arch 64 was a pretty stupid name and so the colonel people said now view sucks and the Debian is going to do the same so so we're calling
an arm 64 sadly the Fedora people have called the air 64 because they're all brown noses so which is a bit unfortunate actually because now from users point of view this is essentially the same thing is going to have two different puppet names and but there you go
another live in the length of year just because we're talking about this subject it's very confusing unless you do this for a living build architecture is the machine you're building on the host architecture is not the host your building on is the machine you're building for and target only applies when building compilers is the code that B is being generated so when I say host I mean the target architecture as it were I think you're trying to build for let blame the no people it's all very sad and the number of things where they've done it wrong the colonel people have got it wrong if you look at all the build stuff in the kernel that you they use host where they mean build it doesn't help anybody but that's life Josh already mentioned this bootstrapping is thought of as a very rare thing it always was until we notice that actually we do it every year and it will be nice to be wasn't quite so hard so I'd okay as I say once I'm 64 is done I certainly don't care anymore and it it's in some ways nice and another slightly unfortunate that linaro doesn't care anymore starring about now because we've got something that works using an embedded and so I've been doing this a long time and they've been paying for that and that's very nice but they quite like me to stop doing this and do something more useful and so I'm trying
to get as much of this done before I really have got to stop yeah well there
is that here or possibly the x32 equivalent for arm which she'll know how to help be along in a couple of years just watch that's true exactly and but
that was probably recipes to bootstrap yes so bootstrapping is a pain because of Bill dependency loops now Joshua already showed you basically wise works but I'm quite pleased with this little things i'm going to show you anyway because it's shiny and so yeah that is a typical loop from a while ago but you know involves about 20 packages and green ones are source and the white ones
of binaries go on do something yeah doesn't work so well of it doesn't there we go so yeah
and also build this lets you want to build poplar but you can't build poplar without installing that which comes from the Qt and you guys install that until you build this which comes from the by odbc and you can't build that and lets you install lib gtk2 and you can't build that until you build cups which gets you back to Pablo and you go bollocks so
yeah and you think why do I need some of these things so the point is that Debian is is maximally configured you know we turn everything on that's sensible which and there's a lot of language bindings so there's all sorts of reasons why a whole load of stuff is needed which isn't really needed needed and it's just sensible for a binary distribution now but that's exactly the opposite of what you want when you're starting off from nothing so that's why this problem is basically not really a big problem in openembedded and build root and bass rock and all sorts of sensible built from source every time distributions but if you've got a binary distro where people expect to have their stuff already built with everything turned on
so it works and the bootstrapping is awkward so as has already been explained
the process for the last ages has basically been build an image you can build on with something else such as bill drew or OE I think it's been REE generally for the last two or three ports for arm stuff at least and then just sort of bhajan hack and turn things off until you can build stuff now in the arm 64 case we haven't even got that
option because there isn't any hardware there's nothing to run it on so we can't try and natively build it all from something we have got models which since October but they are impressively slow it's a 10 megahertz ish sort of machine i think is the kind of thing you get well it's not slow slow but it will take a long time to build or debian one yeah there's m68k people laning so it will be nice if we could do this a bit more organized Lee and automated Lee and repeatedly and so on so clearly you have to cross build some of your stuff you've got nothing at the very least first you've got to get yourself a tool chain and and then you've got to get enough of an image that you could install that and build some stuff natively now that how much you do cross before you switch to native is up to you you could build the entire district across if you wanted to or you could build as little as possible we'll come on to how much that isn't a bit about 100 packages couple hundred and everything else native or anything in between whatever seems expedient and in order to do that you have to be able to linearize the dependencies so we need
to do some cross building how we going to do cross building and so for a long time so the two basic problems are your cross dependencies and then actually making builds work so we have various tools to try and work out whether a dependency was a cross dependency or native dependency so office basically libraries you need the host architecture version whereas all the utilities no make and stuff you just want the build
architecture version it just has to run and we have various tools for trying to decide which of these you needed or just installing both always and hoping that worked and d package cross which has been around for over a decade I think which is basically a tool for managing the paths you install things on so that you can install both a native version and a cross version somewhere else so they paths historically libraries used to go in use a triplet libraries and headers which is fine except that the path you're building against is different from the path you're finally going to install to and which makes your builds complicated and wrong and broken and gives libtool endless opportunities to screw it up so the nice thing about multi art so having done multi arch for an entirely different reason which was basically co installed ability of 32 and 64-bit libraries for x86 was the driving force and pretty much for free you get the ability to co install libraries for different architectures and headers for different architectures and you almost get the cross dependencies defined for you if you can Co install it then it was probably something you wanted to cross depend on and and basically just have to mark up the cases where that's not true and the really cool bit is that the paths don't change now from when you're building and when you're installing and when you're running there's just the path everything becomes canonical and this is really really nice I've become a huge fan over the last year of actually using this it's really rather nice and you don't have special packages for cross building either the normal packages installed in the normal paths and everything just is and the other thing we need is this
profile concept which is fundamentally pretty simple it's not anything very clever it's just saying that when I want to build this package I want to be able to do it without live SQL dev because that's optional so if I specify the
stage one protocol profile then not this package that's it now in fact the syntax is going to change it turns out almost certainly two square brackets instead of angle brackets but may and then in your rules file obviously you can do arbitrarily complicated things but in most cases it's something fairly simple that says if you set the bill profile to this then don't bother building that package and configure turns off this feature and that's that's the sort of genius work we do so a little bit more
detail and multi arch for people who haven't seen me which one about this several times already it used to be that libraries were just installed to use a lib and if you wanted you know fage architecture always went to the same place so you couldn't co install them because the names clashed you could only have one so by putting an architecture qualifier in so use a lib triplet you can now install them side by side fundamentally that's it and so you refer to packages with a package name colon architecture qualifier I said what you
said the file names though now don't change it means you can easily run things in place with qemu if you want to just install stuff if you got an emulator it can run it if it's something which will already run on the CPU you've got it'll just run so we sort of had this for years with lib 64 and lib but that was a really sort of cheap crap version which only did 32-bit and 64-bit so for example if you wanted arm 64 and x86 64 stuff some reason couldn't do that in multi watch you can you can have any arbitrary combination of things it should just work the big catch under some circumstances is that your versions must stay the same you can only install the same version of this library for each architect you can install different versions of different architecture in the same file system if you want to do that you need to put in a chroot and for a binary destroy where we build rebuilt everything that doesn't really matter we do that anyway it can be slightly awkward during the bootstrap process and in some other circumstances where you want to build the latest of everything you didn't want to build it against the distro versions so you add an architecture by saying deep package add architecture blah and I forget install package architecture name it's very simple to use this is the other half
which is the multi art specification saying which things are Co installable which things are basically we stinkers between libraries which Garko installable and tools where you don't care which architecture it well as long as it runs and we also have allowed for things which could be either depending on how you want to use it and practice
that's turned out to be not very useful and usually easier to split the package into two halves into the tool half and the library half and specifiers one of the others but we do have a few things most notably pearl at the moment is allowed and then that is from the point of view of whether the things go installable doesn't necessarily tell you whether the thing depending on it wants the same architecture or a different architecture usually it once the same but just occasionally a bill will need the build architecture version of a library not the host architecture virgin oil need both which is where we have these modifiers any in native which allow you to basically to mark up the exceptions I'm going this route to
be fast because I've already given that tall once and there's a fair amount to say so here's an example just in case this isn't making sense to anyone yet so for the S Lang package that's its normal set of build dependencies and basically you want these for you want the arm 64 version of and all those other tools at the top you want the build architecture usually x86 AMD 64 version of because you just want them to run and as you can see it's not obvious from the package names which it is so we used to have heuristics like if it ends in dash dev then it's probably a cross thing right I'll accept 4d package dev and auto tools dev actually because they're different and and similarly if it begins with lib we probably want a cross version except libtool and so that is
what ex debbie is it's just a great list of the thing and all the exceptions it does a surprisingly good job under the circumstances so the whole thing about this is that it's nice and neat and actually works properly and yet so here I just now you can just install stuff and cross stuff and you get your bill dependencies so yes we've always done a
bootstrap the Dewberry nearly two years ago now inside arm when maverick was cool and trendy which used ex dead because the multi arch stuff was way too immature at that point we didn't have a proper package tool chain we just had a magic binary tarball which was the existing a out 64 cross tool chain and the problem with that is that you'll find you and cut installs lots of things because everything depends on lipsy 6 or lib GCC or lib c++ so we had fake dependencies for all those things which the toolchain actually supplies but we didn't have real packaged binaries of and by a lot of hacking making things cross build taking out the bitch didn't need you know so doing all this profile
stuff manually by just chopping it out and working out what order you could actually build it in and writing some evil scripts they built a root filesystem good enough to send two people to get started on earlier sixty-four work so so this is cool so it's done already marvelous we haven't got any work to do what's the problem well the problem is that arm is a imaginary property yeah imagine property company so so I asked how rude I was
allowed to be about the legal department and and to their credit they said so long as I pointed out the arms and IP company first I could say whatever I liked so um so um so the problem here is that we have all these patches for stuff which basically are very exciting making random packages cross build and legal wouldn't let the people who done it send it to anyone and you go why not um the thing is that most flops licenses grant a patent license to the code and then they don't just grab a patent license for the four lines you added they granted patent license to anything you might want to do with this entire code base ever so legal go it does what you want to upset a patch to ssh to make you cross build that means we have to grant a patent license for everything we ever did to anything anyone would ever want to do with ssh ever that sounds a bit scary we'll have to look at that and basically they're busy and decided that they couldn't be bothered looking at that enough to actually enable people to send the code it so all this work was done but they weren't allowed to send it to anyone and the problem is from legal point of view this isn't even that unreasonable you know it would be a lot of work to check even though it's obviously not a big deal so anyway we've got to do it again properly so this this is the scary stuff that couldn't be allowed out you know if cross building set cross for like 50 packages but you know checking that that doesn't that that the package you put that in doesn't conflict with any of your pattern licenses for anything ever is actually very difficult so such is life so on the
one hand they used all the community stuff they did it using upstream methods and latest thingy and they asked me more or less now how should we do this and did what I said which was nice but then they were entirely unable to contribute the work they've done which is a bit crap but it pretty much illustrates the problem la renault has generally linaro tries to work in the open and we work with these great big corporates who fundamentally want to do everything behind closed doors so they can make big marketing announcements and it's a continuous struggle dealing with that culture clash so the actual bootstrap ah
yeah so I wanted to do this in debian because what it's just better isn't it but but multiple i have to file every patch twice that's the main incentive from my point of view but unfortunately when this really got going kind of middle last year we just about frozen so a lot of multi arch patches weren't in debian and were already in ubuntu so it was a hell of a lot easier to so i use quantal for a while mostly because it wasn't changing all the time and at some point that became painful and more things were fixed in raring so we moved over kind of end of november i guess so this time everything was done in public up streams to go along I so one thing I
should possibly not mention is the what I'm actually able to read the repository
containing all these things so some of the patches I've submitted may be surprisingly similar to the stuff that wasn't up streamed but I have a number of hats and the convenience of changing them can be quite useful sometimes to be fair nearly all of it was that exciting so it's not a big deal so use multi-arts this time because it's ready enough various things are still broken but it's actually pretty good and all the standard tools so rather than having a special tool for installing cross dependencies you just use apt and you just use d package to build things we still have d package across but not for its ability to manage packages but because it keeps the auto conf cache data for for things which you can't cash
settings that you can't discover without running a program so we just write down the answers and say use those and referee pro for the repository and s build we talked it about cross building and so this is really nice you now cross build things the same way you need to be build up my stuff put me on the slide we had a profile support and this useful invention which we should have done years ago for cross build-essential so basically that's just a meta package to install a cross toolchain see library of the host architecture and a cross version of package config which is a link partly because i think it's a good thing generally but also because there is no qemu forearm 64 I was keen to avoid anything which required qemu to run random scripts now a lot of crossing gets a bit easier if you got qemu because every time you randomly run something the wrong architecture it's quite likely to still work and but we don't have that luxury forearm 64 so the
process is fairly straightforward you need to prepare a local bootstrap repository for putting your hacked and baje stuff into or indeed your perfect and absolutely correct stuff but there still isn't anywhere in the mother ship distribution yet to put things so you need a repo d package is the thing that tracks new architectures you just add
the new architecture to a table and then everything that does anything about architecture stuff always ask d package architecture which knows about the mapping between new names and Debian names and Colonel names and any other names we might need set up at route and make your tool chain work painful part sort out make sure that D package Cross has autocom variables for your new target architecture you've got across build essential package listing the new tool chain packages and importantly auto conf has config sub and guests files with the new architecture in so that everything can find it and say the same thing configure your token physic new config and then build stuff so how much stuff exactly well Josh showed you approximately how much stuff this is numbers from a while ago but as far as I can tell this almost exactly right so on my table of finished and not finished things there's 65 source packages and 12 source
packages which only produce architecture all binaries i-77 source packages in total which need to be built and only 65 of those for the target architecture that is to make a well that's the set of packages which actually end up in a chroot with build-essential in it and you actually need to build quite a few more that's where the hundred and forty odd number comes from for all the things which those depended on in order to be
built so far as i can tell ran over 140 packages gets you a image i've got 128 so far till chain this is the classic
three-stage bootstrap for toolchain you need set of Linux headers for the new architecture a really basic c compiler you can build against that that enables you to build a really stupid lipsy then you can build a secret part of that actually knows about the Lib C and then you can build a proper lipsy apart from
the selinux part you haven't got that yet and then you can build a full version of GCC um this has always been fun games it's currently that process is automated by a cross toolchain base package which is largely marcin's yeah okay I've got hecklers I I believe this can be managed into the profile mechanism so that you don't actually need a third party package running the build process but I haven't got it working yet so that's how it's been done so the way that works is in order for
that package to depend on the stuff it needs they need to be binary source packages of the sources you need so the Linux or Billy until sauce need you tipsy source and GCC source which are installed bunged a bit hatched a bit and then built through that three-stage bootstrap it's always awkward for new architecture for the reasons will always vary for your given bootstrap case you know in this case the arm 64 support that arm done was against Colonel 3.7 whereas the colonel in quantal I was building it at the time was 3.5 and he go near but it turns out actually all you're trying to do is make the headers package and it's really not that fussy about the details so as long as you can get a Linux source package out of it you know ninety percent of the stuff doesn't matter whether the colonel works or any of that rubbish you just need to get headers package out with
some stuff for the right architecture so it's a bit of fiddling but it's really not very problematic worse was EG lipsy which again the support was done for EG lipsy 2.16 with the version in the distros 2.15 or if you're doing in debian 2.13 and that is kind of painful but it turned out that the 2.17 support was in experimental and a git repo somewhere so actually i was able to select the 2.6 was in experimental and i was able to take that and apply the patches and kick it a bit and get it to work except that that only builds with GCC 4.6 you haven't got a GCC 4.6 you've only got a GCC 4.7 so you have to find out whatever the obscure reason is why it didn't work with 4.7 a diff I've been specifically set to not use it I've forgotten now but we got it I got it going eventually so there will always be a certain amount dicking about like that for new architecture it's annoying it's not helped by the fact that the GCC and EG lipsy packaging in debian and ubuntu caters for debian and ubuntu and linaro and 12 architectures and a couple of operating systems so it's this it's not really one source packages like 17 packages all managed into the most amazing patchset doctor is a pretty scary guy so you have to understand how all that works for you do anything
saying a picture root is essentially trivial if you go to this web page it will tell you what to do you install s build and type s bill create your route a path a distro name and that's it that'll make you so that's just an x86 to root for the your build architecture
in which you can install cross build-essential and carry on for out 64 and generally any new architectures will be some bits missing so in this case there is no lib SSP and therefore stack protection so you have to turn that off but that's okay we have the package bill flags these days which is really nice and everything I've used about one package year of 100 and something didn't use it so you just set you tell d package bill flags to noble that flag every time anyone gives it and in your chroot and it just works it's great and for the initial toolchain the the library search path was wrong as well so we had a library search pattern Dobler in here which also worked pretty well the in prob with that is that it always applies so you can build everything except packages which need to also build some build architecture tools at that point they get given the host architecture path and it falls over and dies so you can't do things where you've got to build for both architectures without having a tool chain that actually uses the right paths but the that got is quite a long way also you want to tell the system to prefer the packages from your bootstrap repository which have profiled builds and things so that you don't get the wrong source so
building is now cross building is now very simple you apt-get install cross build-essential architecture name and you get build dependencies dash a architecture for whatever package you want to build and assuming the multi arch foo is all correct which for a large number of things now it is i think in raring you can now build 496 packages out 1,200 just by doing that that's pretty good change into the package and
run d package bill package a architecture so cross building is almost the same it's congruent with normal buildings really quite nice now in practice this will not work quite a lot of time because for example since it runs the checks it'll all fall over so you always want to run with Deb build options equals no check now we could have put that ND package bill package but we can conceive of circumstances where you might actually want to run the test even though your cross building for example if you've got an emulated environment or in fact if your tests aren't architecture dependent so we left that as something which you can still enable if you need it similarly you need to tell also conf if it's not a comfort package to go and use the auto come from magic runes to set it up so in practice you always take this and that's a bit dull so actually you just s builders both of those things automatically so you just do s build blah gone off the bottom but that says dash dash host equals architecture name so s build will basically do all this for you and that's very nice if you want to do a profile build you set dead build profile before
running deep package bill package or you can use a dash p option now this stuff is not yet in the main all everything so far is in the main distro this isn't yet because we're still argue about syntax um but if you use my bootstrap repository there's versions that do these things or the linaro cross toolchain bucket as of last week Josh wrote a very handy patch to apps so the apps now understands a a build profile option I glad I did that because I looked at the code and went oh my god I've don't understand c++ at all it's complete mystery to me even after someone told me which file it was which function it was they said this is the bit you need to change and you go no no II don't know what to do we find someone cleverer please so yeah so this is very
nice an S build will pass on a profile which means these two things happen and I got a bit bored of typing all this stuff after a while so I've written a very stupid script called dim strap which just closes the loop of updating from source building something signing it upload running repro process incoming and then doing the next package because because always works you know generally you're building something because you need it as a bill dependency for the next thing so you've got to put something back into an apt to repo each time before you build the next thing so yeah here's my little chart of this is actually the cross build demon we were running for an existing architecture well you're much more likely to already have the build dependencies but in fact it's exactly the same process for dim strap except
there's not a demon you just type dim strap package name or give it a list so this works for everything except where the multi arch dependency the app get billed dep dash a architecture step fails which is still true for quite a lot of things especially anything that needs pearl these are just the runes you
type to do a dependency analysis using those dead bill check which is josh's tools so you basically say use these source repositories and binary repositories about binary package files in fact and source file listing the sources you care about and then you can just you can get it to give you an enormous list of all of these packages and i'll just so whether it's currently buildable or not whether it's bill dependencies and install dependencies
are available or you can just run it for one package with DES check only which you'll say can I build this and you get a little thing saying no you can't build it because you haven't got that yet now in practice you don't need is theoretical analysis much for the chroot step because it's such a short list you can just try building it see what went wrong and which is how I've actually done most of the work now we might have a window in here here we are so in fact
this is the output of running effectively s build on all of these things so except for the ones that say manual build that was when s build would have failed to install the dependencies but I could install them manually and then carry on and as you can see this has gone pleasingly green and over the last couple of months and we're down to about five things that haven't built now we don't actually know whether
they work or not that's a different
problem so this is the set of bill dependencies that I had to nobble in order to build the 65 binary source packages 65 binary producing source packages for about 60 for bootstrap so EG lib see these days ones live selinux i think it won't slip something else as well actually which obviously you haven't got to start with so this shows you this is what most profiles are about it's generally language bindings so you know swig and Ruby bindings for live selinux which obviously are things you
really don't care about at this stage and similarly python bindings for things like crack lib python bindings Debus python bindings and so on java bindings joy d bus is a bit fiddly the whole d-bus g lib thing in fact i was pleased to find that if you look in the packaging for this there's a whole lot of comments saying if you need to bootstrap this just a noble these four and you go wow and a few things i think Leba silly legs already had staging information in it it's great so and during this process canonical have actually done quite a lot of work on cross building arm hf so our load of things got fixed in the last few months so that the package does in fact ross build which means that i can expect it to work in the arm 64 case as well so that's that's the set of stuff you need to change in the base thing so I not that many and often in fairly simple ways so have I over gots as I say
because of the Debian fries and they car 200 pending multi arch bugs it was much easier to do this in a bun too so that status they've all been talking about is basically the raring status cross tool chain is available to install and it works and the cross support stuff exists as I say if you want the profile support you need my special repository and we have basically done the work of
multi arching Perl and Python Python as far as i can tell works Pearl has been a very abstruse argument on the pill mailing list about whether it's actually right or not for all conceivable cases of pearl API changes and as you upgrade things from one pearl API to another and to be honest I don't actually understand the argument anymore I don't know whether we're finished or not it seems to work from my point of view except that because pearl is allowed you have to say pearl coal on any all over the place in packages that depend on it which is boring but simple and we haven't done very many of those which is were quite a few things need a little tweak to actually build so the question was is there a plan to make pearl not allowed and if there is a thread on the pill mailing list which is a very abstruse argument about exactly how it should be multi arched and split up and it's not entirely clear what the final outcome of that will be I think we need to use it for a bit and see where their anything's broken about what's been done it appears to work I managed to build an excess package and you can build normal stuff and you just need to specify call any to install things as far as I can tell that's sufficient I guess we're going with that for now and and if we find there's a problem later we'll have to change things obviously the architecture all packages we basically down to worry about because we can use
some other architectures that already got built as I said we've got 128 source packages built for arm 64 and 72 of the 77 that I need to install to make a chroot built and so I reckon it's about 10 more packages to do I was hoping to get it done for now so I could show you a demo and go ahead and it's done but yeah I blamed DB which was a pain in the ass talk about we hmm so the only thing is right I've built stop a build engineer right I haven't run any of this it's probably all wrong not least it's built with lipsy 2.16 early and like we've discovered that there's some fairly serious breakage in there so we almost certainly to rebuild it with 2.17 anyway with all the multi lunch foo that is just a manner of doing s-sorry dim strap list packages or we can almost you know scrip to rebuild now which is quite nice well we can script to rebuild the scripts probly not quite as short as we'd like and yes so yeah those are the things that still left so I'm just get trying to get pearled cross build so thanks to Niels fine work over there there's now a helper package debian pearl cross i think phil crossed debian which basically keeps the enormous config foo files which pearl uses to work out how it should cross itself and i've made one of those but it didn't actually build some tedious reason i think that will take long new plug needed something or other so this is
really handy because you just go and look and go so it's ldap so and pretty sure that's an optional feature of new PG so we'll just add a profile to say no I don't want held up and that should be done Plymouth abit of a pain because it wants Cairo which once lived free type which once live fontconfig which once live xcb which was live x11 either crowd
sure the order in there but I don't want all that crap so I'm pretty sure we can turn off the ex feature of Plymouth which point it should get rather easier to build at you could just not install this as well yeah I think there's some problem there though or does that actually work yeah I think you'll find your Bunty will stop waiting for Plymouth to display something you you could probably kick it in the head yes yes we could said so you could almost certainly make you don't need notebook to boot either so you could already make an image from the packages we got and see whether it works with a little bit of jiggery-pokery what else of note to it so yeah one thing that's quite tiresome bootstrapping against a moving architecture like unstable or currently raring is that
cross build-essential expects to have the matching version of lib GCC and the next Lib C dev and Lib C of the build architecture so every time someone uploads a new g lipsy or a new GCC or a new kernel suddenly you can't build anything because cross build-essential won't install any o Dole and you have to go back and rebuild those packages now that's trivial for the colonel headers and trivial for the colonel headers and it's not too bad for the GCC stuff that's it to be ok was with the pain for eg lipsy so for example we just had a new EG tipsy 2017 great i wanted one of those doesn't cross build does it so I'm
currently screwed and can't build anything until I fixed that and that was because it only actually builds because there's some patches in the cross based tool chain foo which should be upstream in the main thing so there is there is a bit of a pain that tracking feature obviously in a release distro this doesn't apply and it hasn't caused me much aggravation it's just every so often you notice that everything failed today the one thing we've realized is missing from all this beautiful technology is when packages express it for an explicit dependency on something a tool chain like GCC 4.6 the eg Dipsy needed and nothing in the system says oh actually we're cross building you actually mean cross GCC 4.6 so it just says I can't install GCC 4.6 colon arm 64 now I give up so we need a mechanism for that and we have a plan calling Watson scheme gone up but that will delay all this being totally beautiful finished for a little while the other thing there are things further up the tree which are going to cause us pain object introspection is used a lot in
the whole genome food tree and certainly for maverick you get away with just turning it off everywhere and it still worked I suspect that's not true anymore and the only way to make it work is actually to write a cross object introspect er which will know what to do so we'll hit that at some point but we don't have to do that in order to get to a native machine so we'll have enough to make a build image which we could put on real hardware and then just build everything natively so you don't need that but you would need that to be able to cross more things which is probably useful just a little observation of this bootstrap partly because a lot of this this isn't you you know Debbie nearly never really worried much about cross building and the multi art stuff is new most of the work is not in arm 60 for specific things so how far are we now from having enough packages to run the bootstrap 344 I guess actually depends
with you kick Plymouth in the head ok that will be nice to publish when that happens so people can actually try it then have a debian CA truth available on a bun to chroot boynton we're about 65 packages away from a debian chroot so yeah is very nearly done I think you never know quite now until you've actually finished how many more things that all i could say plymouth sort of implies at least ten more things some of which I know don't work pretty much out
time but that's okay we're about done yeah so you can all your own arm 64 stuff using the open embedded image which linares released so that works if you need to run stuff we want to try your stuff that's fine but obviously if you want a real distro then helping fix cross failures is a jolly good thing there's a raring list and there's a raring arm hf list the Canonical's now maintaining and there's my arm 64 list and there's URLs here and that's it
pretty much used up all my time so anything anyone has a burning desire to know just on that on the pill stuff
it is probably just we need to calculate some of the values that arm 64 needs to provide for pearl which we haven't actually done because they're not needed for order come for anything else yet they are pills Pacific so I copied the another 64-bit architectures list and then put a few arm things in yeah as also the embedded version which we can compare with so yeah I've got a file which might be correct it's actually it for one define it stripped the hash off the beginning and then said understand define and baft yeah that that's because the same problem is you actually need to have a version lock if I config file so you gotta make sure you start with the right version in the first place you can't use files from all versions right yeah I spent four hours rearranging the order of all the things because they come out completely different from different pearl versions for no good reason so when you try and if your file you go oh it's totally different but it ends in fact it's almost exactly the same in a different order hateful hateful thing I think I should probably stop unless there's anything important thank you very much
Loading...
Feedback

Timings

  769 ms - page object

Version

AV-Portal 3.21.3 (19e43a18c8aa08bcbdf3e35b975c18acb737c630)
hidden