Project Builder: A GPL continuous packaging solution
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 97 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45689 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Continuous functionOpen sourceSoftware engineeringIntelBuildingSystem programmingConfiguration spacePressureMechanism designSlide ruleNumberSoftware engineeringPhysical systemXMLUMLLecture/ConferenceComputer animation
00:30
Software engineeringIntelBuildingSystem programmingConfiguration spaceOpen sourceSmoothingDisintegrationGraphical user interfaceIntegrated development environmentTorusSystem administratorEvolutionarily stable strategyDistribution (mathematics)Metropolitan area networkData recoveryOpen sourceSoftwareBitMultiplication signDistribution (mathematics)Lecture/ConferenceComputer animation
01:15
System administratorDistribution (mathematics)SmoothingDisintegrationGraphical user interfaceIntegrated development environmentPressureOperating systemSoftwareDistribution (mathematics)Computer filePhysical systemSystem administratorLecture/ConferenceComputer animation
01:50
Distributive propertyReal numberRepository (publishing)SmoothingINTEGRALDistribution (mathematics)Configuration spaceNumberMathematical optimizationSystem administratorInstallation artMultiplication signSoftwareShared memoryPhysical systemLecture/Conference
02:36
System administratorDistribution (mathematics)SmoothingDisintegrationGraphical user interfaceIntegrated development environmentCore dumpDistributive propertyRecurrence relationSoftwarePhysical systemRevision controlConfiguration managementState of matterOcean currentComputer animationLecture/Conference
03:17
Software developerIntegrated development environmentComputer fileCurvaturePressureInstallation artComputer fileWebsiteConfiguration managementFile Transfer ProtocolPhysical systemSoftwareFile formatType theoryIndependence (probability theory)Integrated development environmentSoftware engineeringProcedural programmingCategory of beingComputing platformDistribution (mathematics)Cartesian coordinate systemComputer animationLecture/Conference
04:15
ExistenceIntegrated development environmentContent management systemCurvatureComputer fileNetwork topologyAreaPhysical systemDistribution (mathematics)Transport Layer SecurityConfiguration managementTexture mappingModulare ProgrammierungMultiplication signResultantVideo gameLecture/Conference
04:47
Integrated development environmentComputer clusterContent management systemCurvatureComputer fileTerm (mathematics)Revision controlPhysical systemDistribution (mathematics)View (database)Multiplication signCartesian coordinate systemProduct (business)EmailSoftware testingLecture/Conference
05:47
Continuous functionSoftware testingOverhead (computing)Maxima and minimaConsistencyDistribution (mathematics)DisintegrationContent (media)Distribution (mathematics)Well-formed formulaState of matterSoftware testingSoftwareQuery languageDatabaseInformationProcedural programmingMereologyCartesian coordinate systemBitMultiplication signComputer filePhysical systemNetwork topologyComputer animationLecture/Conference
07:24
Continuous functionOverhead (computing)ConsistencyMaxima and minimaDistribution (mathematics)DisintegrationContent (media)Software testingInternet service providerMetadataComputer fileSoftware testingControl flowOverhead (computing)InformationTerm (mathematics)DatabaseCASE <Informatik>SoftwareMaxima and minimaDistribution (mathematics)INTEGRALComputer animationLecture/Conference
08:55
Computing platformPhysical systemPoint (geometry)Unit testingLevel (video gaming)Functional (mathematics)Latent heatSoftwareCodeSoftware configuration managementCycle (graph theory)Function (mathematics)Classical physicsExecution unitDiagramProgram flowchart
09:27
Computing platformMultiplicationLogic gateSystem administratorSoftware developerSoftware testingSoftwareInformationMetadataPhysical systemServer (computing)DisintegrationType theoryNumberComputing platformInformationLevel (video gaming)File formatDistribution (mathematics)Proof theoryValidity (statistics)Line (geometry)EmailAddress spaceDifferent (Kate Ryan album)Descriptive statisticsGroup actionUnit testingPhysical systemMereologyOperating systemPoint (geometry)Shared memorySoftwareData managementMetadataVideo gameSystem administratorRevision controlCycle (graph theory)Software testingINTEGRALMultiplicationSource codeParallel portSeries (mathematics)Lecture/Conference
11:46
Physical systemSoftware testingCodeSoftwareInformationMetadataSystem administratorServer (computing)DisintegrationSoftware developerUsabilityStreamlines, streaklines, and pathlinesPressurePRINCE2Physical systemSoftware testingServer (computing)Revision controlMultiplicationDirectory serviceOpen sourceMultiplication signCartesian coordinate systemDistribution (mathematics)SoftwareData managementComputer animationLecture/Conference
12:52
Software testingPhysical systemSoftwareCodeInformationMetadataSystem administratorServer (computing)DisintegrationSoftware developerUsabilityStreamlines, streaklines, and pathlinesLangevin-GleichungData managementPhysical systemScripting languageSoftware engineeringCartesian coordinate systemMultiplication signWritingSet (mathematics)MetadataOrder (biology)DampingComputer animationLecture/Conference
13:40
Content management systemRepository (publishing)Source codeBuildingIntegrated development environmentPressureLocal ringNumberControl flowConfiguration managementTerm (mathematics)Computer filePhysical systemMultiplication signDistribution (mathematics)Repository (publishing)Revision controlElectric generatorBuildingCombinational logicDifferent (Kate Ryan album)Subject indexingLecture/ConferenceComputer animation
15:00
Content management systemRepository (publishing)Source codeIntegrated development environmentBuildingLocal ringRevision controlSound effectData managementPhysical systemFile formatVirtual machineIntegrated development environmentComputer animationLecture/Conference
15:32
Content management systemIntegrated development environmentLocal ringBuildingRepository (publishing)Source codeDistribution (mathematics)Physical systemVirtual machineBuildingLecture/Conference
16:03
Content management systemBuildingIntegrated development environmentLocal ringRepository (publishing)Source codeType theoryVirtualizationMechanism designIntegrated development environmentMereologyEvoluteVirtual machineVirtual realityQuicksortContext awarenessLecture/Conference
16:53
Content management systemRepository (publishing)Source codeLocal ringBuildingIntegrated development environmentComputing platformMetadataoutputConfiguration managementSummierbarkeitPhysical systemComputer fileLecture/ConferenceComputer animation
17:35
Computing platformFingerprintCross-platformCycle (graph theory)Food energyConfiguration spaceSet (mathematics)Physical systemOperating systemLocal ringSource codeRepository (publishing)MultiplicationMaxima and minimaCodeIntegrated development environmentType theoryCartesian coordinate systemSystem callLibrary (computing)Control flowComputer fileDifferent (Kate Ryan album)Point (geometry)Service (economics)Program flowchartLecture/Conference
19:08
Computing platformSkeleton (computer programming)Validity (statistics)Data managementMacro (computer science)Variable (mathematics)Physical systemServer (computing)Electronic mailing listProcess (computing)Software testingDisintegrationPatch (Unix)Revision controlPressureExistenceVirtual machineInformationSlide ruleAdditionMetadataSet (mathematics)Repository (publishing)MiniDiscPhysical systemDistribution (mathematics)Multiplication signEvoluteProgram flowchartComputer animationLecture/ConferenceMeeting/Interview
20:31
Validity (statistics)ExistenceMacro (computer science)Physical systemSkeleton (computer programming)DisintegrationDistribution (mathematics)Stability theoryIntegrated development environmentReading (process)Revision controlMedical imagingLecture/Conference
21:17
Skeleton (computer programming)Validity (statistics)Revision controlData managementVariable (mathematics)Macro (computer science)Physical systemServer (computing)Electronic mailing listDisintegrationSoftware testingProcess (computing)Patch (Unix)Bit rateExistenceIntegrated development environmentRevision controlMedical imagingSkeleton (computer programming)BuildingComputer fileMacro (computer science)MetadataTemplate (C++)Service (economics)CuboidMereologyLine (geometry)MathematicsDescriptive statisticsOpen setPhysical systemGraphical user interfaceNumberOrder (biology)EmailSingle-precision floating-point formatConfiguration spaceData typeMechanism designOpen sourceType theoryComputer animationLecture/Conference
23:10
Skeleton (computer programming)Validity (statistics)Data managementServer (computing)Electronic mailing listDisintegrationSoftware testingProcess (computing)Patch (Unix)Revision controlPhysical systemMacro (computer science)Variable (mathematics)PressureExistenceServer (computing)Revision controlRepository (publishing)Subject indexingEmailElectronic mailing listSoftware testingFile Transfer ProtocolVirtual realitySoftwareVirtualizationComputer animationLecture/Conference
24:01
Personal digital assistantSkeleton (computer programming)Validity (statistics)Data managementMacro (computer science)Physical systemVariable (mathematics)Server (computing)Electronic mailing listSoftware testingDisintegrationProcess (computing)Patch (Unix)Revision controlFreewarePressureArchitectureContent management systemCore dumpSoftware developerVirtual machineSoftwareSoftware testingNumberScripting languagePatch (Unix)Physical systemConfiguration managementMultiplication signComputer fileComputer animationLecture/ConferenceProgram flowchartDiagram
24:48
Computer fileDistribution (mathematics)Directory serviceBuildingType theoryVirtual machineFile Transfer ProtocolMetadataLecture/ConferenceComputer animation
25:42
Software developerFreewareRevision controlComputer fileTemplate (C++)Configuration spaceDefault (computer science)Parameter (computer programming)Distribution (mathematics)Virtual machineVirtual realityScripting languageContent management systemPressurePresentation of a groupElectronic mailing listEvent horizonSlide ruleSet (mathematics)DiagramSource codeLecture/Conference
26:12
Revision controlComputer fileConfiguration spaceTemplate (C++)Default (computer science)Parameter (computer programming)Distribution (mathematics)Virtual realityVirtual machineScripting languageSoftware testingContent management systemNumberComputer fileConfiguration managementBuildingPhysical systemInformationIntegrated development environmentVirtualizationType theoryParameter (computer programming)Source codeLecture/Conference
26:52
MultiplicationComputing platform1 (number)SoftwareRevision controlDesign by contractSoftware maintenanceStructural loadComputing platformMultiplication signConfiguration managementPay televisionType theoryPhysical systemParallel portCodeNetwork topologyUniqueness quantificationSuite (music)Channel capacityProcess (computing)QuicksortComputer fileForm (programming)Different (Kate Ryan album)System callProgram flowchartLecture/Conference
28:44
Computing platformPhysical systemDifferent (Kate Ryan album)System callType theoryRevision controlSoftwareInformationDatabaseBuildingElectric generatorIntegrated development environmentQuicksortVirtual machineCASE <Informatik>Computer fileDistribution (mathematics)Form (programming)MetadataProgram flowchartComputer animationLecture/Conference
29:25
Computing platformElectronic signatureConfiguration spaceData modelData managementPatch (Unix)Content management systemSoftware testingSoftwareForm (programming)Quantum stateCartesian coordinate systemComputing platformType theoryLecture/ConferenceComputer animation
30:03
Electronic signatureData managementConfiguration spaceData modelPatch (Unix)Content management systemSoftware testingFreewareMultiplicationSoftware testingSoftwareBitFeedbackVirtual realityBootstrap aggregatingDistribution (mathematics)Patch (Unix)Level (video gaming)Computing platformIntegrated development environmentBootingMobile appLecture/ConferenceComputer animation
31:14
Computing platformSoftware testingContent management systemElectronic signaturePatch (Unix)Configuration spaceData managementData modelDistribution (mathematics)Set (mathematics)Different (Kate Ryan album)Type theoryMechanism designPatch (Unix)Term (mathematics)Library (computing)Module (mathematics)CASE <Informatik>Physical systemPresentation of a groupCanadian Mathematical SocietyOpen setBitAbstractionInteractive televisionVirtualizationVirtual machineProjective planeOpen sourceProgram flowchartLecture/Conference
33:01
Electronic signatureData modelConfiguration spaceData managementPatch (Unix)Content management systemSoftware testingBuildingBuildingRobotInformationElectronic signatureInteractive televisionSet (mathematics)Multiplication signSlide ruleMechanism designStructural loadTerm (mathematics)DampingWebsiteWeb pageCartesian coordinate systemComputer animationLecture/Conference
34:21
Electronic signatureConfiguration spaceData modelData managementPatch (Unix)Content management systemBuildingSoftware testingVirtualizationCartesian coordinate systemVirtual machineVirtual realityContent (media)Configuration spaceComputer fileEndliche ModelltheorieInformationIntegrated development environmentWave packetControl flowModule (mathematics)Online helpLink (knot theory)Multiplication signData managementNumberInternet service providerConfiguration managementGraphical user interfaceComputer animationLecture/Conference
35:13
SolitonPlastikkarteOpen setSoftwarePressureComa BerenicesMultiplication signFeedbackRevision controlMetadataMechanism designSet (mathematics)Computer fileGoodness of fitDampingDistribution (mathematics)HypermediaMathematicsComputer animationLecture/Conference
36:42
Multiplication signVirtual realityVirtual machineIntegrated development environmentLecture/Conference
37:27
Link (knot theory)Logical constantPlastikkarteSoftwareOpen setCone penetration testVirtual machineNumberInformation securitySoftwareMereologyQuicksortBuildingProcess (computing)CASE <Informatik>Cartesian coordinate systemRevision controlContext awarenessComputer animationLecture/Conference
38:40
BuildingTelecommunicationRoutingTerm (mathematics)Directory serviceRootLecture/Conference
Transcript: English(auto-generated)
00:00
Hello everybody, so I'm here to speak to you about a new project. I'm working on what's called project builder which is trying to Provide a tool to support mechanisms that I call continuous packaging that I will try to explain
00:20
In the next slides just to introduce myself, I'm working at HP I've been involved in software engineering since a certain number of years and also UNIX systems, and I'm developing Two main open source projects. So the one which is presented today called project builder and another one
00:40
Which is mondo rescue, which is a disaster recovery solution for Linux and I'm also packaging software from mandriva bit for fedora Trying to be useful So what what is the problem I had and one of the problem I'm not alone to to have in fact is that
01:02
You have Linux users will Try to use software and most of the times are much more interested by using packages because it's much easier for them To download a package and to install it in the distribution. Most most of the people do not know how to build the software's just The barely know how to use it. So they don't know how to build it
01:23
So most of the people prefer to use their packages, RPM packages, gen 2 packages or whatever you have on your underlying operating system It's convenient because also it respects the convention of your distribution so it puts the files in the right place It's uninstalled cleanly. So when you uninstall the package, you know that you have removed everything which was borrowed by the package
01:47
So it's easy for users It's easy for administrators as well because it remains in compliance with the way they are installing the distribution they can put it in a yum repository apt-get repository and get channels or however you call them of packages and
02:04
pull packages from that channel That's the most integration introduced Administration time. I mean I did as everybody I began with slackware distribution and I did my share of Configure, make install stuff and it's funny for a couple of years, but after that you want to use the software more than compile it
02:25
so It's much easier to use packages. Certain number of people still like to use Torballs and compile their own packages either for optimization performances Because our distribution does not provide a packaging system Because they want to use the latest and greatest version of the software and software does not provide any package for them
02:44
so they want to use that and Very few people are in fact using the configuration management system provided by the project because They don't know how to do it. They don't know where it is It's complicated to extract data from it to rebuild from it each project has its own way of dealing with that
03:04
So you don't have a lot of people and users testing the current state of development of the software When we go on the developer mindset They don't really care about packages. What what is their interest is they are
03:22
grouped around configuration management system could be whatever they have subversion, git, mercurial, bazaar and they want to produce software under a format, which is generally a tar file that they upload on a website or an FTP server To give the availability of the software to their users and
03:43
You don't really know how they are working what type of software engineering tools They are trying to put in place to improve the quality of the software Are you using auto tools for example to make the software independent from the underlying platform? Do they have an installation procedure a clean one which integrates correctly with a package system?
04:03
Are they using any quality tools, software engineering tools? Are they afraid about distribution, picking their application, doing stuff they don't want with it? That's their environment And then you have a third category of people People who are packagers sleeping in the room for example
04:21
And those people are already a strange beast because they take the results of the developers They work with sometimes configuration management system Most of the time today and they want to produce packages for their preferred distribution But sometimes those people do not exist. So you don't have the software package for your distribution. They are missing
04:44
Sometimes they are not on time. They have a life, they have children, they do something else, they have their own projects, whatever So they lag behind the version that exists They could lag behind the main version, but they could lag behind in term of intermediate version And sometimes they don't want to package all the time, all the night
05:02
So they don't they're just really stable version, but they don't want to release intermediate version so that people can test them So it's very difficult to have Except for very famous project. You don't have generally possibility to test intermediate versions for applications and they also have some tools they could use to improve the quality of
05:21
their production. So RPM lint for RPM systems, lintian for Debian packages But sometimes they use it. Sometimes they don't use it. Some distribution have guidelines, some are not So again, the quality of the resulting packages is not ensured So
05:41
What is the solution? It's wrong. It's the wrong header. I should write. What is a solution? I mean, that's my view on the subject Packaging should be a project concern. It should not be just a concern from people making daily packages for their preferred distribution and especially for smaller projects
06:03
It's from a mandatory state as all the other state in the development of the project such as testing The installation procedure should be part of it Sometimes it's not. People should be really concerned inside the project about the fact that their software Will and should end in a distribution and should be packaged in the distribution
06:23
So they should make it easy for people to package a distribution My opinion is that they should do that probably themselves or at least helping a lot to make it possible But they should make it as easy as possible and my mantra personally is package early, package always. Not even often, but always
06:43
I'm fed up installing tar files on the system. Each time I do that I regret it later on because I have to clean stuff manually. I don't have my dependency tree anymore I don't know which file belongs to what package. I cannot query the database to have information on it So it's always a nightmare. So rather than doing that
07:04
I really prefer to package the application before using it Even if I do a test of an application. Takes a bit of time, but it's really much more interesting at the end So my personal approach is to say packaging should be your only way to deliver software
07:20
to your users and users could be Internal users, external users depending on where you're working and for whom you're working. It really I mean, it's not a dream because that's the way I'm working now on the project I'm working on So it's really something that you can really achieve. It's possible
07:40
There is an overhead, I don't want to say there is no overhead, there is of course an overhead, but it's really minimal It's not much longer to produce a package or to produce a tar file that you will give to your users But it brings much more information in terms of metadata and coherency there is a maximum benefit because
08:02
If you use tools to produce your packages that will be consistent, that will be reproducible So we'll get all the benefits of having packages to create a database etc. It's easy to integrate your packages in your distribution So if you're upstream and you care about it You will make it easy for distribution to pick your projects and integrate it and so you will have more coverage
08:22
So you will have more users, more test cases, etc So for me, in fact, if you think about packaging you could think of it as a marketing activity for your project You make it easy for people to use your software, you communicate better, you have more testers You have more users, more return and your project is attracting more people and it's much more easy for everybody to deal with it
08:46
So for me the solution is trying to adopt a continuous packaging approach Which means Generally, you don't if you if you look at a classical v-cycle approach
09:01
I don't want to say it's a better one, but that's one example You design your software You have the user requirements, functional specification, design specification and you start to code and generally you start just here because you don't care about that and You produce code, you produce code in your software configuration management system and at that point at the unit test
09:23
level you can already produce packages instead of producing tarballs that you give to the unit test team if it exists or to yourself on another platform if you want to test on separate platforms But instead of you of using tarballs at that level you can already produce packages and I put here three lines because I'm working on a project which
09:43
In which we try to address multiple operating systems. So not only differences supporting different Linux operating the Linux distributions But also supporting other type of operating systems just as a proof of validation So I'm working with As an example now Solaris because that's what my customers are interested in
10:04
In parallel to Linux because we want to be able to deliver packages for multiple type of operating systems But sharing the sources on one side for the project But also trying to share the metadata around the package management. When you create a package you have a certain number of
10:20
Informations which are similar whatever the package format you have so similar between Debian between RPM between Solaris you have a name you have a version you have a release tag You have a header line You have a description and even some people try to join to make to share the description and the header to
10:41
Translate them all together. I think it's a very good idea But you can share a lot of metadata if you are producing your software between all the package management systems and Even part of the build of your software is the same on different platforms. So you could also be able to group that information and not duplicate information. That's my part of my
11:05
points here is to avoid duplication of information maintaining it only in one place To produce all those packages and those packages can make the life themselves along the Written cycle so you pass some of them to unit tests
11:22
They give you feedback you fix them and you pass an integration test your packages validation up to your customer and The good stuff is that package you're producing here could be exactly the same as the one you will deliver to your end customer So my personal answer to that
11:43
Problem analyzes it. I'm a system administrator on some of the system I'm managing so I just want to use packages I don't want to mess up my system by installing something else and I have a deployment server So I just want to put my packages in my deployment server and benefit from that I'm also developing open source projects. So I want to make tests easy for me
12:04
So when I install a test version of my package, I don't want to have multiple versions and multiple directories Having the risk of Interdependency between themselves making the test failing for not good reasons So I don't want to screw up my system and I prefer to install a package remove a package each time
12:21
I want to make tests on my own application. I'm also packaging software for distribution So I want to make it easy and to have a tool for me to help me produce package Much more easy for for the distribution. I'm dealing with And of course I'm the only developer of the project on which I'm working mostly we are very few
12:42
So I'm also the marketing department of my project. So producing packages from your way to say to people Hey, my project is great. Just it's easy to use for you Just grab the package install it very easily in your package management system and you have access to my application I have a background of software engineering
13:01
So I don't want to duplicate anything as much as possible if you can avoid it so including the metadata around package management and I found no real tool to to do what I wanted to do at that time So I start writing my own in fact I start as everybody writing set of scripts to support my first project and finding out that some other
13:22
projects especially I had colleagues in HP wanting to package some other applications and We thought it could be useful to reuse my scripts with my script were not adaptable easily So I rewrote the project completely from scratch in order to be able to make it project independent In fact as much as possible
13:42
So there are a certain number of goals around the development and the goal to support your continuous packaging approach is to be Agnostic because you don't know what people want to use in terms of tooling to develop their project So it should be configuration management system agnostic It can support flat files if you have flat files
14:03
But as I said, we are in the 21st century. So now it's time to use version CVS, Mercurial, Git, Bazaar or whatever you want and project builder is currently supporting those five configuration management system, but you may add easily more if you need
14:22
It should also be agnostic in the way it's working for generating packages So it should be able to generate RPM depths, e-build, Slackware packages Currently, I'm building packages for my projects for something like more than 60 different distributions So distribution being something like Fedora version 7 for i386
14:44
So this is one distribution, one tuple, one combination of a distribution It supports also the repositories linked to those distributions So it creates indexes for YAML, for URPMI, for APT It also supports Solaris packaging since the last version and it's now
15:04
It has always been but it's much more easy to prove it It's easy to adapt it to new package management system appearing. So I have requests working for HP to have support for the package format of HP UX that I could very easily add to the system without any problem
15:25
One of my issue personally was also it should be build environment agnostic So I want to build packages locally on my laptop or on my build machine and I want to be able also to build cross build for other systems, which I don't have which I'm not running locally and
15:45
Using Schrute and you have some tools depending on the distribution to generate those Schrutes So Rinse, MOC for mainly RPM systems, Dev-Bootstrap for DBN based systems that you can use to Generize the Schrute and build into those Schrutes
16:00
Another way for me, which was more convenient in my opinion was to use a virtual machine so the tool is able to Pre-build the Schrute or pre-build the virtual machine for you and then use it as a way to build the packages Which are seen as native packages for that build environment, be it a virtual machine or virtual environment
16:22
So supported right now QEMU KVM It's not a little bit aware up to now. So it should it's part of the evolution I still need to work on to support other type of virtualization mechanism mechanism more easily and one or so of the goal was to have no impact on the original project, so
16:46
You don't want to put more pressure on upstream you want to be able to package them So if upstream want to package you can use the tool to do it and it's very nice for that That's one of the goal But if you want to package for another upstream project and and you don't want to make an impact on them you have your own
17:03
Configuration management system separated in which you keep all the metadata you need to build your packages and You can do that for that project without themselves not knowing our package in any way and you can just take the tar file with the MD5 sum of the original project and
17:20
Use that as an input to your tool and it will package it from it completely So what we've done recently is we have used a tool to show to some of our customers How to realize a multi-platform development
17:41
cycle As easy as possible. I don't pretend it's an easy way and easy stuff to do So it's mostly interesting for people developing in C, C++ People developing in Java or other type of applications may have different problems to solve
18:01
The idea is to say you have a lot of code in common and you produce That code you can produce it for multiple operating system in parallel by sharing a lot of Common source code having only a minimal set of differences between so for example here It's Solaris on Spark, Solaris on x86 and Linux on x86 And the idea is to say generally people are concerned in the development up to that point so producing executables
18:28
So we have tools inside HP to help them trying to share As most as possible the common code and to isolate in small libraries. What is different and we provide some Tools to help them have the missing system calls in the environment compared to to for example a Solaris environment
18:47
But the idea is to say we can go one step further and adding project builder in the loop We can go to produce the executables and then from those executables data file man pages Configuration files or what is needed for a project to work you can package them using the native package
19:06
System and produce local packages just from a single repository Containing all those information plus some metadata information, and it will be clear in the next slide So in addition of what I've described up to now
19:23
There are some goodies coming with the tool. I have a set of existing virtual machines So currently on one of my disks at work. I have 120 gigabytes of virtual machines ready for me to be called to build packages for all those
19:40
distributions that I've built during time so since Fedora 5, since Mandriva 10.1, since Debian 3 etc etc so I have I keep all my virtual machines because One problem the distribution are not so good to deal with is that Distributions are looking at the future. So they want to produce packages for their current distribution and the future distributions
20:05
They are not very concerned by the past, but customers are customers are often running older distributions that what people are developing on and they need and You have projects such as backport initiative for various distributions. They need to have packages also
20:22
For their running systems, which are sometimes late in In the evolution of the distribution itself So they have to be able to produce packages for an old Debian old stable for Former, Mandriva, Fedora, whatever distribution, RHEL 4
20:42
Who still is using I mean, I don't know if you are working for customers or yourself Are you still using RHEL 4 in your environment? One two three customers four, okay, not a lot Personally the customers in front of which I'm working with
21:00
They have a lot of old stuff like that And so having for them an easy way to produce packages not only for the latest and greatest version of the distribution but also for all the version is something which is quite important So I have my VMs to do that myself and the tool is helping you if you give it the ISO image
21:21
to recreate the build environment for whatever older version that you want to package for There is a certain number of macro available inside the system and you can define your own macro In order to share the metadata So for example, I define one macro for the header of the package
21:41
Which is a line describing what the package does that macro is Stored in a single configuration file and every package I made is referring to that macro So I don't have to update the macro in all my packages when I change the header line or when I change the description Etc, etc
22:01
So I try to concentrate in a single place the metadata and to reuse them through that micro mechanism Across the packages I use and I was very pleased With that because when I added Solaris support to the tool, I didn't have to change Anything in my in my tool or in my description of existing packages
22:23
I just created the template for the new Solaris package Referring the macros that were already existing and the package were just right out of the box. So it's quite interesting When you create a new project, the tool will create a skeleton for you Providing you with templates for an RPM type of build, a DEB type of build
22:44
Gen2 type of build and a Solaris now type of build. So it gives you a skeleton. Of course, it does not You have seen the open source build system demonstration. They have nice GUI. So I don't have the GUI And they have some automatic way of creating the build part of the package. I don't deal with that
23:05
It's up to you to deal with the build part, but the rest can be helped a lot And it provides some goodies for me, which I find useful for me I don't know if it's useful for other people, but I mentioned them anyway I do the delivery of my packages on my FTP server directly through SSH and I rebuild
23:25
Indexes on the FTP server. So as soon as I publish a new version of my project People doing YAL update will get a new version On their server if they use my repository as a main repository to get packages Again, when I do a new version, I have a way to make an announcement to the mailing list automatically
23:46
Printing the list of new packages that I've built in it, etc, etc I Use snapshot features of VM and virtual environments if possible I'm working on integrating tests because as I have the infrastructure now It's very easy. If you launch a virtual machine, for example to build a software
24:03
It's as easy to launch a virtual machine and to do something else such as Testing that the package is installing correctly and that it works the way you want So you can add a certain number of test scripts to the tool It supports patches if you are not an upstream project so that you can integrate the patches into
24:23
Into the packages you are making. It launches automatically lint and RPM lint at the end So if you want to check the currency of your packages It's easy for you Don't want to pass too much time on that. So This is a big picture at the end
24:41
you have the Users who are the developers who are producing the projects through their configuration management system and then provide one tar file for you And you have the packages which generally are providing metadata around the project and Create other files So project builder what it does it instantiates all the metadata and creates
25:06
Another tar file containing multiple build files for all the distribution supported So it creates one spec file per RPM distribution that you need. It creates one DBN directory containing all the files needed for DBN per DBN or Ubuntu
25:21
Type of distribution it creates one ebuild file for gen2 etc, etc And then you can build locally and generate your packages on your running machine or you can send those two tar files Into your VM and the VM is also using the tool and generates the packages the same way and at the end You can push all that to the FTP server and have it delivered to to your community
25:46
Here is a list of commands. So I don't know how it's working during the event There is a possibility so the slide set will be is hosted on the project Subversion so you can have access to it and I suppose that the presentation will be available
26:03
So I will not necessarily detail all those commands one by one. I'm not sure it's interesting for you You will have access to to the presentation so you can look at it and ask me question if you are interested with it Globally you have commands to take from the configuration management system information create the tar files
26:22
From the tar files create the packages send them to the SSH Send them through buildings through a VM or virtual environment, etc, etc. So Certain number of so it's a command line interface tool. It's like an SVN or Mercurial type of syntax you have PB then the command and then the parameters
26:45
I find it easy to use but of course I wrote it so So if you if you look back at building software, so I have customers currently Big ones that I cannot necessarily name
27:02
We're very interested by migrating from Solaris to Linux and I say way But how to do that and how to and the problem is that their own customers So they are producing software for end customers and those end customers They don't necessarily want to abandon the Solaris platform that they have bought in the past or Spark platform. So
27:25
My customers need to maintain in parallel different version of software during certain amount of time and so they need to have the capacity to easily build for multiple targets Reducing the cost as much as possible because if they go to Linux
27:43
It's because they want to reduce the cost of the underlying platform and of the software subscription contracts, etc So if they come from all HP, UX, AI, X that are raised to go to Linux It's because the cost is interesting. So at the end it should cost less, but if you duplicate the build
28:01
And the maintenance load on that They will not gain the money they expect to gain in the process so their idea is to say okay We will have for a certain amount of time to maintain in parallel Two development trees one on our existing platform and one which we are promoting on Linux for the future But we will have to deal with that in parallel during a certain amount of time
28:22
So developers will still work around the configuration management system and they will produce Roughly most of the code will be just generic enough to run on whatever Unix systems They want to deal with and I'm not dealing with something other than a Unix type of systems And then we'll have to generate some of the files which are different
28:43
Providing the missing system calls for example or stuff like that, which are not compatible between different version of Unix type of operating systems and The people on top of the development people working on the packaging of the software They can store in a single database all the metadata around all the packages that those team have to produce
29:05
They will generate their tar files and the development team will generate their own tar files which will be in fact the same in that case and the tar file will contain again all the information for the multiple distributions and Using a build form so they will probably not use VM as I do but they will probably
29:22
Rather you have a build environment. So dedicated machine to build software So they will use a build form to produce packages for the various Projects and sub projects that they want to deal with and they could have a build form using
29:40
X86 type of platform running either Linux or Solaris. They could have Spark platform to build for their Spark application So The tool is I've worked on the tool since three years now I use the tool regularly every day for my own upstream projects
30:03
So I know it's at least working for what I do with it It's released under the GPL. You can download it. You can test it I have other colleagues in HP working on it a bit so begin to receive some feedbacks and some improvements I use a lot of VMs myself to build the software so more tests need to be done on virtual environments
30:25
I have added dead bootstrap very recently for my Debian colleagues inside HP. So I'm a Mandriva contributor I'm much more used to RPM myself, but the goal of the tool is to be really independent from that I don't want to create a tool which has a flavor of one distribution. I prefer to make it generic enough and
30:48
At the same level as what I have for the other distributions So it's something I need to work on including rinse and mock I've met some patches for rinse because it was not able to work correctly. Rinse is nice because rinse is working as well on
31:02
RPM based distribution and all that best distribution so you can because what is it strange in that Native platform if you think about that Environment when you are building packages locally people can build packages on their own distribution But they could be a large set of different distributions themselves. So the tool needs to run onto different type of distribution
31:26
very easily so We can they can use a shrewd which are different from one distribution to another and shrewd mechanism Which are different from one distribution to another what I need to work on is and I've seen that also in the One of the previous presentation is LSB shrewd support
31:43
Knowing that the LSB shrewd that is provided by the Linux Foundation is a bit Insufficient in terms of the tools they are providing so there are a lot of Libraries and modules that are not available in the LSB shrewd that would need to be added before Project builder is able to run in the LSB shrewd itself
32:02
Which is not the case with a fully a complete distribution because even if there is something missing Project builder will download what is missing to have it working automatically. I need to work on making more generic and supporting libvirt and virsh To make the interaction with virtual machine more
32:23
Capable of handling other type of virtualization mechanism. It will add an abstraction layer in the tool for that We can add other CMS other virtual machine systems, but as every open source developer I'm working first for myself. So I put in the tool what is interesting for me if some other people need something else
32:44
I welcome patches, of course. So for example bazaar is not yet here via more exam or not tested currently Need to support more deliveries and just SSH even if SSH is quite generic enough for me, but and
33:03
People wanting to use build farm There is a need to look at how to interface that tool with build bot Which is a tool to monitor a build farm and what is interesting in build bot is a queuing mechanism Which I don't have in project builder and I don't want to mess with that
33:21
I prefer to rely on another tool and build bot is also very interesting in the reporting it provides to you Which what has been built what has been done it creates web pages dynamically containing information What you have you have produced in term of packages. So this this interaction is something I need to work on
33:41
Signature support is currently not provided because I had problems with RPM on Debian. It's simpler on RPM It does not work correctly with GPG agents, for example, and I want to make it automatically available So if you don't have the GPG agent running it's you don't want to sign every package manually It's not possible when you build a large set of packages
34:04
Documentation is improving especially since the time I made the slide that I have made an internal Lab in HP for my colleagues. So now I have a lab document which describe how to start with a project from scratch Which is available on the website of the project
34:21
So it really gives you all the steps to do packaging of your application from nothing up to producing packages for DBN RPM using virtual machines and virtual environments So Dominik is not here I think anymore but because I had a train to catch but one of the possibilities So the tool is written in Perl and Dominik has written a config model module for Perl which allows you
34:46
To easily manage configuration management files and one of the problem currently is that I don't have any graphical interface including To manipulate the configuration files and I have a certain number of configuration files and items So config model will be would be a very good tool to
35:03
provide help information on how to fill the configuration file content Some links on it and I'm done Just on time Exceptionally, thank you. I have one minute for questions. Great. You have
35:35
Some ways to get feedbacks when a build fails on one of the VMs
35:42
When a build fails generally you don't get the packages back So that's a good feedback You're not able to know if this package doesn't work on this version of this OS Well, generally you will money at least as if at just the first when you when I create for example
36:04
Federal 13 is out tomorrow or in two months. I don't know I will create my new VM for federal 13 I will set it up and then I will launch my build on federal 13 most of the time It will work just fine because there is no big change between federal 12 Which is already built for and federal 13 if there are some problems in the build mechanism
36:24
then I will look at them and I will have to solve them in my metadata Is a set of the metadata I am managing so it could be a problem with a spec file It could be a dependency which is not Manage the same way between all the distribution etc. So so do you
36:41
Reinstall from scratch your VMs every time you install or you build or install a package So it's good idea is to use snapshot So you create your VM at first you snapshot it the way You have created it and then you always restart from the snapshot project builder will upload from The main so seats fedora 13. They will take the yum
37:04
Come package missing for building your package it's analyzed. So we are looking at build requires And requires lines and for Debian as well and we install dynamically the packages which are missing inside the VM or the virtual environment and then we try to build inside that environment and
37:25
At the end if it's a snapshot your revert to the original Virtual machine that you had so you're safe. I have one more Do you handle security on the VMS let's say
37:46
If a package tries to reach the network or stuff like that do you cut the network or You don't really handle that part No, I don't really handle that part because for me it's it's a sort of sandbox. I mean, I'm building myself a VM
38:02
So I trust it As much as I trust myself To behave it correctly so building the package the tool has been designed for upstream people wanting to package easily for Because upstream people generally don't know how to package so having a tool for them to have them making package more easily and for beta
38:24
Version is very interesting So they generally know what the application is doing and there is no Good reason to make to put more security in the build process and then in other context I mean and in a in a case in the worst case if the VM is completely trashed because the package made something the
38:44
business the build Steps make something completely ugly inside your VM. You crush the VM your build one and you fix your build Steps because there is something wrong with your build Instructions somewhere But there is no reason there is a an SSH communication between the host and the guest
39:03
As a way I'm communicating and I create a dedicated account inside the VM called PB Which and I always build non root inside the VM so it's quite limited in terms of Damages that you can create inside the VM because you can raise everything owned by your own account
39:24
Which is not a lot inside the VM just have your home directory mainly, so That's a waste. Thank you