Automate, contribute, repeat.
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 36 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21230 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
ComputerMenu (computing)Metropolitan area networkSynchronizationDrum memorySoftware frameworkProjective planeQuicksortSoftwareLevel (video gaming)NeuroinformatikInterface (computing)Link (knot theory)Crash (computing)Multiplication signPhysical systemMusical ensembleVideo gameBuildingLibrary (computing)TelecommunicationFront and back endsTrailProgrammschleifeComputer fileMereologyLine (geometry)Graphical user interfaceoutputHacker (term)Inclusion mapProduct (business)Mobile appComputer hardwarePresentation of a groupDigitizingEvent horizonSoftware testingCodeDigital signal processingSoftware developerPhase transitionLie groupWorkstation <Musikinstrument>WindowRight anglePosition operatorComputer animation
06:20
Point (geometry)Right angleSoftwareImplementationPerfect groupMultiplication signRevision controlLink (knot theory)Set (mathematics)InternetworkingSocial classComputer animationLecture/Conference
07:23
Scripting languageNP-hardAutomationScripting languagePhysical systemFitness functionExtension (kinesiology)NP-hardMultiplication signProper mapComputer animation
08:12
System programmingProper mapProper mapPhysical systemGreatest elementClient (computing)Computer animation
08:51
Client (computing)Formal languageBitAbstractionComputer filePhysical systemDeclarative programmingUniformer RaumSocial classLevel (video gaming)Musical ensembleOrder of magnitudeMultiplication signWordComputer animationLecture/Conference
10:16
Touch typingModule (mathematics)Poisson-KlammerNumerical analysisTask (computing)CASE <Informatik>Information retrievalHTTP cookieStapeldateiComputer fileScripting languageSource code
10:54
Data structureSequenceOrder (biology)Physical systemProjective planeLine (geometry)Variable (mathematics)Forcing (mathematics)Arithmetic meanSource code
11:33
BitCondition numberString (computer science)Variable (mathematics)Object (grammar)CASE <Informatik>Task (computing)WordLevel (video gaming)Extension (kinesiology)Computer fileHuman migrationStability theoryAreaSource code
13:22
Data structureProjective planeGroup actionLevel (video gaming)WindowWeb 2.0Source code
13:59
Physical systemGroup actionSlide ruleForestModule (mathematics)Computer fileLine (geometry)Server (computing)Web 2.0File systemVariable (mathematics)Parameter (computer programming)EncryptionProtein foldingTemplate (C++)Task (computing)Rule of inferenceKernel (computing)State of matterVarianceSource code
15:41
ComputerWorkstation <Musikinstrument>Module (mathematics)Physical systemStapeldateiGroup actionCASE <Informatik>RootProjective planeNumerical analysisComputer virusOpen setServer (computing)Social classFreewareBitSurjective functionVariable (mathematics)Computer fileUtility softwareConnected spacePhysical systemData managementSoftwareDifferent (Kate Ryan album)System callTask (computing)ArmLaptopoutputSet (mathematics)VoltmeterVirtual machineNeuroinformatikWorkstation <Musikinstrument>Direction (geometry)VarianceParallel portWindowScripting languageToken ringLatent heatEncryptionComputer animation
20:03
Module (mathematics)Physical systemAutomorphismBlock (periodic table)Service (economics)Element (mathematics)Group actionSoftware testingComputer simulationCasting (performing arts)Interface (computing)SubsetRevision controlInstallation artXML
20:50
Integrated development environmentInsertion lossWorkstation <Musikinstrument>Scripting languageGastropod shellPhysical systemWorkstation <Musikinstrument>FrequencyIntegrated development environmentWindowScripting languageCASE <Informatik>Computing platformComputer fileWhiteboardInstallation artModule (mathematics)BitSet (mathematics)Gastropod shellPoint cloudMobile appComputer animation
22:19
Firewall (computing)WebsiteGateway (telecommunications)CASE <Informatik>Medical imagingSoftwareProcess (computing)Office suiteBinary imageLogicGroup actionIntegrated development environmentNumerical analysisWindows RegistryMobile appComputer fileRepository (publishing)Computer animationLecture/Conference
23:59
Service (economics)Game theoryScripting languagePhysical systemNatural languageLevel (video gaming)Medical imagingUniformer RaumRight angleVideo gameServer (computing)Open setInformation securityKey (cryptography)Task (computing)Object (grammar)State of matterElectronic mailing listVirtual machineSpectrum (functional analysis)Sampling (statistics)Game controllerEvent horizonQuicksortCoefficient of determinationCASE <Informatik>Slide ruleGroup actionInheritance (object-oriented programming)Service (economics)Installation artStructural load
26:36
Physical systemWindowTelecommunicationComputer architectureVideo gameSet (mathematics)Archaeological field surveyCompilerBitSoftwareMoment (mathematics)AreaRevision controlConnectivity (graph theory)Figurate numberProcess (computing)BuildingOrder (biology)Computer animation
28:03
State of matterWindowTemplate (C++)Scripting languageInformation securityStudent's t-testCore dumpHierarchyLattice (order)Level (video gaming)Software developerQuicksortObject (grammar)Source codeGroup actionPhysical systemIntegrated development environmentVirtual machineWordLinear regressionCASE <Informatik>Point cloudCoefficient of determinationInformation technology consultingBitTheoryComputer configurationService (economics)Video gameAutomationRootInstallation artComputer animation
31:38
Core dumpCodeProjective planeMereologyNoise (electronics)WordWindowSoftwareFocus (optics)Open setCodeFirewall (computing)Data managementInstallation artModule (mathematics)Goodness of fitLevel (video gaming)Condition numberPoint (geometry)Inheritance (object-oriented programming)Internet service providerImage resolutionScripting languageComputer animation
34:44
Module (mathematics)Core dumpCASE <Informatik>Right angleInheritance (object-oriented programming)CodeInterface (computing)Scripting languageImplementationMathematicsPhysical systemService (economics)Object (grammar)Dot productLine (geometry)Medical imagingData managementSocial classWordPower (physics)Computer animation
37:20
Multiplication signModule (mathematics)SoftwareSpecial unitary groupGoodness of fitWindowFeedbackCore dumpBitContext awarenessError messageGroup actionComputer animation
39:10
Module (mathematics)Square numberLink (knot theory)Multiplication signGoodness of fitTowerState of matterGroup actionOperator (mathematics)Medical imagingIntegrated development environmentComputer architectureInteractive televisionServer (computing)Einbettung <Mathematik>Scripting languageShared memoryProjective planeRule of inferenceNatural numberWeb serviceStandard deviationData miningPhysical systemSpeech synthesisTemplate (C++)Device driverInterpreter (computing)Video gameInternet service providerComputer animationLecture/Conference
42:50
NeuroinformatikTable (information)Social classPhysical systemTemplate (C++)Arithmetic meanVideo gameServer (computing)Revision controlData structureDirect numerical simulationLevel (video gaming)Source codeInheritance (object-oriented programming)Rule of inferenceDirectory serviceArtificial lifeTouch typingMultiplication signComputer fileCodeLecture/Conference
Transcript: English(auto-generated)
00:00
with Theo, welcome new people, give him some applause. So hi everyone, as a disclosure, my excuses for the nice chrome bar, I just started with
00:20
this reveal.js thingy and I can't get it to work properly, so whatever. So we're gonna talk about automation, and as a short presentation, so I'm Theo, I work for Ableton, and basically I tend to describe myself as someone who whispers to computers, like I'm more of a hacker than a proper scientific engineer.
00:46
So working with computers is pretty cool, right? But making music is definitely way cooler. So why don't we basically achieve the dreams of our grandfathers and so on, and make computers
01:04
work for us, so we have more time to make music. So if you don't know Ableton, we are a music company, we basically build softwares and hardware that enable music makers, from building a track to playing it live.
01:24
The company was founded 16 years ago, so we're not exactly a start-up anymore, although we're still a rather mid-sized company, we're Berlin-based, and the company was founded in Berlin by Berliners, and our founders are electronic musicians, well not
01:44
anymore, but they have been electronic musicians for a while. Our most known and used product so far is Live. Live is what we call a DAW, so basically it's a software that allows you to build
02:01
a track and to perform it live. It's based on two views, the one you see here, that we call the session view, where you basically create clips, so it's small loops that you can play and you can arrange as you want. And right now, in the, let's call it the mainstream music industry, it's one of the
02:24
most used software. We also have Push, so basically Push is some sort of fancy MIDI controller, so if you're not familiar with music, it's basically an interface that will send signals, notes, like a small protocol, that will tell the computer what kind of sound to play,
02:45
at what pitch, and when to play it. And finally, we recently released Link, and Link is an iOS framework that synchronizes devices, and more specifically, synchronizes their bit time.
03:02
So for example, if you're making music on your Mac, and someone comes in with an They can connect to each other, and you can be sure that they will play in time. So if someone is starting a drum beat, for example, and you had a bass line on top of it, they will play together on time, there won't be any phase, or any weird stuff
03:23
like that. And we basically provide it as an iOS framework to developers, and they implement it in their iPad apps. So these projects are big and complex.
03:43
When I say these projects, I tend to include Live, Push, and Link all into the same bucket, because basically they're all part of one big ecosystem. And if you think of Live more specifically, it's a huge monolithic C++ codebase, so we
04:01
basically have, I think, millions of lines of code, and thousands of files. They involve Qt, which is a huge library for basically building GUI. We include Boost too, which is a big C++ library that does all sorts of fancy stuff.
04:23
And we have Python in the build system, and it's also included in Push, so as a, for the communication layer and for scripting purpose. Now what's interesting in this is that I was working in startups before, I was doing
04:44
backend and system tasks, basically, and I could use all sorts of fancy tools, right, like Docker and all this stuff, I could deploy stuff on AWS, it was not a problem. And for one simple reason, it was that we were using Linux.
05:03
And once you get out of this ecosystem, you realize that if you have to build something against OS X, Windows, or iOS, well, all these tools become suddenly, most of these tools become suddenly unavailable.
05:21
But still, we're not doing rocket science, but what we call DSP, Digital Signal Processing, is serious business, it's a very complex topic, and most users rely on our software's reliability. So basically, if you're Skrillex and you're playing in front of, say, 40,000 people,
05:44
you don't want life to crash in the middle of the gig, right? Same if you're playing at the Bergen in Berlin during all night, you don't want it to just stop or start doing shit in the middle of your set. So although we don't have access to all these fancy tools to help us build a better software faster and in a more secure way, we have to find a way to reach that level
06:05
of quality. The live ecosystem involves long build times, so basically, I think we have the same kind of problematics as projects like Chromium, for example.
06:22
We are building a huge software that takes a very, very long time to build sometimes, that involves a very complex and long workflow, like from a feature ID to its implementation, it can take up to months, sorry, and we also have a culture of perfection.
06:43
So as I said, when we put a release out there, we don't want users to have the software crashing in the middle of their set. Also something very specific to this class of software is that most people that work
07:00
in studios, for example, don't have internet. So they won't update their software all the time, they will update it from time to time, but we also are aware of users who are running very old version of lives and are not wishing to upgrade it. So all of this leads to the need for automation.
07:21
So we have to automate as many things as we can, whenever it comes to build it, or whenever it comes to release it. So sure, you could tell me, we could automate everything with scripts, like with Bash scripts, Python, Fabric, you name it. Actually we use basic scripts, not basic, but scripts for the build system already,
07:45
which are in Python. But if you ask me, scripts are very hard to keep organized, to maintain sometimes, and I'm thinking more of Bash scripts here, and I'm not even talking about extending.
08:01
And yeah, let's just be honest, scripts are cool, but scripts are hard and very often cryptic. I wouldn't read some of my scripts from two years ago. We could also use proper deployment systems. Many of you probably already know of Puppet or Chef. I used them in the past, to deploy from mid-size to big infrastructures, and I really
08:26
enjoyed them for many different reasons. But in general, I find that these tools always involve the concept of nodes, of leader-follower,
08:42
and they're not really suitable with something I really really like when it comes to deployment, which is masterless systems. I don't want to have to think of, oh, do I have my client installed on this server, and where's my master, and can they talk to each other, and all this thing.
09:01
When I want to deploy or automate something, I want it to just work. And also something that's a bit too much to my taste is that you have a ton of useless abstractions. And here I think more of Puppet rather than Chef, but I don't want to have to learn a
09:21
new language. I don't want to have to think of where do I put all these columns and all this stuff. So basically the systems are great for a certain class of systems, but when you think of Tableton and what we're doing, it doesn't really fit. So that's why I introduced Ansible when I arrived at Tableton.
09:43
If you're not familiar with Ansible, it's a deployment system just like Puppet or Chef, or you have verse 2 that I won't name. And Ansible is basically declarative. So it's based on the YAML syntax.
10:00
Everything is written in YAML at all times, apart from the host declaration that you do in uni files, but that's really a minority of the files. And you just state what you want to do. And if you look at this example, you can see that you have a name for each task,
10:20
and then you have a module. So for example, the first one is homebrew, and then you have a copy, and then another copy, and then you say, oh, I want a file being created. So whenever you open these files, you don't really need to make any specific effort to understand what's happening there. And if you compare that to a bash script, I can tell you it will take you probably
10:42
five minutes in the most complex cases to understand what this is about, whereas a bash script with all this dash e dash blah blah blah everywhere, and you'd never really know where to put the square brackets, just, nah, let it go. Another thing that's great about it is that it's sequential. So everything will be executed in order.
11:03
And it will only rely on the structure of your project. So Ansible enforces a very strict structure, where you define roles and you define variables, and they all work in a scope system.
11:20
And once you get familiar with it, it's very easy to understand what's happening, and when there's an error, where it's actually happening, and why. Because you just have to follow the white rabbit. Something I really like about it, too, is that Ansible has few but very efficient abstractions.
11:43
I can name a few of them. The most common is register. So register is simply declaring a variable. So you take a task, so all these steps that you see are called tasks, and when they're executed, they return a Python JSON object.
12:04
And this object is basically describing the outcome of this task. And in this case you can say, well, I want this JSON object to be called, in the case of the first task, foo exit, it exists, sorry, and you can later on use it. So for example, if you look at the second task, you have when foo exist is false.
12:26
So when is another keyword that basically helps you defining conditions. You have with items, which you see in the last one, so I can create a variable somewhere in the variable files, and I can call it myprecious files, of course.
12:42
And then if I use with items with my variable, I can just pass around this item thingy to my expanded strings. And that's another really great thing about it, is that all of these values could basically
13:01
be strings, and they can all be interpolated with the Jinja2 syntax. So you get a lot of flexibility, and it's still at a very low level, because you only have variables, conditions, and loops, basically, that's the only thing you get. But you can mess around a bit with it, to gain some flexibility with some templating, sorry.
13:24
And as I said, everything there relies on the project structure. And you can modify it a bit, but canonically, you always have only one way to organize your project, to organize your roles, and to organize your playbook.
13:41
So the project is the top level structure, and it's the first example. And you can see that you have, it's not all the things you can put in there, but the most important are there, you have the group vars, so in Ansible you define groups of hosts, so for example, you have, I don't know, Windows servers, or web servers, or DB servers,
14:05
you're gonna create groups, and you can attach vars to these groups directly in these files. You have the inventory, where you're gonna reference all the hosts that you want to talk to. You have the roles, and roles are basically kind of modules, so you're gonna define a
14:22
role for nginx, which will be in charge of deploying an nginx server, you can create a role for installing Docker, for installing Umbru, all this kind of stuff. And something very cool too is the vault, so I will talk about it later, but Ansible is able to encrypt all your secrets, and you can access them later on in your tasks
14:43
seamlessly. In this roles, you have files, templates, tasks, handlers, defaults, meta, and variables. Basically files and templates are basically the same thing, so you locally create files, or Jinja files, and later on you can say, well, I want it being created, and replace
15:03
this variable with this value. Tasks are what I was showing before, and that you can see underneath. It's basically all the steps that the system has to go through to generate the final system state that you want, and variables are like, well, I want to match variables to values,
15:24
and another interesting thing are the meta, so it's basically for including things. So if you say that Docker relies on some fancy Linux kernel or file system that you have created, you can do it there. And all of this is aggregated in small files that we call playbooks, and a playbook
15:45
is basically binding all of this stuff together. So you give it a group of hosts, or a single host, that you want to run your project onto, you tell it what kind of connection it should use, if it should use facts, which are kind
16:02
of reserved variables, you can provide it VAR files, so in this case, that's a vault file, so you can say, well, I have my secrets vault, which is encrypted in AS256. You can define your own vars, and finally, you can whoever called your roles, so in
16:20
my case, it would be OSX Jenkins slave, or you can also call a task directly. Another really awesome thing about Ansible, and that's mainly, I think, why I personally prefer it, is that it works over SSH and WinRM. Everything you do is done over SSH or WinRM, and what that basically means is that Ansible
16:47
is able to deploy as many servers as you want in parallel on a normal laptop, as long as you have an SSH connection. WinRM is a bit more tricky, because you have to make a bit of setup on the Windows
17:03
side, but still, it's pretty trivial. As I mentioned before, you have a vault that comes in with Ansible. Something that's great with that is that it's yet another YAML file. You basically put variables in there, and then you say, hey, Ansible vault, this is my
17:21
file, please encrypt it. I don't want people to see my GitHub, my token, my AWS token on GitHub. And then all you have to do is just, when you run Ansible, you just say, hey, well, this is my vault and this is my password, and deal with it. You have a galaxy, which is a pretty fancy name if you ask me, where you can upload
17:45
your roles, so say you have created a specific role for whatever, you can upload it and share it there, and when you want to make something very common, like, I don't know, install an open SSH server, you can be sure that someone will have made it there.
18:01
But I will talk about it in a bit more detail later on. But if Ansible is straightforward, if you compare it to other alternatives. Now, I'd like to go through some use cases that we run through, because we don't really
18:23
have the same use cases as most companies that are there, I think, and I selected like three use cases that are pretty representative of what you can do with Ansible. So basically, you can use it to automate your computer. And what I mean by that, is that you can just deploy your workstation, you won't ever
18:44
need another bash script to do that anymore, and deal with, oh, am I on OS X, am I on Linux or whatever, and use this weird syntax, the last bash script you will use is the one to automatically install Ansible on your machine, and to, that's what they call,
19:02
that's what brew calls tapping, but basically telling, for example, onbrew, which is the package manager, one of the package managers for OS X, where to go find softwares. And it's really as simple as telling Ansible in an inventory file, well, I want to talk
19:20
to localhost and just use the local connection to do that, and then you have two different commands there, you have the Ansible playbook that runs the playbook file I showed you, where you can call roles, call tasks, and variable files and all this stuff, but you can also use the Ansible, just Ansible commands, and you can pass it host, and then give it
19:43
just a command to run on the command line directly. And what you can do with that, is that, if you're on a Mac, for example, you can automate the installation of system packages and your dev tools and so on, with included onbrew package. So as you can see, you define a variable that's a hashmap, in which you say, I want moke
20:05
and wigget and whatever, and then later on you can just say, well, I want to install all the elements in onbrew packages, and I want the latest version of it. Same for your desktop app, like you want to have Spotify, VirtualBox, whatever you
20:24
like installed, you can do it too with the other module that's called Kask, because of onbrew Kask, which is a subset of onbrew. And something that's worth mentioning though, is that although you have the interface for that directly in Ansible, as for a lot of services like console, for example, you
20:44
generally have to install onbrew by yourself, but Ansible can do it for you, you can write a role to install onbrew. You can use it to set up your dev environment, so you can update your system settings, you can install, pull and install your dotfiles, and you can also pull some favorite git repos,
21:04
you could also trigger some builds, anything you can think, that's really the great thing about it. So to sum it up, it's like, you do the work once, you keep it up to date, and you will never ever lose hours reinstalling your workstation. And I also tend to think that it's pretty great if you're looking into a good solution
21:24
to onboard some engineers. Like in our case, we could perfectly say that we install all the Xcode toolchain and all the CMake stuff, all the Jeep stuff, directly with just one command that's based
21:40
on an Ansible playbook that we would keep up to date. So you don't really need shell scripts anymore to do this stuff. And actually, something that's really awesome about it is that it works on most platforms. So whether you're running Linux, OS X or Windows, it will work the same way.
22:01
You might have to use some dedicated modules for Windows, but that's basically it. And you might have to set up a bit, but that's Windows right, you're used to it. Now something we do too is to deploy, well, nothing fancy but small apps in the clouds. I can think of two very simple examples.
22:23
In our offices, we have a gateway, so we have a big firewall that's keeping people from going on websites they shouldn't go home, right? So we deploy a gateway, we deploy some small dashboards to analyze what our users are doing with the software, how reliable it is.
22:42
And so the process in this case is still pretty straightforward. You create a Docker image of your app, and Ansible can do that for you, you just give it the path of your app where your Dockerfile is, you tell them, well, I want this name for my image, the repository you want to push it to. You can also tell it if you want to log to Docker Hub or to your private registry.
23:06
Then you can tell Ansible, well, okay, that's cool, I have a Docker container now, but I want to deploy it. So same as with the Docker image, but there you would do it in a playbook. So when you look at the deployment, what often happens is that in the playbook you
23:22
will first set up your environment, and what I call by environment here is like more your infrastructure actually. So you would create your VM, and here that's exactly what it's doing. So this local action is kind of a trick. It's not very obvious, we're not really used to that in Ansible, non-obvious things,
23:42
but well, it's that way. And you can describe, that's the great thing about it, you just describe what you want. You don't really need any logic, you don't really need to think of, oh, so what should happen and so on. You just think of your steps and describe them, and that's basically it.
24:02
But you could tell me, yeah, okay, but how do I tell Ansible that, well, this VM exists now and I want to deploy stuff on it, I want to put my Docker image, my Docker container there. Well, that's another super cool thing about it. Remember I told you about inventories, where you list all the hosts you want?
24:22
Well, they can be dynamic too. So you can perfectly give it a Python script, and this Python script is going to return a JSON object that's listing all the hosts that you have created on AWS. And with a few tags, you can very rapidly automate the deployment.
24:42
So you create your VM, you tag it with a certain tag, and then this inventory will ensure that you have a host that's matching this tag. Once you have that, that's pretty trivial, you just bring your VM in a suitable state. So if I take the example of control, which is a small data dashboard, what we do is that
25:05
we just say, well, I want all the hosts in my security group. That's another way of doing it. Instead of naming your instances, you can just refer to a security group. I want all the slides to be loaded. For example, AWS is going to be all my secrets and they're from the vault.
25:24
And then, sequentially, it's going to run through all the roles. So base will create some users and install some keys, SSH server is going to create, install open SSH, Docker is going to install Docker, and JNX is going to install and configure JNX, and then I'm going to deploy my HAP.
25:43
And how I do that is just, you pull your Docker image, and you're going to, you don't have to do this, but it's a pretty nice way of doing it. You create a systemd service to start this container, and you just start it. Because I will talk about it a bit later, but in our case, we use some sort of stateless
26:08
machines. So when we start something, when we create a new server, it's already in a state where you just want to have to start a few services. You don't want to deploy anything anymore on it. So basically automation is like playing a rog-like game, right?
26:22
You just die and retry until it works. I'm not saying Ansible is going to solve all your problems, or it's going to make your life super easier, that would be lying, but it will make the task of finding out how you should deploy and install things, and figure out what's not working much easier as with other systems.
26:44
And now the important bit is the dark corners I'm currently exploring at Hableton. So to build the software, we use Jenkins. In the setup where we have a Linux master, and we have drones that run OSX, and drones
27:02
that run Windows. Jenkins is kind of the nervous system of our build process. But at the moment it's pretty old, and some people are managing it by hand. And what we're doing with that is that we're deploying Jenkins on an internal vSphere-based
27:22
data center. So vSphere is basically AWS for your own data center, it's a VMware solution. And we run it on MacPros, Mac Mini, Windows servers, we have a bunch of data stores, and we have subnetworks to isolate everything. The reason for that is that when we build live, we need to build it on a Mac.
27:45
We can't say, well, I'm going to cross-compile it for Mac, or I'm going to cross-compile it for Windows. That's not possible for us. We really have to build it on the proper architecture to make sure that everything is in order. And the great thing about that is that the latest version of Ansible that came out
28:01
I think in January, brought a lot of improvements for Windows, for VMware environments, and even for Docker. But beware, because it's still a bit buggy, and the transition was kind of tough. And as I said before, automation here happens in two stages.
28:21
So what we want is to create a Linux master that's constantly replicated, so it can just fall and it's fine. And then we create all sorts of drones, we don't like to call them slaves, we call them drones. And this happens in two stages. First provisioning, which basically is trying to create a reproducible machine state, something
28:46
we can always start on. Like you press start, you know it's going to work, and it's going to be in the state you want it to be. And then just the deployment, which is basically bringing these templates to life. And in general, the only thing we have to do in the case of Linux master for example
29:01
is starting a few containers. So if we were to sum that up, we have templates that we provision, and we deploy these templates and give them name with a convention to remember basically what's there and what's running. For the provisioning, we create a VM, we bring it in a good enough state with Ansible,
29:24
we stop the VM, still with Ansible, and we create a template out of it, still with Ansible. And our hierarchy is pretty straightforward. We have a root template, which is the base OS, it's for security purpose, I can't really
29:42
get into detail, but we have to do this step. Then we have kickstart templates, so we configure everything that is needed to build. So we're going to install Xcode, developer tools, CMake, whatever is needed. And the last stage is we create deployable templates, which are following a naming convention,
30:04
role, OS, date. And that's it, you just have to deploy them all. Now a bit of sugar on top of the cake, we use console too, so if you don't know what this is, it's a service discovery system that's installed on our master and on all
30:23
our slaves. And whenever we bring these VMs to life based on this template, they declare themself, and now all the system, every pieces of the system know each other. So if you want to automate things on Jenkins' side, or if you want to make some scripting
30:41
under the hood to maintain some certain states, you can perfectly do that because they know each other. Now, there's still some stones left in our shoes. Very honestly, Ansible 2.0 was a disaster. There were a lot of bugs, there were a lot of regressions, they changed a lot of things that were working and ended up not working anymore.
31:04
This here is really nice, that was my first experience with it, I'm an AWS guy, I started my career, I always worked with the cloud. So it's pretty nice, but don't ask too many questions, right? Because it's a closed source project. And Windows support is still rough around the edges, it's a bit untested, like what happened
31:25
to me quite often is I'm like, hey, this doesn't work, I'm gonna fix it. And the guy's just like, yeah, sure, it works on my machine. So and last but not least, Jenkins is kind of hard to automate. And now we get to the core of what I wanted to bring you guys.
31:44
It's Ansible is the perfect project if you want to get into contributing Python code. If you're a beginner or experienced, it's a very good entry point. So Ansible is a widely used project, it's very very active, and I think it's a super
32:02
good entry point if you're looking to commit some code in a project. And when I mean code, I insist on code, like not documentation or anything, because the documentation is really great already, but you can just dig into it. Because all along the journey, you will use a lot of third party modules and snippets.
32:24
As I wrote it down, don't be a hero, just do it, there are a lot of things that are already existing that people made and problems that people solved for you. And all these modules and Ansible itself, once in a while, will have what I call less
32:40
frequented corners, things that people don't really use. Like if you use the vSphere module, or everything that's related to VMware, it happens quite often that something is there, it's supposed to work, but it doesn't really work. But that's actually the great thing about it, because now you can just jump into it,
33:03
and you can try to fix it yourself. And instead of being painful, it's actually very easy and rewarding to dive into it, fix it, adapt it, or even add something to it. And I really insist on that, it's really simple.
33:23
For example, the homebrew installer role that we use to make sure that when you install a Mac, or when you deploy an OS X drone, gets homebrewed so it's able to install the common packages, well this thing was mostly working, except in our network setup,
33:42
because we have a firewall and we open ports on demand and so on, and I was like, well okay, what's not working? And you just go take a look at the role, and you're like, okay, so this git thingy doesn't work for whatever reason, I don't really care, it's not my problem, I'm going to just go take a look at what the homebrew script is doing for installing, and trying
34:01
to replicate it into the role. So you can just do that for any role you find. If you've built your own role for your own case, but you're like, well maybe that could be useful to someone else, well you can just publish it. Go ahead, does it, because you can be sure that it will solve someone else's problem.
34:23
And for example, WinRM is okay, but Microsoft worked hard to provide OpenSSH support to Windows, so we basically created a role that will install OpenSSH on Windows for you, and it will be as simple as including this role in your playbook.
34:46
And to get to the core of it, most modules are just simple Python modules. They return formatted JSON objects, and they follow a very strict convention. So whatever you do, it's all about trying to document first what you are trying to do,
35:06
and then building a Python module, but it could be anything else. If you're into, I don't know, Perl, there are still people who like Perl, you can do it. You can just create a Perl script that will return JSON.
35:21
And here you can see that this is a Darwin service, so that's basically a launch team implementation that we worked on, and we use OSX a lot in our cluster, and well, starting services on OSX works with launch team, and Ansible was not supporting it, so I was like, yeah, well, sure, let's just give it a shot and try to do it myself.
35:44
And it was crazy simple. You just try to find a module that's called, in this case it's service, you try to find it in their modules core, so there are modules core and modules extra, that are out of the core of Ansible, and you just go there and go in system and like, okay, so how do
36:05
they manage systemd, or how do they manage upstart, oh, okay, that's there. So maybe I can just copy this class, try to find the key methods, what they're doing, and just put my own code there. And in one or two days, maybe two, let's be honest, launchd was there, and yeah, there
36:27
was basically no pain, the only pain was this launchctl command, that's really weird. Also although the code and the feature were accepted, this change happened during
36:43
a redesign of the old system interface in Ansible, and that's one of the black dots, so just know that it might happen if you go that way, that sometimes the community is like, yeah, sure, that's great, we want it, that's super code, but we are changing stuff right now, and it might happen that it just takes a while for them to just come
37:03
back to you, so just bump them, bump once in a while and say, hey, what about my feature? The community is wide, there are tons of people super reactive, it's very inclusive, I've never seen a sexist comment, a racist comment, people are always trying to help,
37:22
they're positive, there's no like, oh no, sorry, that's shit, we don't want to see that, just keep it for yourself, it's fine, it's really like trying to improve, and you have core contributors on top that sometimes comes in and say, okay, maybe this is not fitting with what we want for the software or something like that, but it's always constructive, and as I mentioned it a little bit so far, be aware that when
37:48
you create a module for Ansible, debugging can be quite tricky, there are a few things to know for that, you can use PDB basically, but know that you will have to force Ansible to use only one fork, because if it starts forking everywhere, it will be really hard
38:05
to figure out who is doing what and where's the error at, and it also has a ton of logging but you have to give it some time, you really have to kind of try to figure out okay, so this is about that, so it just takes some time, but you will get it eventually.
38:25
And yes, also something when you contribute to modules, like if you take the example of, I contributed a bit to some windows modules for example, well it happens that the person that created it is not as involved as before, or it's in a transition state, and you might
38:44
not get as good feedback as you would get on the docker module for example. And that's basically it. Do you have any questions, ideas, or rants? I love rants.
39:09
First, a yes or no question. Do you use Ansible Tower? We thought of it, but I think the ratio, price, what it brings was not worth it.
39:20
My follow-up question is then, how do you ensure the state of your infrastructure? Do you have a central place where you run playbooks, or every operations guy runs them from off their laptops, or how do you do that? So yeah, maybe it was not clear enough, but we don't really rely on Ansible for that.
39:40
Ansible is really a tool for us that runs some steps, through steps, and then we flash things. So we basically have templates, and we build our infrastructure state with templates on top of each other, we just enter it all the time. And Ansible is just involved in creating the
40:02
states all along the way. So we don't really need to figure out what's the state of what. It's already there, we know what's the state of what. We have root, kickstart, and then we have our deployment images. So we don't really need this sequential state.
40:21
Now I'm not sure if my question is answered already. In our company we have lots of small modules, and then of course the big environments where most of those modules are deployed. And well, one big Ansible playbook somewhere,
40:42
which deploys everything, I think is not a good idea. So each module has its own Ansible playbook, but how do you share, for example, what is it called, roles, like where to deploy to. So if you add a new sandbox, if you add new servers, then all modules, and there can be
41:00
hundreds, need to be updated in their playbooks. How do you solve that? What do you mean with updating roles? Okay, if I create a new module, then I create also a new playbook. Yeah, okay. But if I create a new environment, then all the playbooks need to know about that new environment.
41:25
What do you call an environment? You mean like a group of machines? Yeah, well, servers to deploy to. Well, you use the groups in the inventory for that, right? Yeah, but how do the many, many, many playbooks in the many modules share the same inventory? I'm not sure I understand the question, I'm sorry.
41:42
I'll ask later about it. Yeah, sure. So you focused on a built system in interaction with Python, but from what I know, you also
42:01
use Python as embedded interpreter in life to provide ability for driver producers to implement their hardware, etc. And do you actively develop this system for embedding scripts or rather you go away from it?
42:27
So basically, I think my colleague Alan can talk about it. So we embed Python, but the project architecture is made in such a way that it's
42:41
really monolithic and everything is there. So everything is in one place. So we don't really manage that. When we manage it, we have to do it by hand. If you want to update it, you have to update it by hand. Ansible really doesn't touch anything that's in the live code base, for example. It's something that it can do though is that we can install a bunch of tools that
43:03
are needed to test this code base, but we never go in there to try stuff. It's time for one more question. Hi, thanks for the talk. Very simple one. In the structure that you defined in the beginning of the talk with the folders, we
43:23
are used to have a subfolder and their roles for each of our roles. And inside that subfolder, we have the files, templates, everything. It's this normal approach of are we doing it wrong, I'm not sure. You mean you have roles, source?
43:41
Roles, then every role has a subfolder. So nginx, docker, and so on. Exactly. And I don't know if that's right or not. Yeah, that's basically the way you do it. Where do you store your playbooks then? Your playbooks are, that's a good question. Your playbooks stay in the top level.
44:01
So basically you keep the playbook in top level, the YAML files. So whenever you come into your directory, it's already there. That's something I didn't mention, but we have one playbook per class of servers. So for example, we have a playbook provision Linux master, deploy Linux master.
44:20
We're going to have provision OS X drone, deploy OS X drone. And then we tend to create a directory for the inventory, because we have dynamic inventories there and so on. We have a bunch of them. And yeah, like you said, the roles are just in roles and then you put them all in there. And that's really the canonical way.
44:44
Any other like super fast question? Or runs. Well, another quick one. Have you configured docker using Ansible? Have I configured docker?
45:01
Docker. Yeah, basically we have to do it all the time. Have you configured docker for user access, not root access? No, we don't need that. We do it for DNS and how it talks to the system, but that's basically it.