So hi. I'm here to talk how to build systems at scale, how we build it in the past, how the future is looking like and how to scale that out. There's a lot of false info around but don't despair, there's help around. As you can see here

that little tugboat trying, trying, trying so hard to prevent the catastrophe or the little helicopter. Have a seat. So first a little bit about me. I'm Andreas. I've been doing IT now for round about two decades or so. I hang

out a lot of the CCC. I've been to nearly every FrostCon so far. Thanks very much for having me. It's always great to be here. In the past I've been very privileged to be at a small Amsterdam startup called Booking.com when we grew that one each year 60% year over year. So we started out with a few

machines, a handful of people, five people in the office and we ended up being the world's largest hotel booking site with I think by now 10,000 people, billions of revenue each year and this talk is a little bit based on what we experienced there, where we've seen what works and what we see

nowadays is very much hyped in the in the newspapers or the CIO Weekly magazine or anything the area or if you Twitter or hackers news it's it's not all accurate in my opinion so that's why the talk is contain yourselves think before you follow the herd. Since I did the booking gig I've

stepped out a little bit from daily operations. I nowadays do a lot of information security in the financial industry which is very good because it allows you a little bit of reflection. You're not involved day to day anymore so you get a good overview and while doing that yeah I've seen as I said before I've seen what works and what's what I would consider a waste of time

so let's let's start with the good old days right a little bit have a look back in time we all know the good old days everything was made from wood and it wasn't so much better right you can see it nowadays too if you follow

some people on Twitter they go like you know I I'm not up for all this IT shit anymore it's it's horrible the people are horrible technology is getting worse and worse I think I'm I'm seriously considering woodworking unfortunately it's wrong the the old days they were not really better they

I would say they mostly sucked if you look at corporate IT you had processes IT was always the gatekeeper IT's job was to prevent people from doing work there's the old joke from Dilbert I think it was Mordor or Mordak the prevent of information IT was seen as a cost center you had chargebacks you

had all these processes which at the end of the day kept your people from actually getting work done everything was horrible there were so many silos you had the the typical life cycle for an application in a company would be you

have the architects they designed the software you have developers who write the software there will be consultants to install the whole stack and afterwards is handed all over to your operational team to run it and the users to use it but the users actually just hate it because it's so bad bugs

get fixed yes of course bugs get found first then they get fixed eventually usually this six months time between releases because it's so complicated every release you take down the website for two days or something because you need to reinstall everything and if you're lucky you

get the bug fixed in time because there's QA review that the change advisory boards needs to agree that this is important it needs to be tested again and maybe the test was not correct so it's actually not it's not going to happen this year it will be next year as I said it's it's really horrible it was kind of like IT done by checklist here's the checklist

monkeys go to the checklist you'll get bananas at the end of the day not a good time so since then things got better we got the thing it was the internet and the internet is really really great we know all the song things have been really tremendously increase in speed we got the web

applications you have called that native software providers they run they don't want a brick-and-mortar style they only have a website they can do upgrades every day stuff gets improved and delivered to the customer

immediately one rollout five minutes later the world has seen the new feature you don't need to test it anymore that long you don't need to ship it out you don't need to bug the customers with upgrades and you see Microsoft following with that with the windows 10 policies where upgrades are mostly updated automatically you see software like Salesforce is the

software as a service application being very successful because they can roll out new features so much faster than Oracle apps or any of the other SAP customer or other ERP systems so this is seems to be a real success so everybody wants that you have old companies brick-and-mortar stores

enterprises they see these web applications or websites and they go like this works really great we want that too but they can't compete they have the six month cycles everything is complicated they have the processes and suddenly a web startup comes along they are much quicker they can deploy

quicker they can do upgrades quicker they write new features there's a new article in the news or a new hype they can pull it out in say a week or something they have something which works customers flock to it because it might not be perfect but it works and I'm so much quicker with Oracle I need to wait six more months or something and maybe I don't get it or it works but I

need to buy another one and that's the concept behind the disruption by web startups your web startups are faster more agile leaner than your old companies and they eat their lunch uber is a much touted example it's not

really a very ethical company but they seem to be very successful with the with getting the message across that they are much better than the current taxi industry but it's quite important to to see how was that done right let's do a quick recap how the web native companies can be that quick

first of all they do generally need a very very clear business objective what is it that you actually want to solve for booking for example it was a very simple idea the idea was let's make it very easy for customers to book hotels worldwide that fits into one sentence that's all you need to know

because at that point you people can define it can decide that okay this is what we want to do we have ice in my area of responsibility I see these three four issues which prevent people from doing that simply because there's too much confirmation needed there's too much paperwork to be done can I find a way around it can I fix these things with such a simple message everybody

can basically take charge or can ownership of these issues and get them out of the way make the booking experience easy so this is the one part you need but you also need a very clear tactical objective that could be allow quick rollouts of new features we want to make sure that a developer

is able to roll out a new software feature in he writes it in the more say typical example could be we have a project owner who understands the business part of the application he has an idea let's it would be really cool if we could try this out he finds a developer in the morning who's available

maybe it's a new scrum one or so and he has one he talks with developer developer says you know that that'll probably take me three hours so by lunch the developer starts and he's done by four o'clock in the afternoon wouldn't it be nice if you can just try it out and see if it works so the idea is the developer should be able to roll it out without any interaction say

in 30 minutes tops that's a good feature or a tactical goal in the beginning we had this it worked fine in development let's throw it over the wall to out to production operations and the other alternative is it works on my machine the typical example of from systems operation so that would be

really works in your machine awesome copy your email off your notebook goes to the data center not really that good it's what I said before the development writes operations once it's very difficult to scale out at that point because you will have too much friction there's again the tickets

between teams and all that so they decided this will have to go to scale out development needs to go faster we can't wait for operations to do these things to do the one hour install take down the website for four hours that will that will not work so development got a lot of interesting

what do you call this processes or different changes in processes this agile programming they're scrumming this extreme programming pair programming name your poison there's several in the industry for the development process the idea behind all those is find a way to make it

clear what the priorities for development are have them write the software without any further distractions from the customer changing requirements for example which you will have that the customer thinks you know what be really nice of that color could be red and tomorrow now I thought about it it's blue is really nicer or maybe move can we get green maybe that's not a good way because you're you're hunting after the

customer for that situation scrum is actually really good because you define the requirement and then you have two weeks to implement it and afterwards it's demo time the customer can say yeah that's okay or no I would really like the green and then it's like okay absolutely we can do the green where do you want it is it really is it that important doesn't need to

happen right now or should we put it on the back of the backlog because remember if we do it right now that means the next two weeks we are doing green and a very often the customer at that point will decide that you know what it's fine with the red we said before it's it's more important that we get a new feature installed so that works quite well but once the code has

been written it needs to go out to the service because your web company and you want to roll out the features again right that's the tactic go deploy code quickly so this needs to be automated code deploys typical example we want to automate the code deployments this allows for the faster and easier

process I said before you look into different options use you might start small with an SSH copy or so you just copy the code to the website we start the web server all is good if you're working a bit larger and say you don't have five web servers we have hundred that takes a bit longer so you might want to parallelize it use PSSH or PSCP maybe you're looking at

our state because the data is getting larger so depending on your needs there are different options right you need to find the right ones for yourself while the developers people are doing that the operation people haven't been lazy either right they've been busy as well they've been automating machine configuration changes this puppet for example for that the chef this

ansible is one of the greatest ones is sold maybe you're still using CF engine and or potentially you might even have written your own shell scripts not the wisest decision because you need to do suddenly support your whole infrastructure button yeah shell script works there's even a configuration management system written as a joke called just damn shell scripts it's on

github it's it's if you have into shell it's a nice way to do it fix it otherwise I would recommend go for the real one chef puppet salt ansible those are the decent ones um you're also looking at automating machine installations because it really doesn't scale well if you have to send someone

out with a on sneakers with a disc to the data center and go like this goes in install this goes out next machine that's not good you would be looking at PC network booting remote installations maybe using I lo whatever your hardware has or if you're lucky and you're using Amazon well there's a

rest API you can just install the machine maybe you work with a golden image maybe you want to install by scratch it's really up to you you can completely decide what you want personally I'm not a big fan of the golden image is one because I find it lacks a little bit the agility because if I want to update one package or so I would need to re-spin an image

then roll it out to everyone reinstall everything if I use a regular say puppet for my package management it's just package added and 30 minutes later I'm done I don't need to reinstall the machines I'm good but it's up to what you it works for you the benefit of all these things is that you get a repeatable and transparent configuration change

applied it doesn't matter if it's a single machine or hundred thousand machines or a million doesn't matter if it's on bare metal on the cloud change route LXC Docker makes no difference at all right provisioning should be done in under five minutes there are companies who have if they're working with bare metal they know exactly the capacity planning they have

every quarter they buy new hardware 400,000 they know how much they need this quarter and if you need a machine it's already running in the racks just connect to it other people might be deciding okay you know what we can live with a higher latency for scale out because HP delivers takes say five weeks right if you know we all need something in six weeks that might work

for you too if that's not working and you can't afford the having the spare hardware around go look at cloud look at virtual digital ocean Hetzner might be something decent in Germany this again thong thousands and thousands of options it's all up to you the idea behind all this is it's called infrastructure as code it's a term formed by Etsy who've been

revolutionizing the eBay for handcrafting things and they've been very very successful at pushing the message how they work and they've been very active at conference like velocity or monitor Rama generally I would suggest it's a interesting talk with them if you ever meet any of the

engineers have a good talk they they generally know what they're doing it's always very interesting to get new ideas how other people are working on it afterwards you deployed your code right you probably want to test it this is not a joke that's what a lot of people do right because where are

you going to test it are you going to test it in a special testing environment how do you make sure that the testing environment is actually accurate to your production environment how do you make sure that the load that you get a good understanding how your load is behaving right you've just rolled out a massive change to your SQL layer can I tell how that will be

behaving I would need to get an exact copy of my environment I would need to get an exact copy of the behavior of the users I might be able to sniff some traffic replay it on my testing environment I can really spend a lot of time trying to recreate my production environment but generally I will always forgot something and my testing might not be as accurate or anything so a lot

of companies have decided you know what we're just going to test in production we are going to build our application in such a way that we can survive if a server fails during a test and the customer will not notice they have decided or they have found a way for example that booking was that every Wednesday they did a capacity test in production it meant that you would

take one cell which is a set of web servers and a database or and the number of database service and you would just increase the priority in the load balancer so that they get more and more and more and more traffic and one point you see that response times go up so that point you know okay we're done

we have found the capacity ceiling of that cell and we have hundred of those cells so we can take the number of requests we could serve with this one cell multiplied by hundred and we know what we can do and at that point you know if say August is your peak load you can then decide you know what August we are here this is what we'll see in August we're currently growing

we are safe or it be I might need to talk to management that either we now spend time on fixing some performance issues or we just buy new hardware maybe hardware is actually cheaper who knows it again it really depends on your situation so at that point best student production it doesn't get more realistic than that and it's risky absolutely it has a

certain amount of risk I would not recommend it for example for metal manufacturing company probably not the best idea Airbus now I would not recommend to test out the latest code right on the production suite but for web shop yeah absolutely banks maybe not on the money path but if you're

working on the PR side or a landing page for microsite yeah why not might be good enough again it all depends on the situation and if you take all these things and you add them together you get this oh my god it's devops right to make all this work and satisfy the business objective you meet you can't

continue with the old style anymore you need to abolish the silos they don't scale they are horrible barriers to entry they they make your life miserable basically and they slow you down so you need integrated teams where you developers and your operations people and they do work together they

know each other they they talk they sit together in same corner they have full responsibility for the whole stack you can't just go the well developments over their operations there the us with them doesn't work anymore you need to get rid of that um the the old style where you had the built one deploy or what I explained earlier with the the consultants the

architects all of that needs to go that needs to be one full team and if you do that that way you can actually deliver the velocity and agility you want you would say normally this is common sense you know it's it's really important or really good that your developers and your engineers talk to each other but this is it's actually very rare in a lot of companies that

it's so where that there's this joke says common sense is so where it ought to be a superpower and DevOps what I said earlier the name has basically codified all these ideas have you people interact with each other have full and shared responsibility have a transparent stack one team it

supports a whole stack and transparent manner that's the idea of DevOps and it's not a tool thing it's basically how you interact with your people it means failure needs to be accepted because if you are working that fast yes you will fail that from time to time and that needs to be well understood that that will happen at booking for example we had a budget

each quarter was hundred thousand euros or half a million or whatever it was we'll lose this year through human failure and that's totally accepted that's just the cost of doing business and that's a price we are totally willing to pay because it allows it to be so much faster than the than the competition and the budget for failures for example something

completely new in the airline industry even for the non-critical parts it's just not done and when they heard about these things they were like you know that's actually a really interesting thing I'm not sure we could get that into our culture it's it's you basically full support from your management team to make that work but if it works you're going to be very

quick the typical success stories for this would be Etsy booking.com Facebook Google Amazon Netflix all these very large very successful companies do the DevOps thing in one way or another but there's also smaller one startups tons of companies here if you talk to them they actually explain how they

work and it's very very much the similar thing so yeah that that's basically what works but if you look at it there's a fascinating observation because if you look at the the way people are incentivized what works for them where are their bonuses and how do we judge their work you'll see that

development ops while they are in the same team they still have very different incentives your feature development is generally purely results driven it doesn't matter how they got to the code they got doesn't matter if they had to take shortcuts or if it ugly code beautiful code really does

not matter the important part is the features there or it might be that hey you know what that release here is 5% quicker which is really good because it saves us from having to buy 10 more machines this year it will be great it doesn't matter that the SQL query is not the nicest or that they use copy and paste from somewhere for your operations however the it's a very

different game the the result is not actually that important it doesn't really matter if the deploy was error-free or if it had to be rolled back or anything there was a code nice was not no no effect the important part is that it's well understood how they got from the old release to the new release that this is repeatable that they can do it not just on one machine

that they can do it on all machines that they can roll it back that they can verify was it successful is it running is it running well or is it not running well these are all things which are important for ops but not for development Chris current up former colleague of mine from cooking and a good friend explained a very nice yesterday this talk go away or shall we replace

you with a nice shell script he says the infrastructure development operations is judged by the worst case while the feature development development is judged by the best case for feature development it's always oh this is really awesome this is a nice little feature for kernel development as you can see on the Linux kernel mailing list they never

ask hey this is a really cool feature that looks really nice how does it work in the best case the scheduler for example is always so what's the worst case how much worse will it get if we apply that patch where are the bugs how bad will it crash that's a not necessarily the call it the most rewarding experience you've just written a new patch you're super

proud and they go like yeah so how does it break it's I assume it's not for everyone right but be that as a mate we have an amazing new tools right the futures is totally awesome because nowadays we have containers they solve

all these things and specifically we have Docker which is the best ever this is Dan walt from redhead who does a lot of work on Docker and getting it secure because by default it isn't and he always explained that it's Docker Docker Docker you need to say three times it works really well and as I said we have we have this amazing new technology called containers the most

famous example would be Docker and they promised to solve all the examples all the problems I just described before they you can solve with one tool it's amazing right because all your code you don't need to worry about applications or machines anymore all your code is nicely contained in the

application container you load it on a boat and the boat goes and that's boat goes to production and there it's installed right you don't need to worry about dependencies you can deploy it anywhere because every container is the same doesn't matter if you're running it on Apache on a Linux machine on a Windows machine or whatever it's all in there it's amazing and best part it just

works that's most important so if we look closer what doc actually does we can have a look at the website that's what doctor says it's straight from the website it's an accurate portrayal of how doc issues don't worry so much about reading it because I highlighted the important parts as you can see it's a platform for developers answers admins to assemble applications from

components and then the getting deployed into production as fast as possible right sorry and now it's actually not that bad if you look at it right it doesn't talk about synergies it doesn't talk about web scale it

doesn't have react MongoDB whatever so it's it's kind of okay right um it's a decent description I would say but there are problems with that because for example here I see it's developers and system so both the whole DevOps thing really great and here it talks about getting code and deployed into production so wait that's not what sysadmins do that's what developers do

right and it's it's actually a little bit yeah it allows it develops to pack up code into production well haven't we seen that before and I'm not talking about the the old joke about the right ones run anywhere from Java no I I'm more thinking like you know we're fine enough it's ops problem now it

sounds a little bit to me like they've here's a binary blob it works on my machine go install it now in production and it feels like we've seen that before and we've decided in the past that that's not the way we want to work it does not work and I had thought we had gotten completely rid of

that attitude it brings the old joke to mind as well the if you don't understand package management you're forced to be implemented badly because that's exactly what Docker is being pushed at or hyped as it's package up your code ship it to production it'll work and I'm a bit surprised actually because see the technology behind Docker is it's not the best but

they actually managed quite nicely to get to get it to the masses and get containers ready for use yet all they seem to be interested in is packing software and managing dependencies now that's something which has been solved 20 years ago 15 years ago with app get RPM has solved

with a yum yeah I would say but yes but that's where it fails Nick's like

that right no problem no problem but the dependency in software packaging that's well understood right especially if you're within a company it's a little bit better or it's a little bit different on the wide Internet for example when you have these nowadays these appliances VMware appliance or so

it's downloaded test drive this piece of software at that point Docker actually makes a lot of sense but for a rolling out application in enterprise don't think so and I'll explain why so recap um the failboat we have a tool which completely ignores the best practices we've learned the past on building scalable infrastructure we are not doing

the the teamwork anymore we are having a development team which is packaging software in some way right nobody knows exactly how it works because there's this Docker file but there isn't actually guarantee that what you're getting in the Docker file is actually what's described in the build file Docker is working on signing things but again that's just more like

yeah it's really coming from the person who claims it's coming from but it's not a guarantee that what you're getting there is actually accurate or something you're getting a binary blob applied on top of another binary blob if you look at the examples you'll find something like let's take this cloud image we'll add this club image binary from Ubuntu we'll add

this Apache binary on top and we'll take the MySQL binary on top and then we'll add a few more config files and we got another ready-made web server the problem is there's no guarantee that if I build that with the same Docker file in two months again that I'm getting the same because maybe you

you grab data from daily binaries or you've seen the Debian Docker files where code is downloaded and installed via tar.gz from the daily builds there's no guarantee that what you're getting is actually recreatable or anything that too but on the other hand if you're building your own you're

probably safe with that because you can make sure that your people that the application these you're collecting is applying to your company standards but let's say if you take the base Ubuntu image you are most likely safe but I'm not sure if you're safe enough for your lawyers for example

and that depends again on company size right the interesting thing is we've we've kind of had that before that you're getting your binary blobs applied on top of binary blobs and then shipped into production to run applications we had about 40 years ago it died 25 years ago and it was called Smalltalk and that's a good reason because shipping a system state

in time has never really been really manageable and has been a good idea because Smalltalk tools were often written in a way that here's the code but you know we just live patched it packaged it up and that's what you want a good idea typical example in the company for example is imagine you have a part of Docker or you have a nice environment of thousands of machines

very heterogeneous and you want okay how many machines with heartbeat open SSL or shell stock bash do we actually have if you a old-style enterprise you can ask you or say you're redhead enterprise satellite server they know and you can fix those machines if you're running puppet this factor it

knows the answer to that if you're running chef or high knows the answer if you're running ansible or salt there's answer for that too right it's all managed if you run Docker the answer unfortunately who knows there's no way to tell what's in your package which becomes a bit of a problem if you need to decide okay it's time to reroll those packages or those images or not

you can fix that by installing puppet in your images for example but that's not really smart because suddenly it's not an application container anymore you're actually building a full virtual machine in Docker and at that point why do you bother anymore can't you just packed up the code and sync it to your destination for example but the good news is even if

it's even if it would be horrible right it can't be that horrible that can't serve as a bad example to others that's actually the sad thing I find in Docker the technology is okay they're really good use cases for containers and Docker successfully managed to to get them to get a better

interface in place on the Linux containers LXC if we look at for example the centers people they have been bootstrapping go I think in two videos one to two and one to five for arm now and they did it with Docker containers because that was the best way how to get a complete build environment separate from your host chain and then build something or cross

compile it so that he can run it on arm seven I believe is what they're working on now that's actually quite cool because it's quite hard to do with change routes because you will always pull in information for more information from your host you could do it with a full virtual machine but at that point you have a lot of hassle so that contains was actually a good choice and yeah that's great or if you are say you are a virtual hosting

provider like host Europe you might decide that it's not a nice idea to have a lot of the shared web server where we have 50 me servers or 50 hosts 50 users on and we separate access to them via say as you PHP for example you could just decide every web server gets every customer gets their

Docker container with just an HTTP D in there and the PHP daemon or PHP fast PHP or just a PHP mod or whatever they want to run what Ruby my passenger mod Python whatever you want to run and we'll have then the local answer or the proxy server from Docker and we direct the traffic to the right machines that's actually really really interesting use case but I don't see a

lot of people talking about that it's always still oh let's push our code into production with that if you have Se Linux working you can actually get it quite nicely separated but if you don't you can get out of container it's it's

kind of like giving root to someone on the internet if you if you take a third-party random container right but if you build the containers yourself you would get more security for example because you're running you make sure that your patches running as nobody in there or as a user you would be getting way more separation than if you run a multi-user HTTP D on a single

machine right so it's not perfect it's not a VM but it's way better than what we have currently in the multi-user hosting environment for example but so that's why I say it's a very sad thing because all the hype is considering containers as just another packaging format and it's it's very

backwards because we are we want the DevOps benefits as we've seen they work everybody wants them I've been working in the financial industry I see a lot of banks considering how can we use those how can we use the lessons from DevOps and all that in our environment we want to roll out code

quicker it's we've been currently okay with taking down our website for four hours in the morning at four o'clock and roll out a new release every quarter and then have 15 people so get up early to test if everything is okay but can't we do it better maybe and I know that certain banks for

example ING in the Netherlands is very very involved with the the quality modern computing scene around there and they they go to a lot of the DevOps days they try to get a lot of input from the community how they can do things better and some things work for them some things fail horribly just they

are trying to get 20 year old 30 year old stacks of code sometimes 40 year old still running COBOL into modern environment with a fresher which which has never considered which never considered those cases when they when they build these technologies but for some things it actually works so that's

quite impressive but yeah with the DevOps we want the DevOps benefits but let's completely ignore everything we learned there that's not a winning move that it's not really called progress I think it's called backwards it's completely backwards actually and that's why I'm like yeah I'm not a fan of the current hype of the the yeah the docker container wave there are

better ways to do that if you look at for example the scalability right thank you yeah there's the way there's more right we we see scalability how to do it on the intent it's it's always very interesting to read how other people do it there's fascinating reads how Facebook scales out how Google

scales out and it's only very interesting to take away the current hype I think there would be microservices and containers being managed by Kubernetes it's the latest and greatest it will solve all your scalability needs because what you need to do is you take your application you separate it into special services so for example you have

a webshop right you will have one service which does the querying your warehouse if you have still stuff you have one different service which will do the authentication you will have one service which will do the billing and if you get a nice little armada of services that way you will have the nice benefit that you can take down one service for maintenance while people

can continue surfing through your site maybe you can't have new people logging in for the five minutes you're doing the rebuild or something but your full website will still be working and you can see this Facebook does that for example very often that you reload the page and suddenly there's a story missing which was there before or maybe your chat box is loading a little bit

slower or anything these are all different servers in the back end and if a service is not working that's totally fine you can still use Facebook it's just missing a small feature or anything so the idea is very nice and if we can package those up in containers we can have a whole fleet of them on on a few machines or anything and then we need the

management environment but the problem with that is there are places where this works and where you can get benefits from that but realistically chances are it's not your company it's not your project the this is let's say a typical example you would have is you have your Kubernetes or you have your

containers right we said before you manage your hardware you manage it with puppet or chef or ansible if you have virtual machines on Amazon you images you probably use chef on them too because you want you want to you don't want to install your central say database all the time all over again or anything if you're not using the RDS from Amazon so you will have some

kind of configuration management system and now you're getting the containers but it doesn't come with one and it's not it's suggested you don't get one for that so you're looking a little bit what do I do now oh wait there's kubernetes right that's awesome let's use that and now you're spending time to build up a completely new infrastructure just for

mentioning in containers without getting any of the synergy effects and there we got the bingo from your existing hardware and that's just not a winning move right there's a different talk of mine which describes a little bit more how we've actually scaled out message leads basically best lessons learned from booking but one thing we learned there

was growth takes time right it does not happen overnight you will not have the situation that today we had five visitors a day but tomorrow we'll have five million suddenly consistently over the next year so we need to be able to completely scale out from one day from five one machine to five million to 500 machines that just does not happen right with every order of

magnitude you grow so from one to ten from ten to hundred four hundred two thousand to ten thousand two hundred thousand machines you completely need to revisit your whole architecture nothing will work anymore right and typical example is one machine you installed that by hand ten machines you probably kick-started it but didn't do anything else hundred machines you

would probably be looking very much at puppet or other configuration management systems thousand machines you will definitely need something automated to install and configure manage them ten thousand machines yeah puppet will not work for you anymore because it's just so much change and everything and there's so many failures you need something which deals better takes out machines which fail out of the load balancer at that point

a very funny quote from a Facebook engineer was so I'm looking at a machine which has an error right developer says machine is acting funny I log into it be SSH and SSH doesn't ask me for my key so this means I have locked into that machine before obviously this was broken before so

let's just trash completely we're not even looking at it anymore because yeah I would normally not look into it I would not log in if I log in it was broken get rid of it that's something you will not that attitude you would not see was ten machines but at ten machines you would still be looking at what's actually going on I need to try to understand what's going on that's how to see how to fix it so it's a completely different thing right and at

that point when you need to redesign your application completely anyway then you can decide is it now sensible for me to go for containers Google for example yeah absolutely they have a tons of containers and tons of applications that works if you're a webshop or a bank or something with three four five

maybe in ten applications the hassle of running a separate management system for that it's not worth it so look into your existing tools if you can reuse them and the other one is also in business perspective do you really think that it's a good way to spend your time now develop all the configuration infrastructure and the management infrastructure and all the

code frameworks for your awesome container situation or anything or do you think it might be a better way to spend your time maybe writing new features which attracts customers to you because customers really like features they couldn't care less how you're running your code if you're running in a container if you're running on bare metal if you're running a virtual box or whatever it is you like right they don't care and I

think at that point I'm actually at the kind of the end of that I have two more interesting reads one is an awesome one it's called the future it's a future at Paul bigger it's on circle I it's a very sarcastic view of the current state of distributed systems so if you in that into that go have a read

it's you're rolling on the floor laughing it explains how a business guy is coming to a web developer and asking so they say you you know how the new things work could you please explain to me how it works and he wants a he wants a single web app something very simple right he ends up with about 80

into 90 containers probably on eight virtual machines or anything and a bit of management infrastructure but it's really great and scales out very nicely for a single web app hilarious read and the other one is if you're more interested in how we did it how we built booking at that point is eight rollouts a day it's a presentation on Christian again so slide share explains

a little bit the business requirements a company had at that point what they decide was important how they decide how they solve it and what they have to accept for a company to to be able to do eight rollouts a day how much errors do you expect expect the day how much effort you need to put

into capacity planning what errors will you find on your way what are the traps what works well what doesn't work it's again very interesting read and on another note if you should be in interested in MySQL or MySQL performance and you announce them next month there's a quite interesting

keynote at Percona life there the Percona performance consulting company they have a yearly conference called Percona life this year it's in Amsterdam it's in September there's a very nice keynote of how booking was built and using such amazing technologies as bare metal hypervisors

hipster free technologies such as Apache MySQL and pearl and how you make billions of money with that per year and I think the keynote title is boring.com but it's totally worth it because it works and with that I guess thank you very much for your time on you have any questions Alex I expected

a question what was that to your happiness all right thank you very much if you have any other questions reach out to me on IRC or anything I'm

always happy to share anecdotes and systems it's great fun thank you