Case Study: Switching from Linux to FreeBSD


Formal Metadata

Case Study: Switching from Linux to FreeBSD
How technology supported advocacy in ExperiencePoint's conversion
Title of Series
Chvostek, Paul
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Berkeley System Distribution (BSD), Andrea Ross
Release Date

Content Metadata

Subject Area
This talk will outline the strategy used to migrate a small Canadian software company's Linux-based server infrastructure to FreeBSD. Part advocacy-strategy and part best-practices, the hope is that you'll come away with some extra tools to promote implementation of FreeBSD in your workplace. ExperiencePoint is a small (20 person) Canadian company that creates training simulations as web applications. The business is wholly dependent on its web server infrastructure for delivering its product. In 2011, I started working for ExperiencePoint and began the process of replacing its aging collection of Linux servers with a more robust FreeBSD server infrastructure. The Linux servers in question had been set up in a hurry, and the skilled software engineers who had set them up were not professional systems administrators. Linux was selected as the server operating system, but there were great opportunities for improvement and change. This talk is the story of that change. In addition to addressing the management concerns of replacing a "known" (Linux) with an "unknown" (FreeBSD), we'll explore the kinds of opportunities you should recognize in Linux environments you may come across. If you can improve reliability, reduce risk and improve performance, that's even better job security than switching to an operating system that nobody else knows.
Point (geometry) Server (computing) Freeware Spline (mathematics) System administrator Flash memory Mereology Information technology consulting Wave packet Revision control Web 2.0 Mathematics Reduction of order Office suite Information security Units of measurement Physical system Addition State transition system Spacetime Constraint (mathematics) Process (computing) Product (category theory) Software engineering Bit Directory service Set (mathematics) Cartesian coordinate system System call Radical (chemistry) Process (computing) Software Integrated development environment Internet service provider Mixed reality Quicksort Remote procedure call Whiteboard Data management
Point (geometry) Product (category theory) Confidence interval Characteristic polynomial Expert system Interactive television Mereology Theory Information technology consulting Wave packet Goodness of fit Mathematics Strategy game Term (mathematics) Ideal (ethics) Energy level Game theory Quicksort Data management Social class
Point (geometry) Machine vision Group action Server (computing) Product (category theory) Scientific modelling Flash memory Client (computing) Group action Mereology Particle system Computer simulation Mathematics Integrated development environment Labour Party (Malta) Bit rate Internetworking Equation Game theory Quicksort Office suite Social class
Simulation Server (computing) Product (category theory) Decision theory Scientific modelling Sound effect Bit Arithmetic mean Mathematics Network topology Data storage device Game theory Arithmetic progression
Spacetime Scientific modelling 1 (number) Mereology Field (computer science) Computer icon Theory Wave packet Number Cognition Mathematics Hybrid computer Subtraction Data management
Point (geometry) Machine vision Multiplication Mathematics Integrated development environment Scientific modelling Fitness function Combinational logic Convex hull Bit Quicksort Physical system
Topological vector space Building Group action Decision theory Scientific modelling ACID Replication (computing) Mereology Web 2.0 Database normalization Maxima and minima Direct numerical simulation Casting (performing arts) Video game Mathematics Single-precision floating-point format Diagram Vertex (graph theory) Physical system Email Spacetime Constraint (mathematics) Electronic mailing list Infinity Bit Hand fan Connected space Maxima and minima Arithmetic mean Graph coloring Website Lastteilung Absolute value Cycle (graph theory) Quicksort Data management Point (geometry) Server (computing) Presentation of a group Connectivity (graph theory) Virtual machine Point cloud Number Causality Term (mathematics) Average Internetworking Database Ideal (ethics) Graph drawing Absolute value Metropolitan area network Expression Limit (category theory) Calculation Database normalization Integrated development environment Vertex (graph theory)
Machine vision Scientific modelling Decision theory Insertion loss Replication (computing) Proper map Web 2.0 Summation Maxima and minima Mathematics Strategy game Linker (computing) File system Cuboid Office suite Data conversion Extension (kinesiology) Information security God Cybersex Boss Corporation Process (computing) Block (periodic table) Software developer Electronic mailing list Functional (mathematics) Process (computing) Phase transition Lastteilung Right angle Whiteboard Freeware Data management Reverse engineering Point (geometry) Trail Server (computing) Implementation Virtual machine Online help Surgery Number Database Authorization Energy level Router (computing) Associative property Subtraction Proxy server Machine vision Lemma (mathematics) Planning Group action Cartesian coordinate system Table (information) Database normalization Integrated development environment Computer network
Teilnehmerrechensystem Machine vision Outlier Scientific modelling Decision theory 1 (number) Leak Web 2.0 Chaining Plane (geometry) Mathematics Computer cluster Linker (computing) Stress (mechanics) Software framework Diagram Office suite Error message Decision tree learning Area Process (computing) Software developer Electronic mailing list Shared memory Physicalism Bit Price index Process (computing) Phase transition Hard disk drive Convex hull Pattern language Lastteilung Right angle Whiteboard Quicksort Data type Data management Reading (process) Server (computing) Computer file Login Latent heat Root Energy level Scripting language Key (cryptography) Information Projective plane Planning Directory service Group action Cartesian coordinate system Local Group Calculation Cache (computing) Word Software Integrated development environment Backup Identity management Separation axiom
Revision control Server (computing) Term (mathematics) State of matter Operator (mathematics) Projective plane Electronic mailing list Insertion loss Line (geometry) Cartesian coordinate system Data management
Point (geometry) Product (category theory) Electronic mailing list Expert system Archaeological field survey Planning Bit Field (computer science) Maxima and minima Spreadsheet Mathematics Order (biology) Energy level Configuration space Data conversion Abelian category
Machine vision Group action Server (computing) Scientific modelling Virtual machine Product (business) Web 2.0 Latent heat Goodness of fit Mathematics Computer hardware Streamlines, streaklines, and pathlines output Physical system Boss Corporation Product (category theory) Projective plane Electronic mailing list Planning Term (mathematics) System call Telecommunication Phase transition Order (biology) output Arithmetic progression Data management
Point (geometry) Machine vision Game controller Server (computing) Scientific modelling Transport Layer Security 1 (number) Shape (magazine) IP address Metadata Web 2.0 Web service Mathematics Database Automation Proxy server Subtraction Units of measurement Address space Physical system Decision tree learning Product (category theory) Seitentabelle Shared memory Set (mathematics) Virtueller Server Word Software Integrated development environment Computer network Phase transition Backup Lastteilung Quicksort Reverse engineering
Point (geometry) Machine vision Sequel Decision theory System administrator Mathematical analysis Group action Disk read-and-write head Mathematics Software Natural language Quicksort Address space Reading (process)
Point (geometry) Slide rule Server (computing) Divisor Decision theory System administrator Modal logic Scientific modelling Connectivity (graph theory) 1 (number) Insertion loss Weight Wave packet Number Web 2.0 Goodness of fit Mathematics Bit rate Database Bus (computing) Energy level Physical system Installation art Boss Corporation Mapping Software developer Projective plane Electronic mailing list Variance Benchmark Proof theory Arithmetic mean Process (computing) Software Integrated development environment Finite difference Order (biology) Computer network Right angle Lastteilung Freeware
so so I'm sick I'm I'm currently working at a small company that I create software for training quality also say that we create the against it's in a 20 person company that has a co-located servers and an application a set of applications that are written in Flash communicate with those servers and and yeah the business is completely dependent on its on its Web hosting infrastructure for and for delivering its product and I so in 2011 as working there and I started the long drawn process of converting the aging badly set up Linux infrastructure to something that would be a little bit more resilient and uh able to deal with a whole bunch of challenges that we identified and that the Linux servers and sort set up in a hurry so they they weren't really set up well they're running a bond to and they had done a the end at least they had been that the version that had been selected was a long-term support versions of that was kind of a plus but on the guy had set them up you know is a skilled software engineer but he didn't really know very much about how to do systems so he did the best he could given the given the tools at his disposal but time but there were great opportunities for improvement and change so this is the story of that change yes no was what 10 LTS and zeros of 4 yes something which became more and without I can never constraints so in addition to addressing management concerns about the the node and that that was already installed replacing a known with an unknown we have and we're going talk about some of the opportunities that you should recognize in linked environments that you may be dealing with that you may come across and basically if you can improve reliability and reduce risk and improve performance that's even better job security than replacing and old on I so the in work so OK my 1st full-time job I ever it was in 1990 working for a company that and SKOS units of gold and so the the this is just a little trickier history and skills UniTran Fock space applications and we had terminal servers that ran on uh we had to do boards and there's a 3 to port multiplexer that allowed us to have a remote terminal servers and in in another office across the street by the spline and this it was fun at the time you know was using new and interesting technology but but it was it was very much of its time and so I was there for a couple of years I moved onto the unit Canada Canada's 1st commercial internet provider and and that suffering there was mostly some ask for and then there is a bit of Solaris coming in at the time but there were but I think some other things as well but it was ancient history and we might have a little bit of 386 BSD the then yeah so Solaris 2 was was part of that and then then I moved on to Canada's 1st public access about writer and we we're using BST less at that point I'm not sure that's the right way of thinking OK cool and so we use this US we also use Linux we uh had to you know use a very old version of Linux I think it was pre version 1 so was late 93 that Linux 1 came right yes that would be 1 I remember the party that we had when we had 24 hours been to use up to that was that was fun and then we got a call during part the it and so then that company that sold to another company that was using it the but where there is a mix of previously in the but then I got a job to bring Canada running Solaris 2 5 which I think I think was the last version that included a user UCB directory where you had all of your Berkeley tools and so then after that I was a consultants and went through all the previous the version 2 through 5 through war and then got a job at the the Canadian Broadcasting Corporation as a Linux and Unix administrator in this is kind of fun because they're running RedHat and uh part of my job was to migrate them over to Susie but it was no no they're running Manhattan Slackware and then we had migrate overseas you can step on because easy and also an existing relationship with Novell which had to be respected and I think this since moved back to to read had given up on on these the other johnny-come-lately and so but there seems use also using Solaris and HP-UX and even had back the point of so thank you so
the interesting thing that I discovered in these are sort of early consulting days that I that I hadn't really count on was that was that if I found that in general the more successful the company the the easier it was for management to accept the idea of change and new ideas and I mean it seems kind of obvious when you look at it but when you're in it it's not always obvious on but more importantly young and small successful companies seem to be the best at this so if you're 1 of those that can become a lucky and the biggest obstacles to to change seems to be seem to be laziness and fear and if you have confidence in your ability to adapt them than it cancels out here change so I discovered something else which is that everyone is a child Part for 1 thing people love to discover and that they love new gadgets that making things that that of finding things that make them go went up and then they want explanation makes them happy and that if they get answers the questions that also makes them happy if they don't they won't be happy so used to say when I was a consulted that term I like my customers rich and stupid and in fact we all know this but the best customers are neither well no the rich but they're not stupid they're the people who have a very good idea of what history fast and how to achieve their goals and they're very organized and they know to trust the advice of the experts that they had because they know that they're hiring of experts so interestingly these characteristics work for employers as well and so as this 1st said when I started working at experience point here in ability look good theory i and I experienced point is 1 of the small companies that is a joy to work for it is as I say training businesses in certain things we have we have games so it's a it's a gaming the and atmosphere in the office and within the name like experience points would kind of expect but so we had to products that experience . 1 of them is this day-long class teaching corporate change management called experience change and the other 1 isn't a half-day interactive seminar on innovation which is called design and the design thinker is courtesy of a company in California called ideal who has come up with this strategy for training innovation I don't quite get design thinker but I did get the idea of change management and the interesting thing is that this is not change management
and title style it's completely different from anything that you would deal with at the IT level this is corporate change management so this is changing the culture and dealing with psychology and that but like so many of companies that try to establish themselves with other businesses they've got a model and
the model is this sentence that model that so that is developed from a couple of other models the i see that the the change model is is used it would be used by a company that needs to migrate from an old way of doing business to a new way of doing business and you know you would see something like this with Research in Motion or Kodak for pretty much any company that lasts long enough to outlive the environment that I grew up in is going to have to do a change at some point end of so that the thing is that all of the simulations are basically multiplayer games the they're written flash with back
and there are people who is rate so this is an example of Labour office and animations at 4 AM other things like and the thing is moved what I was thinking but anyway that I so so anyway actions that get implemented at the client on the client side in the flash game and flash uses its own sort of magic particle that talks to stuff running in each servers and if any part of the equation goes down the server they're having problems the Internet is having a problem we don't have a product and we can look so that's kind of bad when we can deliver however we get something that on the facilitators side so there was a teacher in a class there's somebody who is running the course and the facilitator has a number of
tools that let him do things like what a horse race of the teams are the other the participants in the game so we can see as as participants progress and how they're doing because all that stuff is storage was In our servers and so this is the portal and presses got access to this but if the i if we want we can review means after the fact and watch what kinds of decisions are made throughout the simulation and those decisions gets
mapped in this fuzzy tree that that shows their effectiveness throughout the simulation it's it's there is a little bit of magic to it and it's a lot of fun and I think we have a great product and it down and it works well most of the time and of course you know this when it doesn't work well that's that we have to come up with all this alternative ways of getting access to stuff but back to the change model the of the change idea this seven-step model of changes something that is based on
work by a few people I mean there are a lot of different people who have come up with models for how to deal with corporate change 1 of the 1st ones was this guy named
Kurt Lewin knowing hurdle this was the first one covers a current 1 there is a former professor at Harvard Business School and has written a number books on corporate change so and he's a time in MIT graduates is it is really well respected in the in in the in the leadership training and corpora change management field and then the other guys Kurt Lewin and he's dead now but
ironically the idea that work but in the early part of century he was out working on behavioral psychology and he he taught and came up with with theories and 1 of them was that was a 3 step model it is as if you don't like the John postal and Jordan hybrid of the change management world you know not just because 1 that but I there there's some of that the icons or if you're if you're in that space anyway so cognitive model of Lewin's model was a 3 step model and the idea was
basically so that when you're making a change you want to unfreeze the environment that you're in your I make a change and you want to rephrase the idea was 1st you convince organizational stakeholders the changes necessary and and good then you make a change and then you make the new way your permanent way of doing business and it is nice and simple manner you know long cane cutter and he added a bunch of detail so the
detailed sort of fits inside gluons model and it makes sense and then you know it gets expanded in courses and so on expand that model here but I will say that it was sort of what led to experience points
model which which kind of combines and reorder some of steps a little bit so I can't tell you why 1 would be better than another because the system and not change Training person but but but apparently our model as well enough like that it's taught alongside the car model and you know with the balloon model kept as a as a reference and it's taught in multi million companies in in 9 in respected business schools so I think we must be doing something right on OK so nothing
introduction let's get interactive this distinction is hands and How could you thinks and so obvious who here is solely or ultimately responsible for the architectural decisions in the places that you OK so hands down everybody who owns the company on the other half space so everybody who remains is you must be pretty lucky to be working in an environment in which you so well trusted that's rare and you know you have to if you're not 1 of the people with his hands up here you're dealing with struggles all the time and the next expression who here is really a Linux system in disguise at the other tho is the case so the idea here is that yes exactly that I know lots of people have managers who knows spend mailing list about cloud and I don't know anything about it that's the only thing that the is the cause never mentioned in this publication so that was my life at C and was the primary reason I left the CBC and then every day that goes by you hate yourself a little more it's time to change your employer so that doesn't mean the during le that means change employers I every machine that's you have deployed as a number of components and those components and this components will eventually die and the the mean time between failure is the term that's thrown around quite a bit and all these components are you know they're going to have an eventual life and they're going to die eventually in you might come up with calculations were you can estimate the length of that that the life span of a server you know at the shortest edit and TVs and that's uh that's a possibly when the server could go but really you know that's that's just a bunch of bull because this is not how things work the components will fail at any time and the NTBs is made up of averages and the problem in the DF is an average and individual component is going to to that average you can't predict when and where drive is going to fail I you can predict when a fan is going to start spinning spinning things start capacitors blow and that's just like so you will get a color model I and you or you know when you're on vacation and on a sales person has an important presentation doing and that's just how it works so the way Google would fix this is not how we we would necessarily always fixed I you have to deal with smaller budgets in the smaller counts and that you have to do with people who probably don't even know about society acid man asset management or our life cycle and maybe haven't even assign an annual IT budget or or planned for for upgrading things so well what you need is an upgrade path and the path he didn't point so before we before we figure out what we're going to do we should think about where we want to be so let's look at your simple
single-server environment so that's that's some bad for a lot of reasons of we need to build some redundancy in this we need multiple parts so so let's say they we had to web servers OK so that's fine and dandy but we don't have the we don't have everything we need at this point because to web servers are going to need the databases synchronized so OK let's add a replication let's but also and maybe will load balancer in front because you can't just change a DNS to have uh tend to have your websites hosted on a different server and well OK so I've got another single point of failure where you going to do about that will make to load balancers so you get the idea here to have everything but to Internet connections and well OK so maybe a 2nd rotor behind that 1 the and if you're if you're really worried about trucks going and you're building in or truck bombs blowing up the color than any cast and that's completely out of scope because I can't really diagram that takes still once we've got that once we got sort of an ideal it's something that we considered expressed management and say hey you know we need done we need to fix things and here's a here's a plant here is a way that we can do it I when I started experience points we had a very simple network diagrams of at 1st this doesn't look too bad you we've got to web servers we've got to database servers there's something that might possibly be you replication of you may not be able to make all the changes you want because of budget constraints are limited risk perception but know we can start so here's what I was starting with of there is a there's a bit wrong is that is wrong with that so we got an infinite connection we've got a load balancer and we've got various servers what we don't have well actually it was asked a question that I remember so it alone is an issue was that the absolute minimum number of nodes that you need for redundancy anyone should something action that's a good number but I would say that the number is to and the reason for that is
that redundancy it is the ability for things to be admitted without loss of function at least in this OK so we got to is a minimum and that's not always what we're going to want to do I in this
case I get here and yes I think because the most that's it I guess we'll go back to the experience by change model and so let's make the 1st step in trying to understand the situation that we've got what have we got all understand there's also a direct so we've got a we've got in a small network here but if we start filling in the details starts look really horrifying for example this rather cyber you know cyborg right it's the little embedded Linux box it's basically like like a links is running tomato without much of the functionality I it kind of works but it's really not designed for that kind of environment you know you might use it to off your home office or even a real this as long as you had the 2nd 1 or plan to upgrade 2nd thing here is that this is not a data this is not replication this is the beauty which is for anyone who doesn't know basically have for Linux and this guarantees it corruption you're not you not replicating the database you're not making a copy of things what you're doing is you're ensuring that you're failover process will give you a broken database of you know it's like a file system if if it goes down very comes up 30 and that when it's things are broken locks don't get copied and probably blocks of get copies of the things that work to the table that is being written to most frequently and that's the 1 of the most important because that's what you run into most frequently and and is just not so the next thing years we had the applications and so you might think 0 well somebody setting this up would put I would make cyber God a proper reverse proxy make a load balancer well now instead what we had was and different applications on different web servers and just in case design thinker went down we added design thinker hosting ability to the back SQL Server also my future 1 as no I think that would be embarrassing stand up you the so to be fair this stuff was set up by a developer who didn't really have a background with this kind of of thing or don't take pictures of it that's just silly but not at these things up in a way that that know I did make sense to minute it with a small company in use the he was the most senior technical person and so it works and it was good enough and and people were happy and then I started working there and people became less happy both go back to the temple again this up so the next step in the change model is an list so remember that were still on the planning phase the left hand side is planned management level right hand side of implement across the company and less is basically pulling in people who are going to help actual planning process in a really small company there doesn't it doesn't make a lot of sense to have a huge change team you don't need to enlist a lot of people you're the only 1 doing anything anyway all you need basic because they get get your boss to approve things and you know in that you know a larger company you may be lucky you may actually have responsibility and and the the ability to to take action on doesn't always happen happen the same people on yeah last nite over beer I was hearing people talk about sneaking previously into their environments and I think a lot of us have done that as well and you really want to avoid being sneaky if you can because well it's great to get previous in your work environment the last thing that you want is for the people who make decisions to associate Free BSD was something that undermines the authority that's the best way to make sure that you want continue to be able to run free BSD and that that environment will run after they kick you out I so OK after unless we get envisaged develop a vision and strategy so in this change model and if you've already convinced higher up folks that things need to happen then then you're probably the only 1 coming up with a with a vision for how to make those changes anyway but and of course you know if you're small enough team the solutions are obvious whether or not the correct so when I started at experience point this was something that I put up on the corner of the whiteboard within the 1st couple of weeks of starting their that look familiar tracking 1 yes well industry were there so I don't know everything that was wrong with the environment there but I did know that I needed something that would start turning people towards the idea of change that interestingly I don't know anything about change model I had no idea what this company that I was there working for actually did I say you know the lots of stuff as far but but at at least you know I knew that a server environment needed redundancy in we had none and this was something that I could use to leverage change that would make them more dependent on me which is always good for job security but we need to get a process started and so we needed target here's the target and it's a simple drawing and it was supplemented with a lot of hand waving and excited conversations and got people enthusiastic and and I left the drawing on the corner of the were probably for 3 months and there were there were always people going by single what is that that's interesting sales people and you know support people would would just not understand this new these are things that the united and and so you know there was lots of opportunity to sell the concept of moving toward something that would be better stoked remember the number of machines needed for redundancy I 6 3 4 web server because I want to make sure that even if we had to take a server out for uh for upgrades or surgery or whatever that we would still have redundancy so web servers of the things that you change most frequently you know a router or if you're doing a BSc router or a load balancer or a database server packages don't change all that much but when you're an active software development company and in your writing PHP which you know changes every 2 weeks anyone keep things up to date and you want to add this new packages new of you HP extensions that that's something where where there's there's greater opportunity for breakage so are so yeah everything else tended to be just that just a single spare for the load balancer OK so you could you could spread traffic over to load balancers on the front and have both of them picking up traffic thing but it was easier in
this environment just to say OK we use CART will have a load balancer will have a spare load balancer and put on the front and so so the the the Land actually there's just once which we don't have separate physical that we will I didn't want to push for separate physical switch infrastructure and setting up a less PF on every host was going to be more than I wanted to deal with a and absolutely yeah yeah you want you want your goal to the achievable so you know this yeah so so you have 1 if 1 hard drive fails you know it may be an outlier in NTBs calculation but we've got got there's idea at the time I looked at the idea of having an affair servers to to share back and but it's turned out not really to be necessary when we do a deployed we are sink a copy of of a of our software and changes in link and we can copy all the software and then we can change the symlinks all so the web server they're acting as their own files stored in the database so that the compromise here were yet to achieve quicker wins and so that things would would remain doable I back change model so once were out of the plan and we've got a basic idea memoryless planning is management so that's high level it's none of the specifics were not coming up with you know we're going to upgrade the server the were going to take this out on that date and replace the harbor with this specification because that's not what management cares about they want the pretty pictures in the corner of a white word but they they want the pretty pictures they want to be able to make a high-level decision that you will implement you'll do all the heavy lifting on so before you actually get the heavy lifting which is act that 6 we have today In the change model which is of course designed for larger companies lighting a fire is kind of important you gotta get people on board with with the changes that are about to happen if management has decided that such as going to happen you still got the rest of the employers in a large company that that need to participate in this process or are going to have things worse down the throats of the board it becomes a little bit easier of lighting a fire is terminology that I was never really incredibly comfortable comfortable with because are close at here 1 in Toronto that well despite fires you of the fire here 1 that give it a around this that's amazing so they did a good job despite the fact but OK in the chains model the motivator the motivate phase I yeah really really doesn't apply so much because sharing concerns locate you doing that with this with a small group of set ambitious stretch targets another stretch targets 3 people so so a stress target as well as the target you don't plan to reach it's something that on is out there is a sort of a dream that year coming up with that that is designed not to work towards but to make you think about other ways of achieving problem or over other ways of achieving solutions that achieve from of if you do that anyway so so for example in our case we we recognize that that by using a server cluster we would need sessions sticking and so you know we had but if you the ways that we could do without the obvious ones would be just turn on step sessions stickiness on the load balancer which is actually what we did but in trying to come up with alternate solutions we actually learned a little bit more about what capabilities were like for example the fact that we could potentially use cache servers to share PHP session information between the the the the web servers that's something we might do in the future it wasn't in scope for this but it's a it's a new idea and history gives us the possibility for new change projects that will come after this 1 so still valid valuable thing to do identifying a crisis this is kind where the fire comes in on a crisis story so I've been working there for 3 months 1 of the 1st things I had done because the software developer PHP suffer developer who was the guy before on whenever we whenever they would deploy software he would basically check stuff I do whatever was necessary with the with the Symphony framework to make up a directory that he would then FTP over to the server and then change the permissions on the appropriate directories manually for logs and cash and stuff like that and there was like a 4 hour process to deploy a new piece of software that was just the standard way of doing things so 1 of the things that I did very early on was write a script for that and and and if you will are saying I mean who so unfortunately unfortunately more yeah that that was a bit more to it I mean since symphony the framework does need a little bit of stuff to be done instead of doing it manually you can have the framework itself take care of things like permissions and stuff like that what I but but if you make an error in your script and then you go on Christmas vacation and give somebody else the ability to run the script that can be a recipe for disaster and it was because while I was in the plane over the Atlantic Ocean I um somebody type something that caused the script to break in such a way that it did much more dash capital R www data slash so that was that was fun nothing stopped working directly except it we can necessary chance because the keys were changed on Apache was still running all processes that were running as root still have access to everything they needed but we can assess each in legal keys were were changed had changed ownership on and and we can deploy software luckily we didn't need to deploy software everyone indications that it's hands-off which gonna let all come back and deal with and and that's what happened to weeks later I came back to reproduce and actually had a really office CD because I couldn't be Britain single-user methods that some pattern file was known by read and their devices had changed this has all the volunteers and it it was fun but what it really did was it identified the need for a backup server it identified really in in in a very scary but not critical way not like financially damaging way that this is this is an area that really needs attention and so so it is really did a good job of of of helping the chaining forward so I 1 of the other things that that got put on the white board was a list
of applications running on various servers and so this but the I think the the other model or the diagram was up to the right of this but so different servers were listed here and and and
and the applications of them but the most important thing here was that it started
identifying some risks and I remember where the motivate state mean the line of fire and so you identify the risks and then get people scared and they if there already scared of so this was by far not a comprehensive list it was this is an introduction the White where the guy raised a little while later after the Scot committed to fill in some of the documents but yeah cold fusion that server is still running actually yeah so so the more comprehensive version of this list I was implemented as a risk register so this is a project management term that sometimes it's called an operational risk loss and so so
I started this with with the following fields and basically you come up with a list of things that could go wrong and you come up with ways that you can fix them up and and at a very high level it's it's survey on and prioritized to-do list and so once you add a few more fields then it starts getting prior some priority you and probability and impact and you can you pick a 1 2 3 or 1 to 10 for each of those and then priority is simply the product of the of the other 2 of the 2 fields so you know what order to put things in because you know that something that is high probability but low impact from really is isn't that much of much of concern something has a something is not going to happen 0 probability then it doesn't need to be a priority at all and then 188 turns it into a bit more like a planning tool so and yet probability is always against impact is always against but you the expert so your guess is probably pretty good and you wanna have conversations with people as well because because you want find out how what that what the impact is going to have a even people in the business of like if x were to disappear at 9 PM on a Friday before a long weekend With this really be a problem it would happen at 9 AM on Tuesday would it really be a problem of and then this gets
implemented in well for us Google Docs we used were big google . cn that experienced point that makes up it's really easy to share so a rather than sending around physical spreadsheets we we really like the idea of shared on the and some of them were kind of interesting it's still hard right so here those changes the same your configuration is not not stored in and so now the stock well the starting price yes sometimes story on that to
model so speaking of sharing documents were kind of document communication I and the idea here is that were envisaged so we envisage phage phase is is where were trying to figure out more of what we actually want to do and we're trying to I trying to trying to communicate this no and as it were trying to keep and the stuff that was envisaged is what we're communicating the input that we ask for it is is what turns into actions in the next step we want to involve people we want to the so we wanted to do as much as we can to make everybody in owner of the change that's going to happen but and you know a lot of model on a small company like it's it's going mashed together and then if you have open minded people around 2 turns out not to be much of a challenge to to get things going but then so we move on the on the last bit of if you will if you make all of your priorities and projects in progress visible that's going to standard reduce a lot of resistance the act phase this is what I was saying before you have come up with a plan for this specific plan what you're going to upgrade what you're going to what you're going be replacing we come up with a high-level overview it but not the specifics of so in in our
in our transition I couldn't just cannibalize existing servers right because we had to deliver a product well stuff was going on I couldn't take down our existing model I should mention that the 2 web servers were whatever the model numbers but like Dell's 17 something or other with no idea you know and then management of the SQL servers were a model of that had of these for measuring controller that if I queried would crash the server so and yes that all that hardware basically needed to be replaced so I gave our good friends and Iike systems a call and order the status servers I'm reordering into and at that yeah they don't me price so I so young basically had to had to replace this these huge amount and things with very small lightweight and expertly built and wonderfully supported machines OK so we use that spreads widely used base camp for a lot of project management and so to do lists are easily communicated using bad as well you know anything that you can use to make it easier to communicate is gonna is gonna streamlines stuff on and based on this most flexible tool it's succinyl waterways but 1 thing is really good is that any time I put anything into it the next morning my baskets and e-mail lists what got so the more I can the more a document in 1 liners you know I don't have to be exhaustive because the pool doesn't let me be exhaustive but more a document one-liners the better understanding of my boss has of what changes are actually happening and what progress there is on matching project so so yeah you really want be bragging whenever you're implementing new systems if it is if you're not communicating the not doing anything so let's see where are we the the change that we
implement was sort of this were not 100 % through the change the SQL servers aren't yet migrated out of volunteers and so we still have that broken during the between them but but but I'm not relying on the 2nd SQL Server I'm doing and SQL dump every 4 hours and backing up on a different shape so it it dies it dies and is it doesn't come up clean OK so we restore backup I The to load balancers they are using don't use you copy completely different beast do do we know the difference between copy you character itself so cocky shares and our address of the cop addressed between 2 next you can't ties and IP address to 1 way or another so cock relies on the servers am informing the switch the year units which of the new location of of a MAC address you copy relies following the the MAC address table to update the IP address onto a new Mac so In 1 case by you're in 1 case you're relying on the switch on the in the other case you're relying on everybody's host everybody's a MAC address table everybody's ARC-cache and you yes yeah and it may be sort of update and you can't control it all in 1 place so it's better to use CART so I so we've got we've got the 2 load balancers the each of them is nation the reverse proxy and the running pound the and source-independent requests to the web servers and like this is quite complete which is the database server is also need to get updates from time to time because we're not posting a package locally so there's a 2nd network here that let them get to you proxies as well it's easy just tiny property rights and access you know you you the very heavy stuff just gets systems talking to the outside world on and then of course because the web servers are jails on Apr servers we can run multiple ones so were working on upgrading our own software to Symphony and and the 1 which is obsolete metadata and should be tossed out simply to come running PHP for PC 5 4 simply 1 can't because it's using things that were dropped in 5 and 55 so so we have a new set of web servers and and any time we have something that is appreciably different from are existing hosting infrastructure if we need to run something that used to model Perl CGI eyes I set up a new web service for that because I don't want to touch things that are running our production infrastructure it's easy to manage more servers right it more virtual servers it's it's more work but once you pass a certain point it have automated thing automate things anyway and so might as well have more and then and then reduce the risk of of of breaking things as you as you make changes so what back to the change model last
phases consolidation 0 balloon model where we rephrase this is kind of the rephrase this is this is where we we use our credibility to suggest changes to new system remember remember the stuff which were coming up with before the the stretch targets well OK so maybe some of them will become the new changes that were needed or we've learned other things that you need to do word stuff that we didn't want the scope to creep into now it's nuggets addressed but we've now set up systems that are going to be stable and I'm going to change and if you're in an environment that has more people you know bring them in so that so that you can be in the a little less stressed when you go on vacation the young you know that I
have friends who are capable Linux administrators I mean OK friends all of the other Switzerland it's suddenly even make it works fairly well but you know every 1 of the problems that we're dealing with an experience point could have been solved using Linux as well the I'm well except for maybe they're using the purposes you can't think is that I don't think purpose implemented the shaking your head but it's not there are actually different 1 of the 1 of them is moving the MAC address around and the other 1 is simply binding the I P 2 different men the kind of you but may be any to more reading something OK that all of the some of the summary reading of that was that was my understanding and well that's true yeah yeah but on the other hand so OK that that's anything that you want to and previously you could you could you could do and so the idea here is that you're implementing changes and then using your expertise in something that you like better united States is when it comes down to a religious or preference decision on all the other stuff that we're doing here is handled by 3rd parties operates the same software that you're running whether it's PHP or post press or my sequel or or Apache or and genetics all of them are basically the same sort the running in whatever whatever 0 that really it then city express your preference well enough others mysterious and I said that
about wraps things up for me I anyone have other common suggestions quote and for I added a picture of my cat and
yeah operate with variances anyway you can that we for now In in
my environment I was pretty lucky in that in that the guy who was and allowing me to make changes the previous senior technical person was this offer developer and he said I freely admit that I don't know this stuff and so if you think is better go ahead and and so yeah I I didn't have to struggle with that so much interestingly all lot of the stuff that we should take for granted maps onto the change model which is kind of interesting but but you know that the that the stuff that we naturally do sometimes is is a is a little bit of against what we should like you know the idea of doing things and surreptitiously if you install previously everywhere you know and boss doesn't want to you can you be staggering yourself in the you but really what was on yeah well that was 1 of the cases you you can yes good all the for the plain dumb ideas is a trick mean we developed CBC is well we we would set up something that was using previously and then we rely on sale would you know what that had set up a while ago but we can't really change it now we needed a big project to do that but you know we can look at the future but it's not really a high priority right now let's just leave it as it what is the change in the price 1 what it was the all of relevant to you yeah alone on each of the components here got implemented as its own projected onto the subproject so everything has to get divided into you know chewable chunks if if a project is to be then it's is never going to be finished so you say OK well that's not the project but this is the project and then that's the next project so replacing the database servers the next project replacing the replacing the web servers was approach was like replacing 1 of the web servers was the very 1st project where we did a proof of concept made sure that this new stuff that nobody had used before I would actually work with our software and then once that was in place you know that the order was basically 1 web server 1 load balancer a 2nd web servers of 3rd Web server a 2nd load balancer and now or up to the network to the database all the all you well that would have been another rates so you know if you identify risk if you say no you know what if you're decision their kind of really caused us to do something nasty could approach and we were at work in a worse situation than we were started we either need to back out or we need to go forward but right here is not comfortable and you know maybe it turns into a new project but at least you've got more incentive for change at that point like so all of yeah train your coworkers but I is the thing is Linux people are available very easily Free BSD people are still not available pretty easily if you advertise for them because a lot of people answering Linux and the previously people the better ones are to if you know you get a much better in Linux sysadmin if he is comfortable with previously well even on trivial this articles maybe even the I'm not sure about that all so that is what I was the what the you really yeah yeah and if if you need to come up with a with benchmarks of performance is an issue you can well yes no licensing tends to be less of an issue because a company is probably just using commodity right they don't really care what the licensing is if if it's downloadable install and they don't have to pay for it that really matter on that but the I will be able by support if you're on vacation what happens you have loss factor that's that's vitally important and but if you're the only person in the company whether or not it's previous during the next you the bus factor 1 this is just of what we would this is what so you what are you use this was yeah the right to the right tool for the job is a great way of of getting the history and things you know if if both of them will work them and and and you have a greater comfort level yeah they have to pick 1 and you're going to be managing and then they should probably pick the 1 that you're picking because you're going to be managing but it is well I don't know if you if they don't like your can it you well that that's the company that you want change that but we great what do you think you're he's remember slide into that was that long list of operating systems I'm also Solaris guy I'm in a I'm an HP-UX die if you really stretch I can run the same you know sure how bad I mean it right when it comes down to it what you have is is a wealth of knowledge on a number of different operating systems and of the ones that you've been dealing with you know Linux has risen to the top because of necessity because a lot of people are running it and then previously has also risen to the top because of Excellence or net BST or open I but you know was once you have your preference the idea is you want to be able to to explain why it why it's better and sometimes better simply because you can do things with it more quickly than you can with what the church the z effective it of yes that's a good question so that's thank you


  846 ms - page object


AV-Portal 3.11.0 (be3ed8ed057d0e90118571ff94e9ca84ad5a2265)