Adding Developers to the DevOps Process

I think everyone a deadly and anything they Anthony I'm going to discuss how beloved that my company where route and know working which have been kind of process that we went through look

so the money and the native often I welcome my great city I wish I whether with lower like this to cure this week where they're going from I've been a full-stack web developer for about 7 years of work mostly with regard on back end and there and maybe angular on a friend but so I worked for a

company where root for about 2 years we started out 10 years ago at the consulting firm and a little before I in the company but the side transition the company to a bill that stop software and service platform for automotive in retail customers of and then the our software does is it combines getting in fights in a platform that a lot of are customers to take or negative down into execution and made their made all the stores for Formica best store to the

the tha

the fell our before I begin talking about the process that we went through incorporating by developers interacting cathode and then what about 1 a thing you a little bit about our decision making process so why do it this tha

I so initially and meaning of our company we had 1 local software architect and all of development was done by off contract developers that we start out with a very basic model of ground application and architecture with simple enough that our software architect could handle all of the infrastructure requirements mainly so I and poured

lover destroying cited that many new way to care to here and configuration files that were like our production environment so that developers can work on top were locally and so our software architects started here monetary to hear the file that commonly in the pattern of using at that point but because of of work in Ruby and Kevin is they can reveal it's kind of a lot from including you shaft the

today at very we have a growing given developers that we move toward this that platform we have a local engineering team of 20 people including 9 full spectrum of unlawful when team and for people on an idea and thinking of and with the growing number of developers we also have of opting out but not the only consists of 3 people and 1 who is specifically hired because of it just knowledge and so our

software architectures diagram last year taken show how our architecture that thing you the we have some friend clients that now and we have a lot of services and of API is built this thread the main knowledge of right and as well as a basic infrastructure foundation that all aware of the need and that at the bottom we have a data engine pipeline we get our data flowing in your right and up to him and the knowledge we have this

a complex architecture now but we also have development stating in production environments may cure any code is go through the winter production the so

this time are also you at the moment there constantly fighting by a day in day out and this is the the the the will it deftly affect their work happiness and so they can what to do I in how 1 developers and help sharing the workload in I cannot them that they'll appetite by book I'm tha the of this size and it's over and and so and also the 1 to make imperfection morally because they knew they could have complete coverage of all infrastructure and so they wanted stating shaft as more every 4 4 main grant like it's typical you the and

also we think it is to the player code and data changes and sometimes you have developers weighted how the code QA ready go to play that something will go wrong imaging and build and sell and because a developer the nearly at the ownership of the code after it loses full a we have the broken below that would people out of fixed and also because of the well ones working at Camp repository evident hard for them to but have all the application context that they needed to make the fervor catered for infrastructure with our of working and this is the leverage

you they're constantly waiting for our the velocity in the late changes to the shaft code so that they can release new feature the evenness but think there's any can affect little economy reflected poorly on the developer about in the article be an extra only at internally other departments with be confused about wiping the taking so long to remove all at the production and then also our customers with the promise but that were kind of a long time coming and so our developers were kind of data learning shaft to the they can help all about being maybe he did then and move the software process along path also of some

developers affecting me whenever there are at Beirut wonder you know all like Devon arts current divided by a wall widely developers didn't know what 1 not it's kind of laid out all view of a wall separating and developedin QA a you know that a lot of lovers before where I will work on my machine so let the code to QA and they can take it from there but now developers have a basic understanding that they need to know about the base the testing and it can help improve their code quality and so in the same way the beloved leader understand I have enough imperfection general so that they can that we have working production and can build better off with so the mother not were had an agreement that developers that moving in the direction of writing can code and helping all out but we have made a rule of law and so I of engineering he used to be offered in the pen you that interview with Hendrick Denmark development not working the other the user OK with this and he also won the nature everyone working underneath him but the engineered in Austin were happy and he new wave of all these fired that popular constantly fighting that they're right unhappy and so he knew that relate to have like not to work with and make them year and also on work on the value there something that we take very seriously and that they helped I registered into a move towards them up for 1 thing we let me be and the bigger which then we have a culture of experimentation so the developers are willing to give you a working in Africa and you know it didn't work out if you work out but we had a good feeling about it but we also know that the playlist more in something but that happens and we very customer focus and 1 here we're doing the best value for customers and the other 1 help each other learn the process and now all employees at square root receiver anal training by the and some developers knew that there anyone you this training but it to learn more about that the

and so we all converged on the idea of integrating that multiple during training came and developers when that you kind of have a part of points so my

team was standing up a new API and we knew that off having created new infrastructure and stand up would take long time and 1 we'll move faster so we had kind of been mulling the idea of learning cap anyway so we decided how about but we have the lovers so in fact I and with the help of often can't stand up our own imperfecta at the very so although had very access there had read access to the camp upon operated at the retrieve the configuration file that we needed but now I granted right access so we could start contributing to half of not seem within a few hours with my team going through a deep dive of high-density graph and a idea now this involves giving on the lovers data have been filled with a little nerve-racking because you know and and in exchange developer could actually do a wrong move and have low production of luckily we had handed back that we had backups and that allowed but that allowed everyone to feel comfortable working in AWS and we also have a culture to aware this experimentation and possibility of affair with allowed nonlinear being all

all code changes in the care about Korea had to go through a poor requesting code review process but not the and so this initially slow down or the because they have all the code that they need to review and this is just another job GABA falsely of there's on but in addition to you the beloved learning more about half from not the in this way not seen actually learn more about how to write maintainable code From the lovers and also the developer not seen together kind of group is coded infrastructure our infrastructure at code mentality I love you and as far as more code review the guy developers felt more comfortable helping other developers that they may therefore requests and review the code to lighten the op work so initially

because of the way we have been using the Pfeffer potpourri we did not have any testing it of before a code merged in end of with more developer working a code and more chances of being going wrong so we had to go through a substantial have upgraded and then I have taken so that we give that at least have a little bit of manual testing for the mean are so a long

process is kind of being seen of my team can gage theory the learning more about you have uh especially how you testing that we had it and as we as we're writing chapter we're talking about a month and day or innovation more became interested in learning chat and of you know we said the inventors of and we started hearing others and so they were that getting interested in how they can work fast and their their own team workflow and up the loss away and gave the feedback taken to the various and but this is all very time-intensive still up in and so not thinking

up with the idea of a deadlock field a cast on across heme group that would come together regularly and hand and go over best practices and how they apply for can and that way all the genes to be on the same page regarding the but the thing quite work out the way wanted you I at it I think it would be at the idea came out to early before developers really under the that were arrested kind working at square root that they would be working with them up and there's been a workflow and so they were really interested in adding other meanings to the federal and adding this additional staff at a job so it does allow but we still had more knowledge sharing going on the lover the readout of theme of for a pair while in meeting the top and of the with e-mails the kind of explained further topics but they soon realize that you the instant thing questions over and over again to individual developers and because the velvety not only 1 to look at the the impact on United command and practices with the on understand what the why behind it and so so we have a company we the that kind of a conglomerate of all of our documentation and so the not decided to add a shaft documentation to become the wiki that developer newly refer back to it

the beloved then that also reading external the company to learn best practices such as learning can look and also in person can training classes eventually you reach a point where developer for comfortable not working with fact that they even went in the added and tech and ended up on the wiki the documentation that not nitrated because they wanted to develop a theory that full with working in a shaft and 1 here what they learned

so the process has a few more bonds in there are then I think any of us had relied from the 1st started out wanting care to them with you 1st law just that have a pretty the learning curve and knowing that they you know how in the beginning already covered the release while those in the case now we have hired developers based on their pipe on knowledge dinner and hearing as well as their job as regards work for and development and so because these developers know Ruby they additionally had to learn Ruby before they can actively work in effect but the also testing led to log developer perforation because we didn't know best practice did fact when we 1st started out we end up with a few very mild a cookbook and the running them manually to test taken to make sure that cocaine is the break anything took upwards of an hour each time and this led to and beloved being idle and of waiting time and it was very frustrating so developers and that him upon themselves to break apart the mother books in this slide deck and that can be more easily tested and we end up with a cookbooks that only care about 20 minutes or so the manually run through the it also

lies at the same time the code 1 example of while I'm when I have to do an early poor but to add a template to 1 of the cookbook I look back to the code and I saw that these cookbook were using 100 of role at the end is that at any pattern cast it's actually not a good idea use or library for various reasons but I you I got to the good pattern follow so I Rome I completing the are libraries and Bentley during the art scene a review of my code they this and made and here with me a better practice and I can not about the inability of the code and of kind best practices in the code for future developers that I fame that code and use the values of the other kind of Monterrey

here it was that documentation quickly grew out of date and developed varied from the size of the documentation they but they information because useless and so they ended up going up scheme a lot more often to ask the same question over and over again and so we learn the importance of maintaining documentation so that the Austrian when you have to go through and review themselves over and over another can be faced with

actually cultivating the right culture forget about that we have the additional complexity of of the having offer developers and and didn't kinds of differences there is really a limited amount time that we can do not sharing with them also they had come in different values in the in that apart from the values so that means having them learn have so kind of talent and also we had developers who accept reluctant to learn that they buy in 0 where application developers really do code you may feel all the wall as the with the wall between developed written off as firm but as we continue on all and more developers guide to writing staff and we can obtain delighted you more and more and how this is helping us be better developers and have better workload even the most reluctant developers can gradient and the burning

that's not the Japs 1 of our developer the few weeks ago and I'm I remember this quote about the in thing he said and lake in the working would have if it's not part remain job it's harder value and retain the knowledge and found is interesting because it gets and by far the had is the developer that by far large and contribute the most to our have code and she felt this way about if you haven't them off theory can be here that almost all the other developers felt the same way and at at square root we found out of the way it happened thrown if that have better documentation that's easier virtue of that developments in constantly refer back to the working with him them all so this is kind of

time commits a very careful bonds worry we say in this process and the latter half of 2015 and it took about 6 months the most the lovers get random and shaft and all its been about 18 months through the process of if taken as a little longer than we want to get all developer working in the but the wire

curve worried we have he he had contributed and that she made all the Lovers that have worked on square code and so we knew that undergirds us the 13 I did ending department

they used to have hundreds of configuration files that they would mainly copied to every new theory that they had been up and through have and the learning task and using kaput intended purpose of it have remained and configuration they're able to automate this process and so on and so that new servers for the good of you very quickly on IP with less aircraft all our new services environment Mr. Khadr read even monthly monthly putaminal configuration than tweets to stand up and it now with everything they to have at we can automatically set up a new environment with several theories in the matter of minutes and now and then and

then and all about steam are working together we have great shared application context and infer from the context so that you know where you might have a working together we opined dealing with Jenkins Bill when they break because we all have at their context and those kinds of issues get it fixed much us now this is still a work in progress of course that they were you know it had better documentation to help the lovers but work with them up and on that we thought it was the Antipatent from the whole code that we still haven't been able to change if you are asking this turning the other maintain status quo in the care about where it's looking forward we hope

that it injury instead and to our intuitive monitoring Herrschaft code so that we make you were compliant actually have a handle on the test we can make sure that can that we've you don't break things and this will help improve happiness because of that said well maintained voice is testing the actual time also we hope to integrate doctor on anything can and help portability of our services then at servers no also laughter viewing the you know the morning I'm really excited about habitat into our rather it into effect workflow and so that are applicator that we don't deal somewhat with application dependency that maintaining those so all all we view this as

a great success and I a on you year maybe year on 1st and then you get out my workload is quickly becoming unbearable and I want to work with developers I'm kind here's some people away from profit there we can take that during the so 1st of all

stop before you need to I We patterns process very pretty simple on the word but as I said earlier it took a belonging to the and I did R. Artstein actually had a workload increased dramatically before it got better and so you need to start the process before a often really have their hair on fire also take care with morning you're adding an additional piece to the workflow today in the workflow of developers and the infinite take care and make it easy for them to have in that he's a whether documentation on this the act that all being and to get questions answered although all integrating developedin the demo processes has led to greatly shared context and have greater ownership of both developer that all through the software and this is lead to better software being built and avoid and on allocating any question that time writing commas trade school for any number of a few of the question please raise your hand opening the microphone in the speaker question the microphone so the reviewers later in the benefit of your question a so you said rated story about

transforming developers will I get a tough job to transform of find developers reward into the shift around so 1 how long did you Johnny take uh I don't know how many developers total 59 whatever number is it 2 weeks to months so how long did it take I'm so for the base the chapters getting right then it affect the Otto a couple weeks but I think is kind of ppm profit where my team an integrated in chest and then other teams after the follow the result of my being kind of went along with that and so each time a different team guided have go would be up the matter find the process of onboarding them and so on light with the latest developer who through and I think the month ago in a her it only took a a few hours before he submitted the 1st attempt here after running about the interaction so it can vary with it you can learn and you go 1st herself great talk on this 2 part question 1st of when when the saw your own developer start to do things you start asking and do things will kind cookbooks were they generate for you with the infrastructure ones are more at deployment and 2nd of how did you democratize still work processes of like environment Pennine and and and cookbook creation and promotion of the ships the American so 1st of all of we were going we're doing a lot more apt the the environment so but the data engineering team there's any of you process process server and Kafka clusters that and have that sort of architecture and so it can vote I have about half of another bump in the road because we ran a huge where developers would forget that they needed to upgrade their a cookbook written academia changes and so the the server but it was there but we randomly choose and so not seem kind of have to formalize processes after after we were and what the work and then we added that to the documentation and so that we can have like highlight them bookmarked by nature before you submitted for class ID do that may care before you run chef-client if the reading this that help less but government company we have all a large development groups good and JavaScript's Avery would doing things like that but they were also very fast paced and the concern about halves or when you're along Woodruff time was the biggest landmine you run into a set sort of mine so how do you passed well elements that can of reminder of life how long it took the arts in the main changes that are necessary for our application code to be deployed and how long the developer the kind waiting for the changes that happen as i reminding them that you know with this time that they're waiting for a the chapter came they connected the learning task at Duquesne themselves so they can move faster and they also found you know i developer the smart they've learned they be the for half it's kind of a domain-specific language I'm and so they understood the process of it and then after that it would make the case the changes after a long run a decent man bus so I'm also trying to so the process just the very large companies hearing we don't time and that like the benefit until they learned this learn something the goal to his interest before but the curious if you noticed how did the adoption did easier and quicker had more of a foundation of cookbooks for people use as reference source for reuse and once this privilege reuse in solar power to that help but to bowl interest yet definitely hold on the with our documentation the man . him with this be going through both the 0 this is a great example of this this factor cap and so they would copy the code snippet in her documentation so that when the kid Lovett came along and I I wanna how did this in that they can refer back to Diamond and Abilene to you know cookbook or a template that is especially well and so it helps developers ramp up and the

up quick question on from the training in readiness perspective did you get colds and developers to training classes which shelf or anybody in to new they can work no we actually relied on the 1 all that really knew other very well and was hired because that is the expenditure knowledge of it would only you know after after we had involved in all of his knowledge and then and development 1 no more than we reach the external resources to however can choose the prévue time for 1 inch request questions isn't there the and did you disappear shift team as kind of a sanitization to how did you of a disappears specific as experts and shift to deal the chain others or expert with is main role that 1 main role that he was the 1 that we turn to most often where question a actually kind of grew organically linked I talk about the idea of a that skilled as time across the in-group and ideally it would have been you know the person who work mostly with that of the king would come together and share best practices and talk about some of but I have not read the of possible because we didn't really point a person to the prime kind of grew organically ad a more development work in the code light I became the go to Jefferson on my team and combines really enjoyed working here that I work with infrastructure that made before as a middle and so I would so I was comfortable working around the Earth and other and so as I read that the lucky that it grew organically in the major find 1 person in each scheme that it can be the point person for the rest the team that that quantum theory thank you for the talk of how did you actually make the value proposition to the developers you mentioned that majority of them are remote and did they send them the pizza vitamin drinks or anything to to help convince them hello world we also show that a few examples of times when you're software had worked correctly on their local machines also development feeding but somehow went wrong in production like something that were currently in production and was because the optical and maybe changes that have occurred context and so on and there's something going wrong production we can use that example of you know if you had more contents into what was actually happening with infrastructure and how little that up this up 1 be having the code you have to be going back through and making quick but isn't that so that kind of value proposition that we it without hurting so if you have a value you more not theory I did not a more customer value but the value and putting more correct code out there pushing work those constantly and so when they had to slow down to the bug fixes and the infrastructure of off then there are some that have heard about 1 spss allows question on great talk by the way and I just cannot chair my experience this kind of similar to we think they think it's on something there is not a question I know when we started chapter what really help the developers get on board is just having people talk about chapter the passion about of was the you know is something really cool is really seeing efforts we actually Disney have around supermarket that the developers can contribute to you and I just call a fire so if you can get the people passionate people about it and get it going in evangelize at that really spreads throughout the whole organization and certain a small group in this new now it's been you know spreading like wildfire throughout companies so I don't know that's kind did you have connects him experience with square root yeah definitely out there that we had a couple of very reluctant lover of going to a but add and development were in fact the the 11 beloved has followed benefit them out quickly OK they're moving you working on that and they fall even if a benefits that can help their job at application developers and so I had to do with how event lighting and instead of very far more developers figure it perfectly 1 more a round of applause for the use it