Adding Developers to the DevOps Process
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 45 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/34567 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
ChefConf 201741 / 45
1
4
5
9
12
16
18
20
21
22
24
25
27
28
35
37
39
42
00:00
Software developerProcess (computing)RootArchitectureProgrammable read-only memoryBuildingThermodynamischer ProzessRight angleCartesian coordinate systemComputer filePhysical lawNatural number1 (number)Information technology consultingSoftwareThread (computing)Local ringProduct (business)Rule of inferenceComputer architectureArithmetic meanCodeDesign by contractComputing platformVelocityNP-hardWaveService (economics)Point (geometry)Context awarenessGroup actionGreatest elementPattern languageFront and back endsWeb-DesignerSoftware testingMultiplication signMathematicsData storage deviceEndliche ModelltheorieNumberView (database)Integrated development environmentInformation engineeringConfiguration spaceDirection (geometry)Moment (mathematics)WorkloadDecision theoryDiagramClient (computing)BitShared memoryInternationalization and localizationSoftware bugDebuggerOperator (mathematics)Software as a serviceSquare numberRepository (publishing)Online helpData managementWeb 2.0Web applicationMultiplicationRuby on RailsRootJSONXMLUML
08:26
Repository (publishing)View (database)Maxima and minimaServer (computing)Instance (computer science)Host Identity ProtocolIntegrated development environmentWeb pageThermodynamischer ProzessOperator (mathematics)Web pageCodeSquare numberProcess (computing)Presentation of a groupOnline helpMultiplication signSoftware testingWave packetWikiRootBackupConfiguration spaceMathematicsPoint (geometry)AdditionGroup actionModal logicRepository (publishing)CASE <Informatik>CurveLattice (order)Self-organizationData managementWorkloadDecision theoryCore dumpEmailBitFront and back endsInformation engineeringScheduling (computing)Spherical capComputer fileArithmetic meanSocial classTheoryCasting (performing arts)Field (computer science)Focus (optics)Graph (mathematics)WritingStaff (military)Nonlinear systemProduct (business)FeedbackPhysical lawInsertion lossMereologySound effectDeadlockComputer animation
16:47
Data managementComputer fileWeb pageSquare numberMereologyProcess (computing)NP-hardComputer networkPlastikkartePunched cardServer (computing)RootCodePulse (signal processing)Graph (mathematics)Key (cryptography)Operator (mathematics)Service (economics)Error messageMathematicsSquare numberType theoryThermodynamischer ProzessServer (computing)CodeInformation engineeringRight angleRepository (publishing)Context awarenessPortable communications deviceSoftware testingInformationINTEGRALSoftware maintenanceMultiplication signRootConfiguration spaceWorkloadCartesian coordinate systemTemplate (C++)Online helpSheaf (mathematics)Integrated development environmentLimit (category theory)Arithmetic progressionComplex (psychology)Control flowComputer fileShared memoryAttribute grammarSuite (music)Flow separationAutomationTheory of relativityTime zoneData managementFrustrationMereologyPoint (geometry)Different (Kate Ryan album)Self-organizationPattern languageCurveSlide ruleTwitterTheoryDemosceneGoodness of fitLibrary (computing)LoginSound effectProcess (computing)Task (computing)Numbering schemeInferenceStaff (military)XML
25:07
Operator (mathematics)SoftwareWorkloadThermodynamischer ProzessMultiplication signContext awarenessFile viewerNumberInfinityPattern languageDemo (music)XMLComputer animation
26:47
Server (computing)Operator (mathematics)Template (C++)Formal languageInformation engineeringLink (knot theory)Thermodynamischer Prozess2 (number)CodePower (physics)Integrated development environmentGene clusterMathematicsMobile appProcess (computing)Cartesian coordinate systemProblemorientierte ProgrammierspracheComputer architectureQuicksortMultiplication sign1 (number)Personal identification numberGroup actionClient (computing)Revision controlBus (computing)Goodness of fitMetropolitan area networkInteractive televisionCASE <Informatik>MereologyElement (mathematics)Source codeData miningNatural numberNumberShift operatorOnline helpVideo gameSpherical capDivisorReading (process)Task (computing)Social classResultantLecture/ConferenceMeeting/Interview
32:20
TheorySocial classMultiplication signExpert systemGroup actionPropositional formulaSoftwareContext awarenessSquare numberWave packetChainShift operatorMathematicsCodePerspective (visual)Roundness (object)WhiteboardProduct (business)Point (geometry)Software bugQuantumEvent horizonNumbering schemeContent (media)Cartesian coordinate systemVirtual machineProcess (computing)RootData managementLogical constantMereologyStandard deviationShared memorySelf-organizationOperator (mathematics)Perfect groupLine (geometry)Server (computing)Level (video gaming)Extension (kinesiology)Latent heatMeeting/Interview
37:52
JSONXML
Transcript: English(auto-generated)
00:05
Thanks, everyone. As I said, my name is Annie Tse, and today I'm going to discuss how developers at my company, Square Root, ended up working with Chef and kind of the process that we went through.
00:25
A little bit about me. I'm a native Austinite. Welcome to my great city. I wish our weather was a little more like this trick here this week, but it's okay. I've been a full-stack web developer for about seven years. I've worked mostly with Ruby on Rails on the
00:40
JavaScript and mainly Angular on the front end. I worked for the company Square Root for about two years. We started out ten years ago as a consulting firm, and a little before I joined the company, we decided to transition the company to build a software-as-a-service platform for our automotive and retail customers.
01:05
Basically, what our software does is it combines data insights in a platform that allows our customers to take corporate initiatives down the in-store execution and make all
01:20
their stores perform like their best store. Before I begin talking about the process that we went through incorporating our developers
01:42
into writing Chef code and the DevOps process, I wanted to explain to you a little bit about our decision-making process, about why we went through this. There you go.
02:00
Initially, in the beginning of our company, we had one local software architect, and all of our development was done by offshore contract developers. We started out with a very basic monolithic Rails application, and the architecture was simple enough that our software architect could handle all of the infrastructure requirements manually.
02:27
As more developers joined, we decided that there needed to be a way to share configuration files that were like our production environment so that developers could work on software
02:42
locally. Our software architect started this Chef repository to share these files. It was kind of an anti-pattern of using Chef at that point, but because all of the developers worked in Ruby, and Chef is based in Ruby, it was kind of the logical conclusion to
03:01
use Chef. Fast forward to today, at Square Root, we have a growing team of developers as we move towards the SaaS platform. We have a local engineering team of 20 people, including nine full-stack developers on multiple web teams, and four people on a data engineering
03:22
team. With this growing number of developers, we also have an ops team now, but the ops team only consists of three people, and one who was specifically hired because of his Chef knowledge. Our software architect drew this diagram last year to kind of show how our architecture
03:44
has changed. As you can see, we have some front-end clients now, and we have a lot of services and APIs in the middle to spread the main knowledge to the front-end, as well as basic infrastructure foundation that all web applications need.
04:03
And down at the bottom, we have our data engineering pipeline to get our data flowing into our front-end applications. And so not only do we have this kind of complex architecture now, but we also have development, staging, and production environments to make sure any code changes go smoothly into production.
04:23
So this is kind of our off-through at the moment. They're constantly fighting fire day in and day out, and this isn't really sustainable for them. It definitely affects their work happiness. And so they kind of want developers to help share in the workload to kind of help them so that they don't have to fight fires all
04:44
the time. Sorry, I'm still trying to get this to work.
05:02
And also, they wanted to make infrastructure more resilient because they knew they couldn't have complete coverage of all the infrastructure. And so they wanted to start using Chef as more infrastructure management, like its typical use. And also, we use Jenkins to deploy code and data changes. And sometimes, you know,
05:27
developers would have their code QA ready to go to deploy, but then something would go wrong in the Jenkins build, and it would fail. And because developers didn't really take ownership of the code after it moved to
05:41
deploy, we would have broken builds that would take a while to fix. And also, because ops team were the only ones working in our Chef repository, it was kind of hard for them to have all the application context that they needed to make the appropriate changes to our infrastructure once our software
06:01
changed. And this was the developer's view. They were constantly waiting for the ops team to make changes to the Chef code so that they could release new features or even just bug fixes. And it kind of reflected poorly on the
06:21
developers, both internal to the company and externally. Internally, other departments would be confused about why things were taking so long to move along to production. And then also, our customers would be promised bug fixes that were kind of a long time coming.
06:40
And so developers were kind of interested in learning Chef to see if they could help the ops team make these changes and move the software process along faster. Also, some developers, especially me, when I first started out at square root,
07:00
wondered, you know, like dev and ops, they're kind of divided by a wall. Like, why do developers need to know what's going on in ops? It was kind of like an old view of a wall separating developers in QA. You know, a lot of developers before were like, well, it works on my machine,
07:21
so let's throw the code to QA and they can take it from there. But now, developers have a basic understanding that they need to know the basics of testing and it can help improve their code quality. And so in the same way, developers need to understand Chef and ops and infrastructure in general so that they can know exactly how their code is
07:45
working in production and can build better software that way. So developers and ops were kind of in agreement that developers start moving in the direction of writing Chef code and helping ops out. But we had to have management approval as well.
08:03
And so our VP of engineering, he used to be an ops person. He kind of knew the industry was headed in the direction of dev ops, developers and ops working together. So he was okay with this. And he also wanted to make sure everyone working underneath him, the engineers and the ops team, were happy.
08:22
And he knew with all these fires that the ops team were constantly fighting, that they were growing unhappy. And so he knew this was a way to kind of lighten the ops team's workload and make them happier. And also, our core company values are something that we take very seriously.
08:42
And they help drive our decision to move towards dev ops. For one thing, we like to think big and do bigger, which meant we have a culture of experimentation. So the developers were willing to give working in Chef a try. And if it didn't work out, it didn't work out. But we had a good feeling about it.
09:02
We also go through playlist post-warms when something bad happens. And we're just very customer focused and want to make sure we're delivering the best value to our customers. And we also want to help each other learn in the process. And all employees at Square Root receive an annual training budget.
09:22
And some developers knew that they already wanted to use this training budget to learn more about Chef. And so we all converge on the idea of integrating a dev ops culture into our engineering team and our developers learning Chef. So here's kind of how the process went.
09:43
So my team was standing up a new API. And we knew that the ops team, having to create this new infrastructure and stand it up, would take a long time and we wanted to be able to move faster. So we had kind of been mulling the idea of learning Chef anyway.
10:00
So we decided, how about we as developers start learning Chef and with the help of the ops team, kind of stand up our own infrastructure as necessary. So all of us had read access to the Chef repository
10:20
to be able to retrieve the configuration files that we needed. But now everyone was granted write access so that we could start contributing to the Chef repository. The ops team spent a few hours with my team going through a deep dive of how to set up Chef and AWS.
10:44
Now this involved giving all the developers AWS credentials, which is a little nerve-wracking because an inexperienced developer could actually do a wrong move and kind of blow up production. Luckily, we had backups in place
11:01
that allowed everyone to feel comfortable working in AWS. And we also had the culture where this kind of experimentation and possibility of failure was allowed. Now in the beginning, all code changes in the Chef repository had to
11:25
go through a pull request and code review process. By the ops team. And so this initially slowed down the ops team, because they had all this code that they needed to review. And this was just another job responsibly of theirs. But in addition to the developers learning more about Chef
11:44
from the ops team this way, the ops team actually learned more about how to write maintainable code from the developers. And also, the developers and the ops team together kind of grew this code as infrastructure or infrastructure as code mentality.
12:06
And as more code reviews are done, developers felt more comfortable helping other developers as they made their pull request and review their code to lighten the ops team's workloads.
12:22
So initially, because of the way we had been using the Chef repository, we didn't have any testing before a code was merged in. And with more developers working the code and more chances of things going wrong, we had to go through a substantial Chef upgrade and then
12:43
added Test Kitchen so that we could at least have a little bit of manual testing for our infrastructure management. So along the process, it was kind of gaining steam. My team kept getting interested in learning more about Chef,
13:04
especially how to use testing now that we had it. And as we were writing Chef code and we were talking about it amongst the engineering organization, more teams became interested in learning Chef, and we started seeing benefits of it,
13:22
and we started sharing it to others. And so they started getting interested in how they could work Chef into their own team's workflow. And so our ops team also went and gave deep dive presentations to the various engineering teams. But this is all very time intensive still for the ops team.
13:42
And so the ops team came up with the idea of a DevOps guild, a cross-team group that would come together regularly and kind of go over best practices and how they apply to our organization. And that way, all the teams could be on the same page regarding Chef.
14:04
But this didn't quite work out the way we wanted it to. I think it was the idea came about too early before developers really understood that for the rest of their time working at Square Root, that they would be working with DevOps in their day-to-day workflow.
14:24
And so they weren't really interested in adding other meetings to their schedule and adding this additional facet to their job. So it fizzled out. But we still had more knowledge sharing going on. The developers would reach out to the ops team for kind of one-on-one meetings.
14:43
And the ops team would send out emails to kind of explain further topics. But they soon realized that they were answering the same questions over and over again to individual developers. And it was because developers, they not only wanted the syntax
15:02
and nice commands and the practices, but they want to understand the why behind it. And so we have a company wiki that's kind of a conglomerate of all of our documentation. And so the ops team decided to add Chef documentation to this company wiki so that developers
15:22
can continually refer back to it. Developers ended up also reaching external to a company to learn best practices, such as learning Chefbook and also in-person Chef training classes.
15:41
Eventually, we reached a point where developers were comfortable enough working with Chef that they even went in and added and edited the wiki documentation that the ops team had created because they wanted developers to be successful
16:00
with working with Chef and wanted to share what they learned. So the process had a few more bumps in the road than I think any of us had realized when we first started out. So I wanted to share a few of them with you. First of all, Chef does have a pretty steep learning curve.
16:22
And not only that, but you know how in the beginning all of our developers knew Ruby? Well, that wasn't the case now. We had hired developers based on their Python knowledge for data engineering as well as their JavaScript knowledge for front-end development. And so because these developers didn't know Ruby,
16:42
they additionally had to learn Ruby before they could actively work in the Chef code. Also, testing led to a lot of developer frustration. Because we didn't know best practices using Chef when we first started out, we ended up with a few
17:01
very monolithic cookbooks. And so running them manually through test kitchen to make sure that code changes didn't break anything took upwards of an hour each time. And this led to developers being idle and just wasting time, and it was very frustrating. So developers ended up taking it upon themselves
17:21
to break apart these monolithic cookbooks into smaller sections that could be more easily tested. And we ended up with cookbooks that only took about 20 minutes or so to manually run through. There's also a lot of existing anti-patterns in the code. One example is when I had to do an early pull request
17:45
to add a template to one of the cookbooks. I looked back through the code, and I saw that these cookbooks were using hundreds of role attributes. And this is an anti-pattern in Chef. It's actually not a good idea to use role attributes
18:01
for various reasons, but I thought this was a good pattern to follow. So I wrote my template using these role attributes, and thankfully, during the ops team review of my code, they caught this, and they cared with me of better practice, and I cared enough
18:22
about the maintainability of the code and having best practices in the code for future developers that I changed my code and used these values elsewhere. Another kind of bump that we hit was that documentation quickly grew out of date,
18:41
and developers, when they saw this out-of-date documentation, they thought the information would be kind of useless. And so they ended up going to the ops team a lot more often to ask the same questions over and over again, and so we learned the importance of maintaining documentation so that the ops team wouldn't have to go through
19:02
and repeat themselves over and over again. Another challenge we faced was actually cultivating the right culture for DevOps. We had the additional complexity of having offshore developers, and due to time zone differences, there is really a limited amount of time
19:22
that we could do knowledge sharing with them. Also, they had kind of different values than us. They didn't exactly follow our company values, so that made having them learn Chef kind of a challenge. And also, we had developers who were just reluctant to learn Chef.
19:42
They thought, you know, we're application developers. We only deal with application code. You know, they still saw the wall between developers and ops as firm, but as we continued on and more developers got into writing Chef, and we kind of evangelized
20:02
more and more of how this was helping us to be better developers and have better workflows, even the most reluctant developers kind of joined in and started learning Chef. So I was talking to Jeff, one of our developers, a few weeks ago, and I remember this quote,
20:22
so I thought it was really interesting. He said in relation to working with Chef, if it's not part of your main job, it's hard to value and retain the knowledge. And I found this interesting because Jeff is by far, is a developer that has by far and large
20:40
contributed the most to our Chef code. And if he felt this way about Chef and DevOps, you know, you can be sure that almost all the other developers felt the same way. And at Square Root, we found a good way of tackling this problem, is just to have better documentation that's easy to refer to, that developers can constantly refer back to
21:03
as they're working with Chef and DevOps. So this is kind of the timeline of commits of our Chef repository. We started this process in the latter half of 2015, and it took about six months for
21:20
most of the developers to get ramped up in Chef. And all, it's been about 18 months through this process. It's taken us a little longer than we wanted to to get all developers working in Chef. But as you can see by our current repository, we have 59 contributors, and that's pretty much
21:42
all the developers that have worked on Square Root code. And so we view this as a great success. Especially in our data engineering department, they used to have hundreds of configuration files that they would mainly copy to every new server that they had to stand up.
22:02
And through learning Chef, and kind of using Chef for its intended purpose of infrastructure management and configuration, they were able to automate this process so that new servers could be stood up very quickly and automatically with less errors happening.
22:25
All in all, our new services and environments used to take weeks, even possibly months, through the manual configurations and tweaks to stand up, and now with everything made through Chef,
22:41
we can automatically set up a new environment with several servers in just a matter of minutes. And now Dev and the Ops team are working together. We have great shared application context and infrastructure context so that we're much happier
23:04
working together, we are fine dealing with Jenkins builds when they break because we all have that shared context and those types of issues get fixed much faster. Now there's still a work in progress.
23:22
Of course, as I said before, we can always have better documentation to help developers work with Dev Ops. And we still have existing anti-patterns from the old code that we still haven't been able to change due to our Ops team just trying to keep up
23:40
and maintain status quo in our Chef repository. Looking forward, we hope to integrate InSpec into our Chef repository, into our Chef code, so that we can make sure we're compliant and actually have a kind of automated test suite
24:00
to make sure that changes that we do don't break things. And this will help improve developer happiness because as I said, one of the main pain points is testing takes an extra long time. Also, we hope to integrate Docker and use containers to help portability of our services and servers in general.
24:23
Also, after viewing the keynotes this morning, I'm really excited about bringing Habitat into our Chef repository, into our Chef workflow, so that we don't have to deal so much with application dependencies and maintaining those.
24:45
So all in all, we've viewed this as a great success and I encourage all of you, maybe you're an Ops person and you're like, yeah, my workload is quickly becoming unbearable and I wanna share this workload with developers,
25:01
kinda here are some takeaways from the process that I hope you can take back to your organization. First of all, start before you need to. We thought this process was gonna be pretty simple and straightforward, but as I said earlier, it took a bit longer than expected
25:24
and our Ops team actually had their workload increase dramatically before it got better. And so you need to start this process before your Ops team really has their hair on fire. Also, take care with onboarding. You're adding an additional piece
25:42
to the workflow, to the data workflow of developers and so you need to take care and make it easy for them to kinda integrate this extra piece, whether it's with documentation or just easy access to Ops team to get questions answered.
26:03
Also, all in all, integrating developers into the DevOps process has led to greatly shared context and a greater ownership of both developers and Ops for the software and this just leads to better software being built and deployed.
26:24
And that's it. I'll probably take any questions at this time or any comments. Great, let's give it up for Annie and I'm gonna hope, if you have a question, please raise your hand, I'll bring you the microphone
26:40
and then please speak your question in the microphone so that our viewers later can hear the benefit of your question. So you said a very good story about transforming developers. I got a tough job to transform about 500 developers where I work into the Chef realm. So how long did your journey take?
27:02
I don't know how many developers total. 59 whatever number. Is it two weeks, two months? How long did it take? So for the basics of Chef, just getting their initial Chef PR took a couple of weeks.
27:21
But as I said, it was kind of a piecemeal process where my team first integrated into Chef and then other teams, after they saw the results of my team, kind of went along with that. And so each time a different team got into a Chef code, the Ops team had refined the process of onboarding them
27:42
and so like with the latest developer who just joined our team six months ago, it only took her a few hours before she submitted her first Chef PR after learning about Chef. Does that answer your question? So it kind of varies, you kind of learn as you go.
28:07
First off, great talk. This two-part question. First off, when you saw your developers start to do things, you started asking them to do things, what kind of cookbooks were they generating for you? Were they infrastructure ones or more app deployment?
28:21
And second off, how did you democratize the work processes of like environment pinning and cookbook creation and promotion to Chef Server? Great question. So first of all, we were building, we were doing a lot more app environment setup,
28:43
but the data engineering team, they were setting up Postgres servers and Kafka clusters and kind of that sort of architecture. So it was kind of both. Oh yeah, so that was kind of another bump in the road because we ran into issues where developers
29:03
would forget that they needed to upgrade the cookbook version after they made changes and so things would kind of blow up on the server. Luckily it was just dev, but we ran into those issues and then so the ops team kind of had to formalize processes after we learned what didn't work
29:22
and then we added that to the documentation. And we kind of have like highlights and bookmarks like make sure before you submit a pull request that you do this or make sure before you run Chef Client on the server you do this. So that helps with that.
29:47
Yeah, in my company we have a large development group that's very good in JavaScript, they're very good at things like that, but we're also very fast paced and the concern that I have is that we're going to hear a lot of we don't have time.
30:01
What's the biggest landmine you ran into with that sort of mindset and how did you get past it? Kind of a reminder of like how long it took our ops team to make changes that were necessary
30:21
for our application code to be deployed and how long the developers were kind of waiting for those changes to happen. And it was kind of reminding them that with this time that they're waiting for the Chef code to change they could actually be learning Chef and do the changes themselves so that they could move faster
30:42
and then also all these developers are smart, they've learned languages before, Chef is kind of a domain specific language and so they understood the process of it and they understood that it would make the pace of changes faster in the long run.
31:07
I think like many of us, I'm also trying to go through the process just with a very large company here and we don't have time and they like the benefits until they learn they have to learn something and then they're not as interested. But I'd be curious if you noticed
31:21
did the adoption get easier and quicker once you had more of a foundation of cookbooks for people to use as reference or for reuse and once they were able to reuse and saw the power did that help like get the ball rolling faster? Yeah it definitely helped especially with our documentation.
31:41
Sometimes our ops team would just be going through cookbooks and they'd be like oh this is a great example of this best practice in Chef and so they would copy the code snippet into our documentation so that when future developers came along and they're like hmm, I wonder how to do this in Chef. They could kind of refer back to documentation
32:01
and have a link to a cookbook or a template that did this especially well. And so it helped developers ramp up and move faster.
32:22
Quick question, from a training and readiness perspective did you guys go and send developers through training classes with Chef or bring anybody in or do any of that kind of work? No we actually relied on the one ops person that really knew Chef very well and was hired because of his extensive Chef knowledge.
32:43
It was only after we had exhausted all of his knowledge and then developers wanted to know more then we reached the external resources to learn more about Chef.
33:03
Perfect time for one or two more questions. There's any out there. Did you guys set up your Chef team as kind of a standardization team or how did you I guess set up your specific experts
33:23
in Chef to be able to train others or expert? Was that his main role then? That's not his main role but he was the one that we turned to most often when we had questions. It actually kind of grew organically. I talked about the idea of a DevOps Guild
33:42
as kind of a cross team group and ideally it would have been the person who worked most with Chef out of each team would come together and share best practices and talk about them. But that was another reason I kind of fizzled because we didn't really appoint a person
34:01
to be part of this Guild. It kind of grew organically as more developers worked in the code. Like I became the go-to Chef person on my team because I really enjoyed working with Chef. I had worked with infrastructure management before
34:21
in a manual way and so I was comfortable working around servers and other infrastructure. And so we've just been lucky that it grew organically and we've managed to find one person on each team that can kind of be the point person for the rest of the team to ask Chef questions to.
34:47
Thank you for the talk. How did you actually make the value proposition to the developers? You mentioned that the majority of them are remote. Did they send them a pizza or buy them drinks or anything to help convince them?
35:07
Well, we also chose a few examples of times when software had worked correctly on their local machines and also development and staging but somehow went wrong in production. Like something didn't work correctly in production
35:22
and it was because the ops people who had made the changes didn't have the correct context. And so something had gone wrong in production. We kind of used that as an example of if you had more context into what was actually happening with the infrastructure and how it was all set up,
35:42
this stuff wouldn't be happening with your code. You wouldn't have to be going back through and making quick bug fixes and such. So that was kind of the value proposition that we used with our offshore team. Especially as they valued more, not necessarily,
36:02
I guess not necessarily more customer value but they valued putting more correct code out there, pushing more code constantly. And so when they had to slow down to do these bug fixes since the infrastructure was a little off, then that kind of hurt their bottom line.
36:22
This will be the last question. Great talk, by the way. And I just kind of wanted to share my experience to see if it was kind of similar to what you're saying because I think it sounds like it's a resounding question. I know when we started Chef, what really helped our developers get on board is just having people talk about Chef and be passionate about Chef.
36:41
It was something really cool. It was really seeing the efforts. We actually at Disney have our own supermarket that our developers can contribute to and that just kind of lit a fire. So if you can get the passionate people about it and get it going and evangelize it, that really spreads throughout the whole organization.
37:01
And it started in a small group in Disney and now it's been spreading like wildfire throughout the company. So I don't know if that's kind of, did you have kind of that same experience with Square Root? Yeah, definitely. Like I said, we had a couple of very reluctant developers get into it but as developers work in Chef,
37:22
these reluctant developers kind of saw the benefits of how quickly our teams are moving due to working in Chef and they saw these additional benefits that could help their job as application developers. And so due to this kind of evangelizing, as you said, yeah, we found more developers get into it.
37:45
Perfect. Let's give one more round of applause for A. Se.