A Peek into an Enterprise Development Operation Team
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 170 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/50853 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Enterprise architectureOperations support systemWeb serviceBuildingPlanningEnterprise architectureGroup actionOperations support systemAdditionTerm (mathematics)Order (biology)Presentation of a groupComputer animation
00:52
Computing platformSoftwareNumberBitClient (computing)Dependent and independent variablesPerspective (visual)QuicksortComputing platformExpert systemGraph (mathematics)Right angleContext awarenessElectronic visual displayTwitterSelf-organizationMultiplication signOperations support systemInstallation artNumberSoftware testingDecision theorySlide ruleProduct (business)Business modelModeling languageSet (mathematics)Bit rateControl flowTrailResultantSoftware development kitBuildingView (database)AverageBounded variationComputer animationEngineering drawingDiagram
04:15
Alpha (investment)Beta functionBuildingWeb serviceConfiguration spaceNetzwerkverwaltungContinuous integrationIntegrated development environmentMaxima and minimaSoftwareBeta functionBranch (computer science)Product (business)INTEGRALComputer hardwareAlpha (investment)Real numberMetric systemConfiguration managementQuicksortPresentation of a groupFocus (optics)WordGroup actionFrequencySelf-organizationService (economics)Scheduling (computing)LaceGrand Unified TheoryTransformation (genetics)Integrated development environmentTemplate (C++)Bound stateExtension (kinesiology)Cycle (graph theory)Revision controlProcess (computing)Data centerWave packetMereologyInstallation artData analysisBitLatent heatLogic gateArchaeological field surveyPattern recognitionCoordinate systemBit rateAnalytic continuationThread (computing)BuildingDatabaseSystem administratorContinuous integrationComputer animation
07:41
Process (computing)Broadcast programmingCapability Maturity ModelStack (abstract data type)PredictabilityNetzwerkverwaltungSoftwareModul <Datentyp>Continuous functionBuildingArchitectureSelf-organizationLine (geometry)Database normalizationWeb serviceGreatest elementTime zoneProcess (computing)Software developerLevel (video gaming)PreconditionerINTEGRALModule (mathematics)Web serviceSelf-organizationPhysical systemOperations support systemProduct (business)Software testingStack (abstract data type)BuildingLogic gateNetzwerkverwaltungBlock (periodic table)Arithmetic meanGroup actionDifferent (Kate Ryan album)MathematicsDatabase normalizationQuicksortCollaborationismSoftwareDivisorTelecommunicationPerspective (visual)Computing platformCapability Maturity ModelPlanningIntegrated development environmentDomain namePredictabilityNumberCycle (graph theory)Scheduling (computing)Focus (optics)Analytic continuationTransformation (genetics)Test-driven developmentInstance (computer science)Software architectureVertex (graph theory)ArchitectureExecution unitCellular automatonDataflowMultiplication signEqualiser (mathematics)Extension (kinesiology)EstimatorCharge carrierComputer animation
13:05
Continuous functionBuildingPhysical systemCurve fittingAlpha (investment)PlastikkarteEnterprise architectureComputing platformFeedbackStability theoryProgrammable read-only memoryUniform convergenceCoordinate systemContext awarenessPresentation of a groupQuicksortProduct (business)Element (mathematics)Operations support systemProcess (computing)Perspective (visual)Cellular automatonAnalytic continuationLogic gateMultiplication signÜberlastkontrolleDependent and independent variablesBeta functionLoginScaling (geometry)Intelligent NetworkVirtual machineComputing platformFeedbackLibrary (computing)Software frameworkPlug-in (computing)Point (geometry)Data storage deviceIndependence (probability theory)InternetworkingTime zoneSoftware testingSelf-organizationPhysical systemCycle (graph theory)2 (number)Shared memoryGroup actionWordStress (mechanics)Server (computing)Web serviceObject (grammar)Phase transitionAlpha (investment)Extension (kinesiology)Level (video gaming)MereologySlide ruleStability theoryBuildingAcoustic shadowControl flowArithmetic meanSoftwareComputer animation
18:18
Electronic mailing listSelf-organizationComputer animation
18:52
Continuous functionSoftware testingServer (computing)PredictabilityBuildingMathematical optimizationGreen's functionBinary filePhase transitionBuildingSoftware testingModule (mathematics)Order (biology)Extension (kinesiology)Physical systemTask (computing)Interface (computing)Maxima and minimaForcing (mathematics)BitAnalytic continuationPhase transitionPoint (geometry)NumberDependent and independent variablesOperations support systemPerspective (visual)MereologyExpert systemFocus (optics)Instance (computer science)AreaError messageMathematical optimizationProcess (computing)Cartesian coordinate systemSelf-organizationQuicksortBinary codeServer (computing)Logic gatePredictabilityStress (mechanics)Integrated development environmentRule of inferenceSoftware frameworkLevel (video gaming)Computing platformTest-driven developmentConstraint (mathematics)Presentation of a groupUser interfaceSoftware crackingSound effectScaling (geometry)Slide ruleUnit testingElement (mathematics)Line (geometry)DataflowShared memoryBusiness model1 (number)Marginal distributionEstimatorEqualiser (mathematics)Intelligent NetworkSoftwareComputer animation
25:47
Phase transitionSoftware testingCodeMathematical optimizationCompilerModul <Datentyp>NetzwerkverwaltungSanitary sewerSoftware developerNumberTrailQuicksortLogic gateGroup actionNumberMeasurementComputer hardwarePerspective (visual)Server (computing)Goodness of fitCuboidMultiplication signSoftware testingBuildingMarginal distributionCodeArchitecturePhysical systemSelf-organizationModule (mathematics)MereologyContinuous integrationFocus (optics)SoftwareLocal ringUtility softwareModul <Datentyp>NetzwerkverwaltungComputing platformConcurrency (computer science)Structural loadLastteilungMetric systemQueue (abstract data type)Right angleCompilation albumData centerSteady state (chemistry)Graph (mathematics)AlgorithmTime zoneClient (computing)Line (geometry)Wave packetPhase transitionPredictabilityBusiness modelVarianceINTEGRALSpherical capMotion captureGreen's functionSystem callError messageWebsiteEqualiser (mathematics)Sound effectScheduling (computing)Computer animation
32:43
NumberSoftware developerPhase transitionComputer hardwareLastteilungConstraint (mathematics)Continuous functionVariety (linguistics)VolumeOperator (mathematics)Computing platformCharge carrierNetzwerkverwaltungArchitectureArchitectureProcess (computing)AreaSystem callGroup actionExecution unitGoodness of fitDifferent (Kate Ryan album)CodeStapeldateiNatural numberSelf-organizationPlanningDecision theoryQuicksortScheduling (computing)Module (mathematics)Software architectureMultiplication signTransformation (genetics)Set (mathematics)NetzwerkverwaltungPhysical systemTouchscreenBusiness modelDataflowStructural loadInstance (computer science)Focus (optics)Variety (linguistics)Dependent and independent variablesTraffic reportingMetric systemEmailComputing platformPerspective (visual)Volume (thermodynamics)Projective plane1 (number)Physical lawMathematicsSpacetimeCopyright infringementNumberSoftwareSoftware testingINTEGRALProduct (business)Web serviceWave packetShared memoryConfiguration spaceContext awarenessContinuous integrationLevel (video gaming)Slide ruleComputer hardwareSinc functionAnalytic continuationRepository (publishing)Phase transitionRight angleUnit testingSoftware repositoryLinear regressionBuildingBitComputer animation
39:39
Information technology consultingRevision controlProcess (computing)AreaQuicksortDifferent (Kate Ryan album)Task (computing)Projective planeNumberSoftware testingDisk read-and-write headCountingJSON
Transcript: English(auto-generated)
00:00
can you hear me okay okay so uh... uh... this is about development operation and in devops uh... we deal with this stuff like this uh... and have to be agile and uh... think on uh... on our feet in order to uh... to react quickly on stuff like this but i think this will work uh...
00:22
with uh... plan b uh... thank you for staying uh... this late it's in the uh... in the conference uh... that's the last uh... talk of the day uh... this presentation is uh... labeled it to peek into an enterprise development operation team uh... so many are familiar with the term devops
00:43
and my name is nils and uh... i'm the manager for for the build services uh... group in in patrol uh... which is a uh... uh... devops uh... team see if this works all right so uh... just a little bit more context uh... the trail is uh... is a product that we build in the devops uh... group
01:02
uh... it's uh... it's a uh... a platform in the oil and uh... gas industry uh... it enables uh... experts to work together and make make the possible and uh... make best possible decisions
01:20
all the way from exploration to production so what that means is that uh... the software models the earth uh... so that the geologists uh... can can they get computerized uh... models uh... out of the earth uh... it also go through uh... goes through uh... reservoir characterization and and modeling and and also to uh... through uh... production
01:42
answer production engineers can can uh... find out where to uh... uh... optimal optimally uh... uh... produce uh... oil and gas or carbon we also have an uh... an sdk on top of the on top of the platform uh... it's based on uh... the ocean uh... technology and uh... you can you can learn more
02:00
about that at the at the stand down uh... down in the galley here that's a little bit about the uh... the context so it's patrolling its ocean uh... this slide here is and this whole slide sets uh... i'm gonna try to lead you through a journey uh... that we've uh... gone through the development operation uh... team
02:23
and and and basically the result of that journey is that we have uh... been able to increase our customer customer responsiveness uh... with improved quality while we've been growing this platform uh... footprint so the graph on the on the left hand side
02:40
uh... displays uh... the trend uh... as far as having a uh... uh... daily installer uh... available uh... you see that the the the success rate of the installer has gone up but also the variations of uh... how successful how often we can get a successful uh... installer
03:02
uh... has also gone gone down you can see in the and there on the left hand side from a uh... a client perspective uh... we've been able to uh... give out uh... fixes more quickly as a result of of this
03:22
the chart on the right hand side is more of an internal uh... this displays how we've increased the number of automated tests uh... while we've also increased the average success rate of the builds themselves uh... and at the same time we've also reduced the the build time
03:42
so these are sort of internal KPIs I'm just going to take a quick quick break here uh... in the audience who's in the DevOps team okay so if you uh... in the audience who's in uh... in a uh... development organization with
04:01
more than fifty developers uh... more than uh... a hundred developers and more than that two hundred developers okay so then then you're gonna understand some of the things that i'm going to be going through my group uh... like i said uh... is patrol build services group
04:20
uh... our bread and butter is development environment hosting uh... it's about configuration management as basically source control uh... it's uh... running the continuous integration tools uh... and it's also licensed administration of this development environment so specific things we set up team branches we set up release branches
04:42
we do auto integrations between these branches uh... we offer personal builds uh... we also have team builds or specific feature feature builds and integration builds uh... as well as the setting up the daily installer
05:01
uh... we create the create the installer by using templates uh... because we also have an SDK on our product we reuse these templates for uh... for internal and external use uh... we also put in health checks to make sure that it's installed on on on the hardware that we recommend and uh... within the bounds that uh...
05:20
that uh... that we recommend and we also uh... offer extensions uh... installers we also do a lot of release coordination uh... in short we can separate our development cycle into uh... pre-alpha period alpha period beta period and uh... to to get to the commercial and so there's a lot of scheduling involved
05:42
uh... alignment of uh... the bills to get them to integrate at the at the uh... predefined uh... and milestones uh... bills of white list uh... uh... external software that we use internally in the in in the product uh... uh... to the organization
06:02
uh... and also documentation overall uh... to the uh... to the development uh... organization the last piece is uh... uh... sort of uh... uh... thread that goes through the uh... the presentation is a big focus on continuous improvement uh... both on the build side and the IT environment side
06:22
and so specifically it's about automates where it's it's uh... feasible and where it makes sense training and documentation of the developers uh... and alignment with corporate IT so maybe some of you have uh... experience with the data centers uh... we serve her have our own internal data center
06:41
uh... and in many many uh... uh... scenarios it would be as if we were dealing with an external data center uh... supplier because we have all this data of of uh... builds uh... coming in at uh... fairly fast rates uh... and uh... with uh... with uh... big breath
07:02
lots of data coming in uh... we also do a little bit of data big data analytics to take a look at this to see uh... you know how can we better uh... create these workflows for for personal bills for for uh... quality gates and and stuff like that
07:23
uh... and so there's certain ways you can do that you can uh... have uh... questionnaires against surveys you can have uh... uh... you know gut feelings and stuff like that we also have data that we collect and and we also uh... look for a pattern analysis in in our in our database
07:42
all right so the challenge uh... that we were facing uh... before the transformation was basically split in three uh... organization process and technology from an organization perspective uh... were geographically distributed team
08:00
uh... as you can see in the in the map there uh... you have a quite a few time zones uh... which uh... uh... is a challenge in itself when it comes to collaboration but also uh... cultures and and uh... and uh... and schedules and and so and so forth Another challenge was the definition of a quality gate
08:23
uh... what does green mean to you is not necessarily the same definition as as uh... what a green bill means to to somebody else or or test even you can do this at all different kinds of granularity levels uh... units uh... module integration all the way up to the product
08:42
schedule alignment uh... our platform also plugs in with other products uh... which may have a different schedule than the uh... than the schedule that uh... that we have for the platform and so there is product coexistence in the system as well uh... although the organization as a as a whole is is uh... actually uh...
09:03
uh... quite good as far as uh... uh... software development practices and and TDD and and agile uh... practices and there's there's also spread within the team because we're so big uh... on the maturity level of practicing these uh... these techniques
09:22
the software itself is is also uh... tightly integrated so you can basically uh... carry a workflow through uh... through many of the sort of the the vertical domains so it's it's a very integrated product and because of that uh... we have dependencies both on the business side
09:40
and on the on the technical side and because the uh... the software has sort of uh... gone through uh... acquiring smaller companies and and acquiring uh... deep science from from other places we've also sort of uh... acquired a tool stack that's also been uh...
10:01
a challenge in itself because it's uh... good systems by themselves but uh... don't necessarily work that well together alright so in this environment the challenge was to uh... to get predictability traceability repeatability and quality out of the system
10:22
so before i go into how we actually attacked this this challenge uh... i need to go through a couple of uh... or a few preconditions and guiding principles uh... number one our software or or or platform is uh... the software architecture is layered
10:41
is modular and it's it's extensible uh... the software process i would say overall uh... we're practicing incontinence integration and there is is uh... buying and test automation and uh... and there's a consensus in the organization that test automation is is uh... is a good
11:00
investment so with these building blocks within agile suffer suffer practices uh... i also wanna uh... highlight the fact that uh... uh... to do change management uh... within the organization uh... it's also uh... something that uh... actually developers are quite used to
11:20
and because we do it quite often uh... it's also sort of a motivational factor uh... because people they uh... they like to develop on the latest and greatest uh... of technology and so basically uh... you have the motivation aspect as well and from management uh... we we have buy-in and support
11:41
uh... because of the uh... the the the communication channels that that we have so from an organizational perspective basically have top-down support and we're able to uh... to implement uh... from the bottom up uh... because of the the motivation uh... generally in in the in the organization
12:00
so these are so the preconditions for the development operations team uh... we have some guiding principles it's uh... automation it's uh... being able to uh... design the system for self-service ability meaning uh... uh... we're not putting ourselves in the critical path of uh... of the tons of developers
12:21
they can basically serve themselves uh... within the group we have a strong uh... level of uh... of knowledge redundancy within the group and there's also uh... at focus on service continuity so and because we're physically located in norway you can't really detect that from the other uh... uh... centers because uh...
12:45
uh... world holiday for instance uh... you can uh... we have service continuity plans uh... in place for that and last but not least uh... there's a big focus on on continues monitoring of the whole system
13:00
which feeds into the continuous uh... improvement uh... cycle alright so that was the uh... sort of the the the context the product uh... the situation the challenge uh... then now the presentation goes into uh... what is it that we wanted
13:21
so just just a brief explanation of our our build system so uh... we have a bunch libraries uh... first and second and third-party libraries uh... we have our platform that we build and then we have because of our extensibility framework
13:40
we have a bunch of internal plugins that we bundle with the platform uh... when when we uh... shipped the dvd all these developers are uh... also outlined the uh... the sort of the phases the pre-alpha alpha and beta and the whole point of starting early with all these developers is to get early feedback in the build process
14:03
and then when we get to commercial we have uh... sort of an uh... it's sort of the equivalent to itunes we have an ocean store where you can uh... publish your plugins and basically uh... be an independent software vendor on top of our platform
14:23
so there's a bunch of build ripples in between these build ripples uh... we've invested quite a lot on quality gates and i i'll get back to that in a minute so basically what we wanted out of this continuous delivery build system uh... was we wanted uh... we wanted to facilitate a Ferrari, we wanted to facilitate speed
14:42
we wanted to go through this very quickly if we had to we wanted to protect ourselves against uh... ignorance of the quality gates so we had to have trust in the quality gates and they had to be predictable by everybody
15:00
uh... but uh... we also wanted to avoid congestion in these quality gates we had to be performant they had to be easy to understand uh... we couldn't create anything that required the dev ops team to go in and and explain the logs and and so and so forth because then you're creating waste in the system by people waiting on each other
15:22
and it doesn't scale very well so i separated into uh... technology process and organization from technology uh... quite simple we want to fast build and because of that we needed dependency awareness
15:41
we wanted the gates to be smart so we wanted smart gated check-ins and we wanted the tools to be uniformly distributed against all the machines that we haven't served part as well as the the desktops were were developers are working locally across all the centers they're contributing
16:02
to to the platform from that process perspective as a as i mentioned in the in the previous slides we want early feedback uh... and we wanted the the release dates to not be uh... stressful uh... raise of hands uh... who thinks it's stressing becomes a little bit
16:22
stressful in the organization when you're one week away from release time after you okay so that's what we wanted to avoid we didn't want this is stressful really states we wanted to practice this process throughout the development cycle redesigned developed test release and and uh... and uh... all this through this uh... stability promise
16:44
throughout the uh... pre-alpha the alpha beta to the to the commercial release and from a process perspective also wanted traceability and continues monitoring because we wanted to be able to adapt if we if we had to sweet we still wanted
17:01
this uh... element of continues improvements in andy in the process so from an organizational perspective uh... what that really translates into is we wanted people to to have clear roles and clear responsibilities there's no question about uh... if the bill breaks who does what uh... what does this mean
17:21
and so on and so forth and and from a development operations perspective we wanted to sell service uh... mentality so whatever we did next time we didn't have to do it we didn't have to go into the servers to to produce some of these logs to give to the developers because they would be already attached with the with the bill lock
17:41
it's just one example and because we wanted within the develops team we wanted sustainable working hours like i said in the and and the map earlier in the slides that uh... we produce time uh... or we support uh... time zones all the way from houston to beijing and so that's pretty much uh... twenty four seven
18:00
and because uh... also because of that and because of the uh... the continuity uh... objective uh... we had to have knowledge sharing within the group so we had to come up with some sort of the system where we can all share without having to uh... sort of shadow each other at all times all right so that was the that was the challenge
18:21
uh... that was the uh... the wish list and so then the question was uh... uh... and i mean you're you've probably been uh... inspired by many of the things in the uh... from the other talks uh... here and you think yeah i want to do this but we're a big organization
18:41
how do you do this how do you do this with the hundreds of people so i'm not gonna say that this is the way to do it but this is what we did we realize that the roads to continue delivery uh... uh... would be a uh... uh...
19:01
along walk it would be driven by constraints by by business we had to have annual releases this is sort of like fixing an airplane while you're flying it uh... so we had constraints uh... but ultimately we wanted the from a technology side we wanted
19:21
an agile and intuitive traceable build system with minimal human intervention uh... and we wanted this to be a part of the culture this this whole notion of continuous delivery so we divided into incremental steps uh... phase number one uh... was all about uh... predictability in the system
19:41
and i'm not saying that the system wasn't predictable before we did this but what i'm saying here was that the the focus was on on predictability uh... in the system so specifically on the on the technology side for instance we wanted uh... server and desktop alignment what that means is if the error uh... the build error happened on your uh...
20:03
on your uh... on the build server uh... it would be reproducible on your desktop as well it wouldn't be like oh it's only happening on the server it doesn't happen for me aka it's not my responsibility uh... so we had to invest in that uh... we also did uh... a lot of investment in in test automation
20:23
and in order to achieve this we put aside a task force that really went after this phase number two at that point we had trust in the system uh... we had predictability that we wanted and uh... we wanted efficiency out of the system so from a technology perspective
20:41
build optimization uh... and from a process perspective the investments that we made in the previous phase on test automation we wanted to preserve those so we we uh... we created a uh... uh... sort of a guideline that that we named keep it green this was uh...
21:01
rules set of uh... i think it was between five and ten rules that basically everybody had to follow uh... we printed those on flyers that went all through the organization and to basically protect that investment that we did in the previous phase uh... we also in order to support this uh... uh... this initiative we also created something called build buddies
21:23
so these would be uh... looking after uh... builds that sort of uh... fell in between the the cracks uh... of the system even though people were supposed to take ownership and uh... there might have been still uh... unpredictable things in the system that basically they took uh... took took care of and communicated back to the devops team
21:44
so phase two was about efficiency getting speed phase three was about effectiveness so even though inefficiency even though you do something fast it doesn't necessarily mean that you do the right thing so in phase three
22:01
uh... we from a technology perspective we focused on uh... trying to share these binaries that we uh... that we had already built and uh... in the server part with the developers so that they didn't have to build them locally the gates that we had invested in and uh... in phase one and uh... sort of retained
22:21
in uh... in phase two we wanted to automate them uh... and have automated gated check-ins in phase three and last but not least uh... from an organizational perspective we created something that we labeled the build federation build federation would basically be the devops team being a central team
22:42
that is responsible for the rules uh... for the entry into the server part uh... and sort of the nuts and bolts of the system and then we would have deputies with uh... with uh... the mandates to make you build decisions out in the different teams
23:00
so that was also uh... one f one effort uh... that was put in place in order to make the system scale okay so in this slide i've made them into looks like maybe it's all uh... homes like uh... three separate phases uh... many of these phases had elements of phases above and beyond
23:22
or above and below them so in if you if you go around the organization uh... i'm not so sure if you'll finds you know these these sharp lines in between phase one and phase two and phase three but uh... but from an overall perspective this this was basically how we uh... we broke the the challenge down into incremental steps
23:47
all right so a little bit more on on phase one then quality gates uh... there's been lots of talks about tdd and and testing and how to do it and how not to do it even the the previous presenter here had some had some thoughts on that
24:01
this is more about uh... how you actually do it from uh... from an operational perspective you've probably seen this pyramid before uh... you have a thick layer of unit tests that are cheap to run really fast easy to develop don't really need that much mocked environment maybe doesn't even need data and stuff like that and you create lots of them is a cheap
24:21
and sickle areas you have modules uh... then it's it's uh... it's testing the module by itself maybe by an extensive sensibility interface and maybe you start merging uh... modules and and and you mark a little bit around when some data started twisting and turning these modules and into stressful environments
24:42
and that's layer number two layer number three is and not necessarily with uh... with the user interface uh... but at this point you bring up the application run through some workflows so again i have to emphasize that the uh... the platform is very integrated uh... this is very important for us to run workflows because the module
25:01
the module tests in isolation might all be green but when you put them together they're not necessarily green because the interfaces are wrong or they're not passing over the the correct data or syntax and so on and so forth the last layer is uh... i call it UI so this is there are many frameworks to be able to support UI
25:23
the focus in our organization was at this level the competency and the organizational responsibility switched it switched from the developers to the quality assurance uh... guys to the uh... to testers uh...
25:41
and uh... they are typically the main uh... experts so they know all this oily stuff that the developers uh... might not necessarily have deep knowledge of so they twist and turn the software uh... by using ui tests and also then testing the the top layer of the of the system
26:04
all right so then when we invested in this testing architecture and uh... defined its quality gates uh... we also want a metric to to ensure that we're on the we're on the right track here uh... so one way of uh... measuring and that is to measure the number of
26:21
tests they have in the system uh... that in itself uh... may not necessarily for produce that much value and so we also decided to to track code coverage because that meant that uh... we were adding value to these new tests creating a different footprint that the previous uh... tests uh...
26:42
uh... uh... we're not necessarily uh... giving you might also argue that that itself is also not really uh... a good k p i or or a uh... a metric to to uh... to sort of support your the quality of your quality gates uh... what i didn't put in here was that uh... the number of defects uh... at
27:04
the client side uh... which i guess would be the ultimate way of measuring these quality gates also went down uh... so we knew that we were doing something right uh... we had some metrics to to try to to uh... to capture our goals uh... and we put that in place and uh... and then we we uh... monitored that
27:25
so basically in summary the investments in in automated testing uh... was about agreeing on the quality gates uh... and agreeing on the on the test uh... uh... architecture uh... and the develops the uh... ensure that uh... whatever the uh... they were doing in uh... these
27:43
even remote places uh... away from norway uh... produced we were able to reproduce in the in the in the servers uh... and then also we put aside uh... the government's plan uh... to protect this investment that that we made and mention keeping green was was one of the uh...
28:01
uh... one of the efforts so that was phase one predictability and quality gates faced here i said was making the bills more efficient uh... so very simply said it was uh... from going uh... from a uh... monolithic you model with this or the big gate at the end of it maybe uh... even manual uh... to a uh... modular
28:23
what i haven't modeled into this picture to sort of uh... spare you the my drawing skills was uh... the interdependency between all these modules uh... but the idea was basically to uh... have quality gate at the end of each each each module
28:41
and and also as they were integrated up uh... in the system the metric that we uh... we use for this uh... spread simple uh... was the time that it took to build uh... system and as you can see here and i i put this in here on purpose because uh... this was sort of working on on moving parts because
29:02
as we were going through this transition we're also growing the that the platform to see the lines of code was actually uh... uh... going through the uh... going through the roof as we were actually decreasing the bill time so so those are it's almost like an oxymoron but uh...
29:21
but that was the achievement and in summary the build optimization uh... we looked at everything all the way from compilers to modularization of the system uh... at very uh... a good system on dependency management algorithms that could spin through the whole system and compute the
29:41
dependencies online and only rebuild what was necessary and this goes beyond what uh... what visual studio can do and also uh... tools like incredible than and stuff like that concurrency was also uh... uh... a big focus uh... on our side
30:01
uh... and then we uh... got incremental build so so basically went from hours uh... hours and hours uh... i can't really say exactly the numbers but let's just say hours and hours down to minutes uh... while we were growing this uh... this platform in phase three we wanted to be uh... we had a fast build
30:23
uh... we wanted to make more effective we wanted to do the right things at the right time and and so and so forth so i put this picture in here just to remind you that uh... it was a globally distributed development team so it's uh... basically spread around all the way from houston to beijing
30:42
uh... with the devops team sitting in oslo uh... and uh... and the local data center as well so to make the bills more effective uh... uh... on on the on one side we focused on uh... hardware and uh...
31:01
we looked at our server utilization and this is not necessarily load balancing within the continuous integration tool because it does that fairly well by itself out of the box uh... but this was more to look at uh... what do the developers in houston need when they get up in the morning as opposed to when we in europe leave for the day
31:21
as opposed to what's uh... sort of nightly uh... builds to the uh... asia or you know part of the organization so this was really looking at the schedule overall uh... both from a uh... from a deep technical uh... perspective all the way down to to the metal uh... but also from an organizational perspective
31:42
so as you can see in this uh... in this graph here we were able to get these uh... uh... loads that were were swinging so basically uh... we had parts of the day where we had huge build queues and people were waiting on these fast bills they were really fast but there was still because we hadn't scheduled them optimally then uh... people were still waiting
32:02
as we're able to get that uh... variance and uh... got into a good steady state load on the servers and and as you can see here we didn't really increase that the server part that much so only marginal increase of of uh... of the server part so it wasn't that big of a capex investment
32:23
sort of on the softer side so we're getting more builds of the system we're building them right we're building them fast we're building them at the right time uh... but the DevOps team is still sitting in Oslo covering all these time zones and so we put a big focus on on uh... load balance from an organizational perspective as well
32:43
and that basically means training so we trained these deputies that were sitting locally to all these different teams to understand the system and to understand how the internal configurations of the system actually worked so that they could self service themselves out of our business hours
33:03
to see that the training time that was needed went down while we were actually getting more and more developers in the system so again we were scaling very well with this organizational model since summary we were load balancing leveraging hardware and people
33:23
one thing that I didn't mention was that we also put a big focus on back-up management of the DevOps team itself and made it visible to the other teams uh... did reviews so that uh... a selected number of people uh... the deputies that is uh... would understand what was in our pipeline coming up
33:42
both on schedules and and uh... improvements and we put a big emphasis on making the reports intuitive uh... so when you look at the report you didn't have to pick up the phone or send us an email uh... because it was clearly written uh... in the bit log what was wrong with your submit on the documentation side we made a conscious decision to to uh... to have that crowd sourced
34:06
uh... although it was referee basically anybody could contribute to the documentation they would be referred by the by the building uh... and last as i said also uh... bill scheduling uh... was really starting right bills at the right time
34:24
all rights last aspect of the uh... bring up in this uh... transformation that goes across all the phases and that was making the DevOps team itself more effective uh... on the screen is is basically the work steps of one of the workflows that the DevOps team
34:43
uh... is uh... has his or her job you basically receive the code, you compile it, you unit test it, you distribute the module, you integrate it and you do some regression tests on that integration and then you distribute it so those are basically the work steps
35:01
on the uh... different contexts or devices that you're in in these different work steps might be in the code repository uh... and allocated service space uh... maybe a continuous integration software repository uh... or on a network share this is just an example
35:21
so as you walk through uh... one of these work steps uh... you can also highlight areas where maybe you spend more time than than uh... you either uh... are told it should take or that you think is taking too long because of low hanging fruit or because of the technology itself should perform better than that
35:41
so basically have you walk through uh... uh... a uh... number of work steps like this uh... second thing is uh... we we took a look at was what is the uh... what is the nature of these work steps uh... what is the volume of it what are the varieties in these work steps coming in
36:02
uh... what sort of skills that is required uh... and what sort of flexibility is needed by the people that actually give these work steps uh... or these uh... requirements to us and how do we utilize the assets so the assets is both on the hardware and on people so then you end up with uh... maybe some there are more project uh... one of a kind
36:24
uh... sort of thing uh... where you integrate a huge module that you're only going to do once or maybe once every year uh... certain things are more of a batch nature uh... whereas other are clearly continuous flow and they should be uh... should be automated so you basically go through the work step and uh... and you find areas that uh...
36:45
uh... that you go through like this uh... and then you implement continuous improvement uh... actions accordingly after after you've gone through this alright so then then we're actually at the at the end of uh... of the slide set
37:04
so to summarize uh... like I said in the introduction we increased the responsiveness, quality while we were growing the uh... the platform footprint and uh... with the KPIs that I mentioned responsiveness to the customer
37:21
and uh... and also quality overall so how was this achieved uh... I would say then just to summarize it's because of the uh... the culture of continuous improvement in the organization sort of triggered this whole uh... investment uh...
37:40
and throughout both the uh... sort of as a backbone in the in the organization but also uh... while we were learning doing all these continuous improvements uh... we acquired this excellence in change management as i mentioned before we did lots of things incremental things within technology within process
38:00
within within organization uh... and we did this all uh... while we were sort of uh... flying the airplane so we're fixing the airplane uh... while we were flying it so very good risk plans uh... and change management plans
38:20
we also and another area that i would like to highlight that i think is important in this achievement story is uh... the continuous effort of intentional architecture so this is both on on the build architecture that basically follows the software architecture of the product itself
38:42
uh... so you have the product itself build architecture and test architecture and all these three follow each other uh... through this attentional architecture in the in the system and all this investment that we're making uh... we're continuously uh... monitoring uh... through our through our metrics
39:02
that we uh... that we decide on and sometimes we bring out old metrics and bring in new uh... but some of the bigger ones like for instance uh... the the percentage of daily builds uh... is is still there as one of the major metrics notion of continues improvements uh... and uh... we can continues uh... monitoring
39:22
is very important uh... for this culture of continues improvement because the continues monitoring basically drives our backlog uh... and uh... both in prioritization and in sort of uh... a road map uh... perspective that was it
39:42
uh... any questions
40:03
yeah so it's a good question so and the question i'm not sure if everybody heard the question was uh... this uh... is is uh... i think you said something else but i understood it as uh... this is sort of a simplified version of uh... of everything uh... if i were to do it again would i do the same thing again
40:23
i would probably do a few things different uh... uh... but would they the things that i would have done differently would they have uh... made things go faster would they have been more effective and so on and so forth i'm not so sure
40:41
i think the answer to question is actually no i'm not so sure if i would do so many things uh... different i have to highlight here that uh... this continues improvement i mean everything was broken down we had a backlog that was prioritized just like any other scrum
41:00
processes so we ran this actually as a scrum process if we saw that we started the project and we said we want to go this way we think it's this way here we were very agile and we were able to move very uh... very quickly we could change our backlog within weeks and we could do different things we also tried to
41:23
isolate the the tests and maybe that's one area that we could have improved on uh... all the way from the very start uh... defined the isolated tasks that could be done by somebody else because then you can hire consultant you can ramp up the effort uh... and and get things faster
41:40
get things uh... done done faster so that's that's one one effort but uh... uh... in general uh... being agile in the devops team so basically practice scrum within the devops team uh... i think think this this may be a little different from from what uh... sort of i t people think uh... uh... this is my impression
42:10
uh... so i didn't hear okay so so the question is about the ratio between the the the devops team
42:21
and the developers uh... so i'm not sure if i can disclose all those numbers but uh... it's it's uh... handful of uh... the the devops team is a handful of people and the number of development uh... or uh... number of developers is hundreds so that's sort of the ratio uh... then you get a feel for the ratio
42:41
and uh... and i have to say that uh... the the headcount of the of the devops team has stayed the same whereas the number of developers as uh... as uh... minutes any other questions
43:05
okay and i say uh... thank you very much and enjoy the holiday uh... weekend