DevOps Transformation at Absa Bank: Technical Evolution; Cultural Revolution
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 45 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/34577 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
ChefConf 201731 / 45
1
4
5
9
12
16
18
20
21
22
24
25
27
28
35
37
39
42
00:00
Transformation (genetics)Surface of revolutionTime evolutionSoftware engineeringRight angleSelf-organizationDigital rights managementSoftwareFault-tolerant systemCalculationService (economics)Division (mathematics)BitMobile appData structureProcess (computing)Social engineering (security)Natural languageJSONXMLUMLSource code
02:03
SoftwareRight angleAuthorizationPhysical systemMereologyCategory of beingIntegrated development environmentSoftwareEndliche ModelltheorieSoftware engineeringData conversionVideo gameGoodness of fitNumberQuicksortProcess (computing)BitPerturbation theorySummierbarkeitVapor barrierComputer animation
05:54
SoftwareContinuous functionIcosahedronCartesian coordinate systemBitOrder (biology)Right angleSoftwareGame controllerProcess (computing)Row (database)Virtual machineLattice (order)Instance (computer science)InternetworkingWebsiteMetropolitan area networkPhysical systemComputer-assisted translationSystem administratorMathematicsOperator (mathematics)LaptopMeeting/InterviewComputer animation
09:19
System identificationShift operatorQuicksortIntegrated development environmentGraph (mathematics)MultiplicationPhysical systemDataflowSoftware developerBlock (periodic table)Process (computing)SoftwareScripting languageCartesian coordinate systemChecklistCodeOperator (mathematics)Metropolitan area networkGame controllerMachine visionWhiteboardVector spaceRight anglePoint (geometry)MathematicsLevel (video gaming)Set (mathematics)Projective planeCurveSoftware testingIdeal (ethics)GodServer (computing)AutomationRevision controlDivision (mathematics)Computer animation
14:25
Multiplication signFeedbackDataflowShared memoryCartesian coordinate systemInstance (computer science)CodeShift operatorMultilaterationTouch typingAutomationSoftwareData structureTerm (mathematics)System callProduct (business)Process (computing)Insertion lossFigurate numberDigital rights managementIntegrated development environmentDivision (mathematics)Point (geometry)Line (geometry)BitPhysical systemINTEGRALArithmetic meanProxy serverRaw image formatQuicksortCompilation albumCurvatureService (economics)InternetworkingVelocityPersonal identification numberRevision controlComputer animation
19:31
SoftwareBitActive contour modelMereologyQuicksortSelf-organizationUltraviolet photoelectron spectroscopyTelecommunicationSoftware developerCodeVelocityCommitment schemeTunisProcess (computing)Boss CorporationFrequencyRight angleOrder (biology)Machine visionSoftwareMultiplication signObservational studyMobile appExistencePlanningKeyboard shortcutSystem callIntegrated development environmentDigital rights managementArithmetic meanRule of inferenceForm (programming)Direction (geometry)Bridging (networking)Local ringBuildingJava appletWhiteboardSound effectHidden Markov modelMathematicsFilm editingComputer animation
26:56
SpacetimeProcess (computing)Embedded systemShift operatorVideo gameMereologyCollaborationismSoftware developerSelf-organizationSuite (music)QuicksortState of matterRight anglePerspective (visual)Goodness of fitCycle (graph theory)WordTask (computing)TrailMobile WebProjective planeType theoryDifferent (Kate Ryan album)Machine visionCubeWebsiteGame controllerSimilarity (geometry)Physical systemExpected valueMathematical analysisSoftware testingCodeMultiplication signBoundary value problemKey (cryptography)Revision controlIdentifiabilityComputer animation
31:08
Process (computing)Right angleGame controllerSoftwarePoint (geometry)Self-organizationWeb pageChecklistInformationCoefficient of determinationMereologyComponent-based software engineeringSoftware developerMixture modelMorphingIntegrated development environmentInformation securityOperator (mathematics)Physical systemBuildingDifferent (Kate Ryan album)Cartesian coordinate systemExecution unitFigurate numberDigital rights managementPlanningCASE <Informatik>Order (biology)Branch (computer science)Product (business)Arithmetic progressionSoftware testingSheaf (mathematics)Knapsack problemService (economics)Multiplication signView (database)BitPerspective (visual)DatabaseCodeGraph (mathematics)Object (grammar)LoginData structureDuality (mathematics)Formal grammarLoop (music)MappingProjective planeElectronic mailing listState of matterFormal languageLecture/Conference
38:18
Multiplication signEnterprise architectureFocus (optics)Point (geometry)Figurate numberCycle (graph theory)Product (business)Software developerMatching (graph theory)Theory of relativityMeasurementVelocityClosed setGroup actionWebsiteClassical physicsRight angleWave packetTheoryConformal mapDatabaseSoftware testingSemiconductor memoryDampingLine (geometry)LengthBitPhysical systemShape (magazine)Endliche ModelltheorieLattice (order)Latent heatSlide ruleFrequencyMathematicsMetric systemPerspective (visual)InformationFunctional (mathematics)CuboidDirection (geometry)Service (economics)Operator (mathematics)Data recoveryJava appletSystem administratorHydraulic jumpComplete metric spaceContext awarenessBefehlsprozessorVirtual machineNatural numberLecture/Conference
45:28
RootSoftware developerPhysical systemProcess (computing)Rule of inferenceRight angleIntegrated development environmentEntire functionPoint (geometry)Office suiteLevel (video gaming)Multiplication signSingle-precision floating-point formatQuicksortProduct (business)Green's functionSoftware testingServer (computing)Data structureService-oriented architecturePower (physics)Chief information officerCASE <Informatik>Arithmetic progressionGroup actionSoftwareMathematical optimizationRoutingMonster groupBilinear mapPersonal digital assistantExpert systemWeb pageLogical constantLecture/Conference
52:38
JSONXML
Transcript: English(auto-generated)
00:05
Okay, guys, my name is Benjamin Henson. I'm a software engineering manager for APSA Bank. I Look after transformation and software engineering principles within the organization Hey guys, I'm Andrew also an engineering manager at APSA and I mostly focus on infrastructure as a service
00:24
So I'll be taking you through tools and processes Okay, so let me tell you a little bit about where we come from So we are from South Africa Johannesburg It's about a thousand miles away from Cape Town
00:41
We work for a company called APSA Bank, which is a financial institution APSA's been around for about 150 years Okay, it employs about 4,000 people just in this IT division in 2016 APSA as an organization made a headline earning up
01:01
7.5% to about 7.25 billion rand You guys can do the calculation and it's still equivalent to a lot of dollars. Okay, and We know that we live in a very disruptive and innovative world Things are happening Airbnb, Uber, Netflix. I'm sure you guys have heard of a lot of that stuff today
01:24
All right Our saving grace really is that a lot of our startups Fail in fact 75% of them fail They are unable to cross the chasm to do some interesting things. They're unable to hit critical mass, right and
01:44
It's it's a little unfortunate because that means it gives us an opportunity to do some things some some interesting things so Mark Andreessen rightfully said he said software is eating the world and If we do not adapt we will die, right?
02:04
Eric Ries the author of Lean Startup He was pretty much on the same vein. He he wrote a memo to himself. He wrote a letter reminding himself that listen our Tech environment the way things are moving is you know
02:21
It's very difficult if we do not strategize if we do not change the way we do things we will become obsolete Okay. So what did we do? We decided you know what we wanted to go on this journey as well So we came up with three pillars we came up with a an agile transformation team we came up with a
02:42
DevOps and engineering team and we came up with a lean team Some of you right now Might see the anti-pattern emerging over here, but this is part of our journey. This is how we learn anyway So let me tell you a little bit about the DevOps journey or the DevOps transformation
03:01
We were fortunate enough to get a gentleman from chef his name was Jeff Hackett and He came down and he sort of helped us navigate this thing what this journey looks like what we need to do You know the conversations we had the tough conversations we had and if essentially we came up with something quite interesting
03:21
We came up with a DevOps acceleration workshop That meant that We got all the software engineering teams within the bank to come and sit with us for two days And in those two days we sat down and gave them a whole bunch of knowledge about agile about an MVP
03:41
How everything came about we spoke about? Conway's model we spoke about Elsa, you know a large number of material and then we said, okay You guys after two days You're good. Go back into your silos go back into your Babylonian systems and do something awesome come back and do something awesome show us what you've done
04:05
What do you think happened? Nothing right Nothing happened and and rightfully so and it goes back to say, you know, I read a book from Barry O'Reilly, he spoke about
04:20
He was author of lean the lean startup and those are those are quotes in the book that said, you know Good people in perversely designed systems sometimes casually Create great acts of harm on colleagues and sometimes without even realizing it You know Jeff Hackett also goes as far as saying human beings are capable
04:43
Intelligent and whole it's the systems that we design that are not good Okay, we need to we need to think about that for a second we need to think about it because we know That if we throw good people in these systems, they will hurt they will die
05:01
So we took another step back and we said hold on. This is all about creating Software, right? We need to orient around Creating great software. It's about making great software and When we do that, it's about orienting around our customer It's about orienting around tools and processes and touching on culture
05:24
right Everybody still with me? Cool, so we took another step back and we said what does that really mean? You know, what does that really mean? And we said well if it's about the customer was we know that When the customer interfaces with the back the interfacing with some piece of software
05:45
So what we wanted to do is enhance that experience We wanted to make sure that Whatever they did added value to their life Alright, so we came up with a magic quadrant. We call that The magic quadrant and our magic quadrant is DevOps is a union of culture
06:06
Processes and tooling. Okay, and what we did is we take DevOps Lean and agile to continuously deliver customer value and innovation What does that look like? Okay, so
06:20
We had to look at the way work flows into a system We looked at the blockers. We look at the things that we can remove so that we can add value back to our customers And we embedded that with a little bit of engineering practices, right? I'm gonna leave it here for a bit Okay, and I'm gonna hand over to Andrew and he's gonna talk a little bit more about what that looks like
06:43
Okay All right, cool, thanks Ben It's not too loud. Is it? All right, cool. So let me take you through our journey through tools and software So as you can imagine a bank is full of strict controls and processes
07:03
They upon layer over the many years that we have actually been banked and you can actually understand that we very resistant to change But hey, we wanted to create great software, but we had a few challenges to get through first So the first one was that it was very difficult for us to get infrastructure to any teams
07:24
the processes were too slow and This eventually ended up taking way too long for instance Used to take us six weeks to get a machine If you just stop to think about that for a second six weeks to get a machine in a world where we want everything instantaneously
07:43
It's just not good enough Also, our laptop specs wasn't great. So when you're trying to run kitchen or anything it Eventually after two machines this comes to a stop My favorite one is actually internet access so because of all the strict controls in the bank our internet access was extremely restricted
08:05
So if you wanted to do research was very very difficult for everyone to actually do research or download something that they wanted So say they wanted I don't know that pick Ruby mine, for example So you couldn't download it. So Sorry, so you couldn't download it. So what's the next best thing you do you go home you download it you come back
08:25
Right. So then you try and install it anyway. Wait. I need admin rights great All right, so this was very very frustrating so we thought about okay What can we do to change this? So we ended up doing was creating a completely separate an oscillated network called the engineering network
08:45
Now what the engineering network is it's completely separated from our banking network Which unblocks all the things that are previously spoke about this is a place where we really wanted people to create and innovate But the thing that's our fund is important is that with the
09:05
Engineering network you need a place in order to make great software So if engineering name network didn't exist actually wouldn't be here giving you this talk So let me take you through what our processes used to look like
09:21
So we analyzed the flow of work going through the system and then we very quickly realized that Infrastructure was actually our biggest blocker. Let me take you through the wire say that right so a developer goes and Requests infrastructure and once they go through that long tedious process of checklists
09:44
You would then get scheduled for provisioning by an operations team Once that was done. It would then go to another operations team to install those Then what those team would do is attempt to install the application with the scripts that they got from the developers
10:01
In attempt to give them their system but right then something classic happened between The developers and the operations team this constant Back and forth of no-man's land of ownership and it kind of played out like this, right? So you've got a developer Why is my application starting?
10:22
Your application sucks, right so So that's how it played out and The other problem with this is that each one of these points wasn't completely automated so there was a manual kickoff between each stage and
10:41
This caused again further delays, but once you've initially got your infrastructure And shit happens, right? You decide that you need an additional dependency or you forgot to install something or you want to change something and guess what? You need anything else go back to request system follow a very very similar process and
11:01
No one wants to work like it's we suddenly didn't want to So yeah, so this is kind of where our tools journey started and This tools journey this board here is actually very it's specifically about Chef journey itself So we try to do it the right way and run our tools journey like an agile software project on the board here
11:27
You'll both find the both the div and the ops work So we own both so there's where it comes through is that ownership between the two? So now we've got this Engineering network and we've now decided that we've got the tool that we wanted to use which is shit
11:45
So now let me just before I just move on to Y chef Let me talk to you about the problems that we are trying to solve so previously in the banking environment We got stuck trying to create this big silver bullet solution to solve all our problems and created this huge
12:02
Goliath overly complicated pieces of software the other problem was For us to change technologies in the bank say like a monitoring tool like Nagios or whatever we wanted to implement To try and do this on 2,000 servers with a system. That's not completely automated Help, it's horrible
12:21
We also couldn't give people safe access to the chef server. How do we do that when everyone's going there changing things? So why chef? So we chose chef because we felt that it was quite structured had quite a full Feature rich approach to it even though the learning curve at the beginning was quite steep
12:42
We found that for our non developers the operations guys It was we it was better for them to actually do the right thing because it had all the tools and test kitchen and everything that they needed so this was It was so yeah, it was it was quite structured. So with our new network our tools and our processes we
13:05
Decided that sorry Yeah, so we decided to carry on and actually we wanted to automate everything so we were excited right we wanted to get started We didn't want to go through these long processes. So what did we do?
13:21
We opted to go for a standalone chef server Nothing fancy to start we were up and running in a day And this was great. So actually we went mad we automated absolutely everything we could get our hands on it was great But what was not so great is when he started running into problems So it's fine for one team to use chef server
13:43
but For multiple teams using chef server. It's a problem So the kind of things that they would do is they would log on to our chef environment and start changing environments either changing it to a version that didn't exist or removing it or removing one of ours and it created this massive mess for us to
14:02
try and manage and This wasn't great They would also what we realized is that not everyone had a pipeline So the strict controls that we put in place for our own pop on was not necessarily what everyone else had so Then they would upload really bad code. So this is where we decided to go with chef automate
14:22
Now chef automate actually handles all of this for us We don't have to manage environments because it's done So what it allows teams to do is it allows them to write and Upload the chef cookbooks through workflow automatically. We don't have to deal with that anymore
14:43
It's a it brings structure and it also brings structure in terms of chef compliance, which I haven't really touched on much So chef compliance is great because it gives you that kind of upfront feedback So We can kind of be proactive rather than reactive with chef compliance and
15:02
It gave us the tools to automate our infrastructure as code and automating all our tools and applications in a very very modular way Which is great, which is something I'll touch on a little bit later about how easy it is to switch things out But it wasn't all rainbows right like nothing is we sort of running to gain more problems. So
15:21
Now what happened is so because we had adopted chef automate so early on I think this was like last chef comp is when they released It and then we adopted it pretty much immediately after that. So we started running into other issues one I don't know if you guys are familiar with workflow Our bold notes disappeared after an upgrade
15:41
Now so now our pipeline just stopped for everybody and this wasn't great Then we tried to because the engineering network has proxies The slack integration wasn't working because it didn't support proxy service So that was a problem for us
16:01
So, yeah, so we had a few issues with it and the one my favorite one is that With work was actually with push jobs We tried we had to do a lot of bit of version pinning on the Linux OS to actually get it to work properly But there's a bit of a silver lining to the story Tom and Simon from chef some of you may know them helped us work through these problems and but the thing that we
16:26
really appreciated The thing that we really appreciated is that the Features that we wanted in automate they actually put in their backlog
16:41
So now let me take you through our new process So this is what our new process looks like and it should be familiar to you Because it's actually a software flow of work you might be thinking okay, so Was what is this doing an infrastructure? Because that's how we do infrastructure as code. So that's what we do
17:04
So previously I mentioned that about this whole engineering network. We've spoken about chef But the stuff that I didn't tell you about a About our engineering network was that it was built in AWS
17:21
It was also automated completely using chef and this was great for us. So now we've got everything unblocked We can go to the internet. We can now get all the things that we require So what we did is we started a team called reason now a reason is both a team and an application
17:44
It's an in-house product so we started the same VP as a Orchestration tool and it connects to AWS VMware and a couple other things in the bank that we want What we also went and did is we integrated it with chef and it's pretty much an API caller as I think of it
18:03
So now something extraordinary happened now we gave this access to everybody So now product teams can actually go and deploy their own infrastructure with chef in one step We previously it used to take us six weeks to build them a system and now takes us 30 minutes flat and that's fantastic
18:24
That's something that we never had before and that was great. So some of the benefits So what are the benefits of everything I've spoken about up until now? Is that like I said with chef automate it allowed teams to write and upload their own cookbooks through a pipeline But now they can just go to reason and say I want that cookbook and the whole system gets deployed
18:47
We never ever logged on to environments anymore. I mean besides to debug them. Everything was automated Fix the problem through code. We never ever did it manually The other thing with workflow is in greatly increased our velocity because of CR CD pipelines and that was fantastic
19:06
So The one thing that I found and I mean, there's lots of things that we can talk about but I mean It's quite involved so you can catch us a bit later But the one thing I kind of want to leave you guys with is that there is no sort of bullet solution so
19:21
keep it So keep it slim and keep it simple because what it's really about is about adapting to change with little effort and that's the key point So what I'm gonna do now is I'm gonna hand over back to Ben to chat to you about culture Okay, thanks, Andrew Okay, so that's the tooling process
19:44
that was the Process right but in order for this to happen we need to have a really strong culture right and To be able to do something that took us six weeks takes us 30 minutes requires
20:01
Some work okay, and in absolute we believe that Culture is probably one of the most fundamental things in our organization So what is culture right? Well culture. Well culture is shared beliefs and values that are developed over a period of time
20:20
Okay, and that's a probably a Wikipedia definition We also know that the longer the organization has been in existence the more defined the culture is So previously I said that look apps has been around for 150 years Okay, so imagine trying to influence or steer that culture in a different direction. It's a very difficult thing to do
20:41
Okay, another thing is is that employees in the organization? Don't really know the things that bind them together in that organization, right? So if you can't manage the culture the culture will manage you Okay, and you don't want to be managed by a hundred and fifty year old culture It's a very very tough and difficult place to be
21:04
Okay, so before I start talking about reason and the awesome things that they did I want to tell you guys a story The Cobra effect some of you might know the Cobra effect and any British people in here. I apologize in advance. Okay, so Maybe a hundred years ago. Maybe a few decades ago. There was the British Imperialist that
21:26
Went to India to you know to colonize and they had a serious problem with venomous snakes Okay, so They set up an incentive. They went to the local Indian guys and said okay for any snake that you guys kill
21:42
Bring it back to us and we'll give you some money. We'll give you something. So what they did is They went back killed a couple of snakes bought it back. They got some money What do the Indians do they realize that hey, you know, this is a good thing. So what do they do they started breeding? all right, they started breeding these snakes and
22:02
then after a period of time the British guys realized the snakes were getting smaller and smaller and So they said know what? Cut incentive we done the Indians let the snakes go So went back into the city and caused a bit of mayhem But that is the sort of behavior and incentive we don't want to be part of that is a sort of culture
22:25
We don't want to you know We don't want to be part of so what we're saying is that we want to meet you where you are and take you On that journey whatever changes and whatever things we want to do in the organization We want to be there to go on that journey with you Okay, so Andrew alluded to a team called reason the reason it's a very fascinating team
22:48
We started off fairly basic with a you know with a simple Scrum board a bunch of stickies and we said okay. What do we really want to do? Where do we want to go?
23:01
What sort of culture do we want to embed in this organization and we? We said okay. Let's start off with it with an agile process. Let's just use a bit of scrum let's just see what it's like because scrum or agile has some Fundamental values we want to instill in a team so things like courage things like commitment things like
23:23
Communication all of those lovely things we actually embedded in the team and said this is what we want to do These are the things that we're going to do so daily stand-ups You know taught them how to do retrospectives did a bit of planning Teams got better and better, and we said okay guys Check your code in at least one today
23:43
Not to say that they weren't doing that already We say let's take an XP principle and say do it at least once a day What happened was quite fascinating? Okay, so guys started sharing their code people started to communicate a little bit better You know they wanted to sharpen their skills so they their Java
24:04
Development and you know the craft became a little bit better There was a sense of mastery within the team you know so we moved away from the from from the boards that we went digital You know we we said hey guys. You know what we're doing. Okay, at least we every two weeks We're giving something back to our customers because that was the goal
24:23
How about we start automating some of the stuff? How about we doing do some autonomy? You know something cool, so we started automating the work and what what what happened was We saw the team having a little bit more time to do other things You know they said hey, you know I'm not here until 12 o'clock in the morning anymore. I've got time on my hands, right
24:47
So let's start investigating. Let's be creative. Let's investigate tooling Let's see how we can do things a little bit better that created a sense of purpose Okay, and that purpose was bigger than the team they realized that the work that they were doing was actually having an impact in the
25:03
Organization so we we took a step back and say hey something cool is developing here. This is this is nice We we've actually carved out a culture This is the kind of culture we want in an organization And I sort of alluded to what the the mission and the vision was for the organization when we started the transformational journey
25:23
and really the mission was to think and build software like a tech organization Okay, we wanted to build software at high velocity and high quality We had started doing that but in the beginning it was tough. We had to we had to let people go
25:42
We said hey, you know, these are the rules. These are the practices These are the things that we want to do if you guys are not, you know If you don't want to be part of it, we will let you go on your different journey You know, we didn't kick them out of the organization. We said Hmm You know, you don't form into you, you know, you don't fall part of our plan. Just try another team
26:01
Maybe and if you change your mind, and if you see what we're doing and you want to come back by all means You know and and and that's that's what happened. We created a sense of mastery. We created a sense of purpose. We created a sense of autonomy and Something brilliant happened as well. So outages happened right as Andrew said we had some outages we had some problems
26:27
People came in to work together They started doing cool things and we realized that we created an environment that Was failed was easy for them to fail in you know, there was no there was no blaming it was it was it was
26:42
It was a place where people wanted to be so and there's a lot of other things that happen You know, we moved away from KPIs and we went to OKRs and all of that cool stuff Then we we said what do we really want to do as an organization? What what do we want to show? From a cultural perspective because this is the key in
27:02
enabling the organization and we came about Three or four platitudes. Well, they're not platitudes, but three or four things that people may see as a platitude, you know These huge big words that you see, you know when you walk into organization, we don't talk about them. We do them We do them and we show the value afterwards so things like collaboration things like creativity things like ownership
27:25
Right things like delivered value back to you back to your customer Those are the sort of organizational cultures that we want to instill in Apsa right so without without taking any more time or without
27:42
Going on and on and bragging about the cool things that we do This is what I'm gonna leave with you, okay I see some guys are taking pictures. Can I you can get the notes from us if you want? Okay, so What should you do when you go back at your organization what we think is that
28:02
look at How work flows through your system, whatever it is identify what those blockers are look at them and try and remove them and Identify customer value look at what you can do to orient around your customer. Okay
28:21
secondly Define your practices up front, right? What we say is, you know write a failing test Right, if you don't write a failing test, it's not going into prod Okay Define your practices up front let them know, okay
28:43
Like I said your cultural expectations Define it Try and define it try and try and build it from from the ground up, you know, don't You know look we know that Google and Spotify have some rich cultures in their organization and they use that to build a competitive advantage because they have the tested knowledge of
29:06
Their people right and we can't take that and impose it on ourselves, but we know that If we start with our own You know, we could you know, we you know, we'll get better at it and be bold lastly be bold What we have is a practices of mastery website and what we do is we share our successes on this website
29:26
So you guys are more than welcome to go and visit and take a look So what's the future for APSA? Where are we going from here? And the short answer is we don't know We we don't anticipate the outcome we work in a way that we start from building what we know and we we just take it
29:44
Going forward So that is us. So if you guys have any questions, please boundaries Okay. Yes, sir Yes Let me just run over there
30:03
That one yeah, yeah, so the Tracking tasks we actually use JIRA. We've got the entire suite installed I've also got bamboo in there, but we kind of use Jenkins as well. The development tools is quite fast, so that's
30:20
Chef workflow obviously is in that tool set We've got a lot of other projects a lot of different types of development. Some of the guys use Code analysis like sonar cube and things like that Then the version control is obviously a lassian tool again Yeah, we've also got TFS for the Microsoft guys
30:41
Code reviews will be code reviews and see our CD is both Jenkins bamboo and workflow and Yeah, any other So obviously we use in spec for the chef stuff specifically we use our spec in spec
31:00
We also, you know, we reason itself was built in Ruby so we use our spec there again question is How many teams what's the size and are they all? Geographically in one place. There's a follow-up question. Yeah
31:20
What? It's not overnight. It's not flipping the switch, right? So it's a journey along with the journey Okay, so I'll ask your question Ask your last question first. Okay, so we've been on this journey officially for 24 months about 24 months, okay, and our team was
31:42
Widely what we were in the same building, but we weren't co-located So we had a couple of guys from is which was operations We had a couple of guys from there Couple guys and other branches and we said in order for this to be successful guys We need to sit in one room and we can't tell a con So we we you know We found the guys who were able and keen to join the journey said let's sit in this room and let's figure it out
32:05
together And that's what we did We got to a point where we started off at we were seven guys we got to a point where we were 25 in the team and We thought she was you know a stand-up and and a planning session is not taking us two hours to do which is
32:24
You know, it's unreasonable like half the day is gone when we want to you know Want to figure out what we wanted to do. So we said let's break the team up slightly So if there's a billing component we make that The team on its own and we you know, we have a UI component make that team the UI but we ran
32:42
You know, we ran our sprints and in our retrospective Unilaterally, right so we were always in the loop and that's what we did so we were scaling out but we were scaling out not necessarily from a component perspective, but more from a Product or feature perspective. So we said, you know, you know the guys are doing billing and they you know
33:05
They're using a bit of AWS Who knows that that team in itself could be could be a seed company with a nut with a knapsack So that's that's how we looked at it. It's just too sorry just to answer your question Also the testing I forgot cucumber, but because our environment is completely automated
33:23
What we decided to do is you know when you've got development environments and you kind of do something on there manually and you forget about it and then it goes to prod and you like that whoops and What we do now is we actually destroy the whole dev environment every Friday and we rebuild it on Monday So yeah, and yeah, that's it. We must say also so this process all started from from an MVP
33:46
It's it's it's it's still a work in progress. It's it's not a it's not a shiny product that you know We can really say Listening to your To your talk I got the impression that perhaps this was a small team within the overall
34:11
You're not doing the same things
34:24
No, so So It's a that's a fantastic question. So when we started out with our transformational journey, we wanted to do a big-bag approach We said the whole organization everybody, you know, everybody has to transform we got you know, we got the you know
34:41
So the top dogs so to say to Talk down to the silos and say this is how you're going to work and we realized that that didn't work But some of those nuggets of information stayed with various parts of the organization They started doing good things. So you you mentioned compliance What we've done now is that we have completely automated that compliance process. So
35:06
a different team in an organization wants to build something cool and they get to a point where in the bank that you have to Check checklist a whole lot of things So the governance and compliance and it's it could be a five ten page document and sometimes we don't know if it's gonna
35:21
You know if it's gonna get into prod or not But that process has now been automated into Jenkins So upfront before you start any depth, you know what your compliance controls are already and you cater for those things Before you get to the end of the process, right? Yeah, so
35:43
We start us more right like Ben said we didn't go with the bing-bang approach So we looked at like he spoke to our internal customers our business units and said like what's your biggest pain point? They said builds, you know the base build of our system takes long because in the bank We got like 13 different applications that are stacked on top of it
36:00
So we picked that and we took picked hand-picked people from different teams and it was a mixture of developers infrastructure security people you kind of you name it and Ben was there as well. He started them off the culture I did that more technical side of it and we we did it we did it in three months We had redone the whole thing and that's where we proved our value
36:21
And we moved on from there and that kind of morphed into reason and we did billing and then we split the team into a reason And billing component because now they're more involved So these are people from different departments right some is some different business units and
36:41
Because of this culture now we put people together and it's again like I said, it's developer Developers operations and we cross functioning them and they're all learning from each other. So it's great So yeah What kind of challenges did you face with your management layers? Do you want to go first or should you answer that question?
37:01
Yeah, so, um, you know at the beginning a lot of people are resisting to change it when we put these teams together I was like, yeah, I'm gonna lose my job You know, they don't need us anymore, but that's not that's that wasn't the case. What we try to do is we So we created this role this engineering management role that can but that person needs to have
37:21
development experience so he can be He can emphasize with the guys and be a little bit more sympathetic to hey, my code's not working like hey Don't make your problems mine. Just I want my software so you can understand what exactly what's happening and From the other team's point of view is we get them together and we kind of get them working with us
37:42
So now we've got database as a service We got paths as a service and we all working together and seeing where we can help each other in different sections It has taken some time that yeah, and look we when we talk about resistance We you know, we didn't start with the language, you know
38:00
because we felt that we wanted to add the value first and You know, we say we want to do something but the immediate answer is generally no you can't do it It's not gonna work Go and do something else. So we said we just kept quiet and we Did this thing that you said can't be done put a smile on their face and they said okay, how about you try this?
38:24
We did that and I'm like, okay this thing this thing seems like it's working, you know Eventually they came a lot closer to us and they started sitting we try to get them to sit within the teams You know to be you know product owners or you know, whatever so they can give us direct, you know direct
38:41
Direction to you know to the products that they wanted Yeah, you had a question the front so how did you measure the change how did how did you know you were on the right track? Certain metrics that you're using We didn't have specific metrics per se so what we had is, you know
39:04
We got you know, we use various various ways to prioritize the work We got the custom in the room and a lot of our customers were internal customers, you know And you know, you can you can almost think of them as standalone businesses, right and We we get up we get a piece of work we get
39:21
You know, whatever we get we we start to you know, we Resize them we use a Moscow model and we know how large that piece of work is We sit down with we throw that into a sprint we get the teams to start calculating
39:40
Calculating a velocity so we start to say well This is what you know, this is the piece of work of this is a piece of functionality that they want We know over a period of time Maybe after we get a constant cadence. We know that this is what the vent this is what we measuring This is how well we know or this is how big this piece of work. And this is how I know this is how
40:03
This is how we know that the measure is correct by by giving it to them and saying We like it. We don't like it. But if you if you if you You know, if you're asking about measurement, what are you talking specifically about the customer measurement or the value of the work? Yeah, I don't know
40:23
Cultural transformation Engagement I don't know what the right measure is. I mean, you're probably right in the end. It's about are you making customers? Yeah, so with our customers, you know a lot of the time we guess what they want and they say oh, okay
40:43
That's not useful at all. And you're like, whoops. I just wasted six months of development So we've got product owners and also those product owners also have a technical and ops background so we can be a bit You know have a bit of empathy for everybody and that's important rather than people We don't have this kind of shouting match culture anymore. So we look at the customer
41:02
So, you know, what is your biggest pain points? Like I mentioned earlier is that bulls our biggest pain point great We'll focus on that first great What's next and then we keep going in that cycle and because we very agile we kind of set things per quarter So we got four quarters a year and we say cool. We're gonna do this and then great. What's next?
41:20
Oh, we were gonna do this, but we kind of want you to do this. All right, great We'll go and do that. So that's where the kind of value and measure comes from. It's a big negotiation You know, we sit down with with a lot of people in negotiate how the work is going to go But we don't necessarily Put a specific figure to to our work. It's more of an outcome based
41:40
Value any other questions right at the back? We Want to yeah, but like everything like this journey, I mean has taken us what year year and a half 24 months closer. Yeah. Yeah, so we want to use habitat for like one step at a time
42:01
There's lots and lots of things we want to do. We creating our own building engine now as well So currently we look at we're looking at docker containers and we found that Habitat is quite similar obviously a lot lower down But we'll get there eventually when we've got the time and also we can't just do it if we don't have a need to do
42:20
It yet. So once the customer says that they want something like that, we'll definitely do it Sure, gentlemen, the front again. Yeah We're just starting this this this journey and I think people are on board like at 1 p.m Third they're they're fine behavior this way. It's the it's 1 a.m. And your website sale
42:43
How do you get You know if your website's down People's natural reaction is they just want to do what they've always done To get it fixed, right? They're gonna go and specific Machine that's gonna fix it
43:03
To go back to sleep How did you solve that? Okay Yeah, so they are that's classic right like oh my database is slow. So let me add 24 more CPUs and 32 gigs of memory Yeah, it's like it's fine for a while then it crashes So when that happens, obviously if it's an immediate outage we go and figure out what's wrong and like Ben says we rally together
43:28
And we get it done sometimes not necessarily with just the people that are team But for longer performance issues that we have issues we actually go and do start doing More testing and start digging through the system
43:42
So that has happened with reason as well and like a billing engine is actually a fantastic model so we actually bolted originally in Ruby and It got so big we like we chose the wrong tool And it is it took four hours to run a billing engine with two million lines of billing data
44:02
So we changed it to Java and Spock and now we can import data into Mongo DB in 14 minutes with 2 million months So it we have to do that investigation and sometimes you have to rewrite it from scratch One aim Right
44:25
Make it change directly on the system Exactly so modifying the recipe right have ops which is their primary goal is business content, right get
44:41
service recovery So they make the system change directly then next time Back to where it was before Are you talking Hey, so you're saying not specifically from a chef or anything perspective someone jumps on the box manly and changes something
45:01
In the context of just check right in the con primarily so instead of driving those complete changes through chef recipes Someone directly made the change where the admin access to do so with the goal of getting the system back up. So the revenue Right, yeah, so the way we structured is that would happen in dev sure
45:21
That's also why we rebuild our systems because sometimes people do think manually and ends up in prod So we rebuild it every week So very quickly we figure out that that thing's not working and why is it not working then we automate that thing But for when we move from engineering network, which is isolated over to production We actually don't have access to production anymore. We do it in entirely through pipelines
45:44
So we actually push the entire thing right across so we've got Dave and obviously, you know Dave uat and we start removing access at every stage so Dave you have root access so you can figure out all your dependencies Rather than your developers being restricted like I can't install this thing and then with uat
46:02
We actually remove root so we can say to the guys Okay, put in all your sudo rules and all the things that you require so that you So that you can actually Make sure that you've got everything you need so you don't actually need root to function on the server Then we actually push it to production. We generally we don't really have access
46:24
Obviously level one support we do but we start restricting that access very heavily. Yeah So it's quite a different approach It is like that's what you're when you're you know
46:40
Someone tells you sorry Yeah, so we do have access but we we can't get it willy-nilly right so to get root access to our system Is a different structure so we can get in through our reason user for example the bullying user But we actually have to request it from a system called power broker and then we request it and we say okay
47:05
That's a valid request just to stop people logging on top production serve and doing stupid things So I guess the question, which I think is a great question I've heard from multiple people in the conference When you start creating pipeline, how do they go to field?
47:23
Let's say it's very good and it takes you ten minutes to stand up to the environment and test everything So you're out between those hours, 45 minutes Because if you go on a server and fix it Then a couple minutes later it shall come back and reverse it
47:41
Or you have a single point of entry to your pipeline And your outage will go 45 minutes Right? Yeah, so we had we had an issue just like that with reason so we deployed it we like whoops And that happens right? So we actually now using kind of green blue deployments Where we deploy to one half the environment because reason is pretty stateless
48:03
And then we like okay great. It's it's it's cool We've done so we don't just say deployed and say yeah, it's fun to go home We actually jump on and make sure we run through the process just like a customer to make sure everything's working And once we happy then we deploy the other side
48:20
And there's lots of things that you can do like that some we only have two layers some people have three four layers like that like Netflix I have a particular question How is this initial push to change the culture? Where do you think about it? The technology side or the AI side?
48:42
So when you're talking about the push to culture as in the people culture? As an over 100 year old company that does things in a very traditional way At some point someone decided that this is necessary for competitive reasons
49:18
So South Africa's the business environment is quite competitive
49:24
So we we as a bank you know had a banking license for a very long time So now we've realized that retail companies are also getting a banking license The post office is getting a banking license The medical industry people like Discovery Health and AIA they are getting a banking license
49:43
They're becoming competitive and we we're realizing that there's certain things that they can do that We you know traditionally they couldn't do before so we we thought about it And people sat down and said hey you know what if we do not change certain things will happen to us And we don't want to be that company that got eaten by you know a retail company or a health company
50:09
And also I think it was also largely because people were people were feeling the pain You know of certain systems and they wanted a way to to alleviate that pain by doing different things
50:22
So when you talk about a group of people we had a very progressive CIO A very progressive CTO said hey guys we need to change the way we work We need to look at different ways You know sometimes you you get guys who sit on an airplane and open up a flight magazine And they see agile or they see something and they sort of shut down
50:43
And say that's what that's the way we're going to work And you know sometimes that's the case but we took that and we said let's you know let's refine it as we go along So that that was the initial thinking right We we trying to be competitive in a in a in a very unstable economic environment
51:04
And the only way we felt that could work was through changing the way we treat our people Changing the way we we engage with people changing the way we do a lot of things right And that's that's where it originally actually that's where it comes from that's the root of it all
51:23
I remember like I said you know the thing is that you've got to be able to adapt quickly And if you think of a banking system it's it's quite big So you've got to be able to make your systems modular enough so you can change And it took us a long time to get there to figure that out
51:41
And it's that's just what we speaking about is a very small portion of the bank The rest of the bank is very very traditional very very rigid But you know we we sort of cleaning our own house first sort of the the IT guys The you know the 4000 people we tried to change and then we can go back into
52:02
You can go back into insurance and say hey insurance how are you guys doing your things Then we can go back into HR and say hey HR how about this We're not necessarily going to prescribe but we're going to sit with them and say what are your challenges What are your biggest pain points how can we help you Where can we you know where can we find synergies and do something cool
52:22
You know but we first need to sort our house out we need to fix us first before we can go to others I'm getting the signal so catch us outside Thank you very much