How Ticketmaster are measuring and managing technical debt
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 133 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/48801 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Software developerArmEnergy levelSoftwareDifferent (Kate Ryan album)Multiplication signShared memoryBitSoftware developerReflection (mathematics)View (database)BlogRight angleGoodness of fitQuicksortBoss CorporationHacker (term)GenderInheritance (object-oriented programming)JSONXMLUMLComputer animation
02:20
Software developerBitPoint (geometry)Chemical equationView (database)Software developerDigital rights managementComputer animation
03:17
Software developerFamilyRight angleCore dumpState of matterCASE <Informatik>Information securityBit rateMachine visionComplete metric spaceStructural loadBuildingLecture/ConferenceMeeting/InterviewComputer animation
04:57
Software developerOpen sourceBitOpen sourceTerm (mathematics)Dependent and independent variablesPhase transitionPoint (geometry)Level (video gaming)SoftwareRight angleDifferent (Kate Ryan album)Projective planeMultiplication signCodeSoftware engineeringView (database)Computer animation
06:20
Software developerPhysical systemProduct (business)ExistenceEntropie <Informationstheorie>Right angleLine (geometry)Multiplication signSoftware developerSoftwarePoint (geometry)Constraint (mathematics)Software testingCodeChemical equationView (database)QuicksortWritingComputer animation
08:16
Software developerForcing (mathematics)Right angleMeeting/Interview
08:45
Surjective functionComputer programCodeSoftware developerFunction (mathematics)FlagForcing (mathematics)Constraint (mathematics)Projective planeRight angleCodeSoftwareComputer animation
09:25
Software developerSoftwareCodeServer (computing)Order (biology)Address spaceGroup actionCodeIntegrated development environmentSoftware testingProcedural programmingReading (process)View (database)Point (geometry)Graph (mathematics)Goodness of fitServer (computing)Right angleSoftware bugProcess (computing)SoftwareOperator (mathematics)Physical systemDeadlockSinc functionComputer animation
11:49
Software developerCodeSoftware testingLine (geometry)NumberCodeMeasurementGoodness of fitComputing platformGoogolServer (computing)Software developerRight angleComputer animation
13:10
Software developerTexture mappingRight angleMeasurementCubeNP-hardMetric systemNumberView (database)Well-formed formulaCodeWebsiteComputer animation
14:40
Software developerComponent-based software engineeringSystem callSoftware maintenanceComputer virusRight angleSoftwareBridging (networking)CodeCartesian coordinate systemTournament (medieval)NumberComputer animation
15:47
Key (cryptography)CodeTerm (mathematics)Software developerComputer architectureComputer architectureSoftware maintenanceCartesian coordinate systemTerm (mathematics)Point (geometry)Component-based software engineeringRight angleOperator (mathematics)Software developerView (database)CodeEndliche ModelltheorieComputer animation
16:33
Data modelSoftware developerInformation securityKolmogorov complexityScalabilityCodeContinuous functionSoftware maintenanceMultiplication signState of matterMeasurementComputing platformStandard deviationContext awarenessIdeal (ethics)Computer architectureInformation securityCodeServer (computing)Right angleCartesian coordinate systemEndliche ModelltheorieResponse time (technology)Complex (psychology)ScalabilityDecision theoryAnalytic continuationWeb 2.0
18:56
Data modelSoftware developerTexture mappingInformation securityComponent-based software engineeringEndliche ModelltheorieSoftware developerNumberPlanningMathematicsView (database)Point (geometry)Electronic mailing listPhysical systemMappingComputing platformDecision theoryDiagramProcess (computing)Virtual machineWebsiteBitQuicksortSoftware maintenanceScaling (geometry)Group actionMeasurementOperator (mathematics)Reading (process)
21:47
Software developerStatisticsStandard deviationCodeGoogolLine (geometry)Computer animation
22:23
Software developerSelf-organizationComputing platformStrategy gameComponent-based software engineeringFigurate numberStructural loadComputer animation
23:01
Self-organizationSoftware developerProjective planeCodeEndliche ModelltheorieSoftware developerGreen's functionRight angleCASE <Informatik>Field (computer science)QuicksortBusiness modelSoftware testingArchaeological field surveyMultiplication signTouchscreenSolid geometryTest-driven development
24:24
Self-organizationSoftware developerWritingPattern languageSingle-precision floating-point formatSelf-organizationMultiplication signClient (computing)Disk read-and-write headBitCore dumpOpen setStructural loadComputing platformService-oriented architectureSoftwarePlastikkartePhysical systemCodeSystem callDean numberConservation law
26:46
Software developerTelecommunicationDataflowInformationDecision theoryKey (cryptography)MereologyDifferent (Kate Ryan album)Decision theoryComputer animation
27:26
Software developerSoftware maintenanceRight angleDecision theoryTraffic reportingComputing platformDot productGreen's functionCategory of beingQuicksortMathematical analysisLine (geometry)Computer animation
28:37
Software developerComputing platformTraffic reporting
29:22
Software developerMonster groupPoint (geometry)Traffic reportingTerm (mathematics)Line (geometry)CodeRight angleSoftwareGoogolAverageSoftware developerMultiplication signSoftware maintenanceSystem callGame controllerEstimator
30:52
Reduction of orderSoftware developerEndliche ModelltheorieComputer programComputer architectureOpen sourceSoftware developerDecision theoryComputing platformMereologyFlow separationCartesian coordinate systemAreaCASE <Informatik>QuicksortCovering spaceMultiplication sign
32:39
EstimationComputerSoftware developerPhysical systemComputing platformSoftware developerComputer programSpacetimeMathematical analysisSystems engineeringMultiplication signPoint (geometry)CASE <Informatik>Product (business)Wave packetComputer architectureMereologyChemical equationFilm editingRight angleShift operatorWeightQuicksortComputer clusterCodeContext awarenessWriting1 (number)Computer animation
Transcript: English(auto-generated)
00:07
All right everybody good, okay, it's the last session on a Friday, so how are the energy levels Yeah, good good good great, okay So you went through the agenda, and you saw this last thing on the Friday, and you thought oh wonder
00:25
What wonder what that's about and the only reason I'm really here is because My my boss said Sort of this time last year That we need to try and crack this problem with technical debt because he'd interviewed about 150 engineers in Ticketmaster who work on you know all kinds of different bits of software and
00:44
apparently one of the biggest things that people moaned about as developers was technical debt and now our leader Of engineering in Ticketmaster is not a technical background so that's quite unusual and he was like I don't know what this thing is so you guys have to come and tell me and
01:02
Being engineers of course we all had like different Ways of talking about or describing technical debt so that was the first question to answer and going through this journey We picked up a lot of interesting stuff, so that's kind of what I want to share with you guys is like Because let's be honest. It's probably the least sexy subject right. It's not like some flashy new technology
01:24
It's something that most people you know think is is desperately uncool, right? And yet you've come along to find out about it, so that's no reflection on you That's maybe because you genuinely have a problem with it. So that's a good place to start Can I just get a feeling for in the room who here is?
01:45
like a developer oh Wow okay brilliant and any architects Yeah, a few architects any any hands off people The one hands off great, okay, so just know kind of what the audience is so I
02:01
Put a blog together on this of what we've done and Very surprisingly it went viral, and we got about 13,000 views of it so It got picked up on hacker news, and if you read through it There's a lot of stuff about the US guys hating Ticketmaster, which apparently is quite common
02:21
But as an engineering professional I I will now say that from my point of view. This is now the single biggest problem that will affect you in your career and It's it's like a debt That's not on a balance sheet anywhere But it can destroy a business and we'll see an example of how technical debt destroyed a business a bit later on
02:44
And what I found like I was I've been an engineering manager for a few years And I just bumped into a guy I used to manage which is quite fun high Chris and Managing teams of developers is really fun because engineers are really you know interesting people
03:01
And I love doing things But what I found from from interviewing a lot of people is that the way this stuff is managed in Companies is often done really really badly There's only a few places that I've come across now having gone into the subject that actually do this stuff well and Anyone remember what this was?
03:25
Lehman Brothers right what happened to Lehman Brothers? They went bankrupt. Yeah, they were destroyed by bad debt right not technical debt, but still debt so We're just going to run through this real fast right so the the core of the problem for Lehman was
03:46
We had all these things about 100% mortgages anybody sign up for 100% mortgage Back in the day or 120% mortgage or whatever they were In any case people were selling especially in the States these kind of loans to people that didn't have like a regular income
04:03
And then these mortgages were sold on by Lehman to These these credit rating agencies would look at these you know mortgage-backed securities And would say you know triple-a stamp even though you know some of those because they've been packaged up were
04:23
You know with people who can actually afford to pay their their mortgages But because all the house prices were going up and up and up you know everyone was like it's all good So what happens is that the pension funds who sell you this vision of your future? That's you in old age, right? Yeah, that's what you're being sold the pension funds can buy these
04:43
mortgage-backed securities Which turned out to be you know a complete load of? I don't want to swear But you get the you get the picture right there wasn't building anything solid it was built on on bad debt Anyone come across Martin Fowler
05:03
yes, of course so Martin Fowler talks about different sources of technical debt and a lot of the time You may you may have heard these phrases while you're you know as a software engineering professional Developing or you know and hopefully with agile. We're getting a bit better in terms of design
05:24
Or you know or doing our best practices right in terms of making sure we've got you know our code properly tested Just in the session with Dan North anybody else in the session with Dan North just now right great guy And you know and he was saying that we really need to take responsibility to make software engineering a profession, right?
05:42
It's it's like any two generations old as a profession. It's a very immature profession And so you know a lot of the stuff that gets built is actually crap right a lot of the stuff that gets built is Crap from from a quality point of view you know and and often what we struggle with is this last one now We know we should have done it right like there's this discovery phase of doing software where?
06:04
You know you don't know what's going to happen because you haven't got a good map of the landscape of where you're going Which is why it's so exciting right you're trying out some new technology to solve some problem No one else has ever solved before and it's only at the end of that project you can say hey This is you know now. We know how we should have done it and so
06:22
You know we we also have these constraints That come from from doing software because we have people shouting at us tell us that they want their features delivered more quickly and If we think about it from the point of view of you know working with a product team the product team
06:41
Probably don't really care too much about The debt that's being created right they care about their features going out on time so that they get their you know return on investment And you know we have to ship to a target The business represented here by these
07:00
attractive people You know we always say we just turned the business so everyone else use that to say the business Right it's like these this sort of mythical people that tell us what to do Now they have no understanding right of the debt in our code base. I'm assuming that that's why you're here
07:20
They don't care about the debt they care about the bottom line, which you know we should care about too because that pays our wages This is an actual picture that was on our wall in Ticketmaster that a Developer put up there because he was so fed up with the amount of debt he was like I was spending 80% of my time writing tests no 20% writing code and there's something really wrong and
07:44
He only lasted three months even though he was like the best developer I've come across because he was so fed up with it and Because that debt is hidden Again, it's not visible to investors right we can't make this visible. It's not on a balance sheet
08:02
We've got every every other debt on a balance sheet financially, but but not technical debt And we can't expect you know our pension companies that are investing in companies You know we don't they don't know they don't know that this debt exists So I said we were going to look at how technical debt killed the company anyone remember this
08:23
Who the company was night capital right 125,000 pounds a second and what this is I really enjoyed Researching this right because what you look at like over months you go on the register And you find out some great article about putting together. You know putting it apart about how it really happened
08:44
And I always find that really fascinating so we're going to just spend a few minutes looking at this because There's there's your forces right of You know just get this stuff done And they have a month and a half To put some code in that's all we need to not really know here that they had a month and a half to
09:03
Get this project out, so there's the constraint straight away on where the quality is going to suffer and This New feature was intended to replace some unused code Which you know was still there, and they had intended to delete the old code you can see where this is going currently and
09:28
the business Made made some quote so so on the day when this actually happened so this software went live or you know with with these new features and suddenly It was starting to rapidly make millions and millions of trades like these micro trades in very very small
09:45
iterations, but The problem was you know it wasn't a test environment This was in a live environment, and what it was doing it wasn't you know placing those trades where they should have been so it was like buying too high or setting too low and You know losing them an insane amount of money so the PR that was coming out of the business
10:04
first it was this I Love it a very large software bug as if you could see the size of the bug No, that's that's how non-technical people view systems, then it became an infrastructure problem. Yeah, blame the ops guys Then more of a networking problem, so you could see that the blame shifting
10:23
And yes, we need to do a better job on our testing environment. Well. Yeah and Obviously the implications So it's a reading through the deep dive This is there during the deployment of the new code one of the technicians did not copy the new code to one of eight
10:41
Servers, so it was missing on one of the servers None of this was reviewed that did the deployment so there's no written procedures So there was no playbook of going through all the steps when you're releasing which obviously you've all got in place, right? This eight server had this defective code right there hadn't been running for years and years and years and
11:02
In the middle of this disaster going on they made it worse by Uninstalling the new code so that that defective code was then replicated across all the other servers so Yeah, that's how to kill a company in what an hour and a half So you know and that's
11:22
From from the point of view of what Dan North is just talking about is you know professionalism is about us doing our jobs And about making sure we've got good DevOps practices in place about making sure that you know That we can do this stuff. Well and not just hack it together in in in
11:42
You know because we're being pressurized by the business right and we'll come on to some of that why that happens Yes, and they're happy retirement Looked at that Okay, so hands up if you know The amount that the number of lines of code roughly in your platforms in your business
12:08
Yeah, and I guess Yeah, that's just don't do 80,000 lines
12:21
Okay, so 80,000 lines of running code. Okay. Okay, but but across your company as a whole would you would you have an idea? 8 million Okay, any anyone know how many Google have got?
12:42
That's kind of out there 2 billion 2 billion lines of code Google have got Okay, how about a number of number of servers you're running One volunteer a number here gone several thousand several hundred You can't say okay, but you know that's good, but what I'm saying is okay
13:04
We a lot of us don't know this stuff right as developers. We don't we don't know this stuff. We don't measure it so Now the thing is that you know the guy said to us he was like I want you to measure technical debt So, okay, you know interesting challenge right interesting challenge, how do we do this?
13:24
So the first place I went to obviously you just go on to Google and say right measure technical debt, right? That's the first thing you do and we came up with this tool sona cube, which is out there and anyone using sona cube Okay. Yeah, it's it's free it pulls together a number of metrics on your code it plugs nicely into github and
13:44
It's really quite easy to install and get a hang of you're gone how
14:05
Okay I'll keep talking and if we'll see how we see where we get to So what I loved about this was it gives you like this dashboard view which includes the number of technical debt days
14:22
Which you can't actually see on this picture So that's going to make you go on to the sona cube site and look for it But it gives you a technical debt percentage But it breaks it down and like it uses a formula based on your code and I was like, this is great You know, we can just get a nice dashboard view give it to the exec and he can we can look at the number of days, okay but
14:41
Sorry, oh, yeah, it's there. Yeah, you're right. Thanks. Great. Yeah, 40,000 days That's quite a lot Don't know how you want to measure that in numbers or in dollars. Yeah, so that's that's why I love this tool
15:01
But you know, so we gave it to him and he said well Is this going to tell me about that issue that happened last week when your virus software was out of date and we had? You know, we had a breach because that component wasn't up-to-date Well, no, that's not going to do that. Yes, and I said, but that's not technical debt either Right technical debt is maintainability of application code. That's what it is. So that's not good enough
15:24
so I kind of realized that That's like engineering here and then there's the business here and there's this gap in between and if he wasn't going to cross the bridge Then I had to so I said, okay fine. We'll call tell technical debt something that includes Upgrading our components even though we'll keep our maintainability because that's what as engineers we know that it is
15:45
so we kind of had to compromise on it and What we learned was Yep maintainability of application code Infrastructure debt Okay, that's a term we started to use for this kind of stuff of like components in our infrastructure throughout a day as in stuff
16:01
That ends, you know as developers We didn't really care about it from a day-to-day point of view But we should care about it because we need to know this stuff is up-to-date. So it's an ops thing Architecture debt now, this is a big one right because it's kind of you anyone heard of the term architecture debt being used One hand, okay. Yeah, it is out there and
16:23
There's no definition though, right? There's no way of Defining it but people are kind of conscious of the of the term So what we did was we ended up After a few months of going going backwards and forwards and having internal engineering conference and debating this stuff out and thing
16:42
What are we going to measure we came up with something that works for us and there are other models of tech debt out This just happened to work for us it's not definitive Some of this stuff we could put out of tools. In fact, all of these things we could put out of tools Code coverage Complexity performance and security and on the infrastructure side. We looked at performance and scalability
17:05
So literally performance on the application side because most of our stuff is web-based We're looking at time to interact just as a measure and on the infrastructure side. We just looked at server response time and Throughput so our ability to scale up
17:23
That's as far as we could go with with automated tooling, but we wanted to measure more stuff as well We wanted an overall maintainability measure of our code and we wanted to be able to look at stuff in our infrastructure like security The state of our DR. So continuity and and a few other things that we've listed there and then on the architecture side
17:46
We kind of bash this out with the EA team and The most important one for them was deviation from reference architecture Okay, so who on their platform has a reference architecture Great. Okay. So what is a reference architecture? Okay, right
18:18
So it's how you would build it if you started today, right? Right. So that's kind of your ideal, right?
18:35
But that's what everybody wants everybody in this room wants. What's now what's ideal? Yeah, you want you want the best?
18:42
So that's that's what we said. Okay. So how far away are we from that? Let's define what ideal is and then we can measure how far we are away from it And those other things helped us as well. So that's that's just what works for us Obviously for your context that we different now
19:00
Great book anybody read this book No, really really great read Well that the first few chapters off after the first year gets quite mathematical and I hate maths So I was like, yeah, I'll take the first bits, but it was really interesting because he analyzed over decades You know how people in business make decisions compared to how you know, we as engineers or you know, more academic, you know
19:24
Guys sort of do that kind of stuff and from a business point of view You're just trying to reduce the uncertainty around a decision So it's not often in engineering we try and make things really precise or really accurate because that's kind of how machines work, but People just kind of want to slightly reduce the risk around something. So that this was the rough principle we went off and
19:47
Kind of came up with this process of manual mapping so we take a component diagram of our system So here's just a simple Component diagram for an e-commerce platform and then what we would do is with the development team is just sit in a room
20:04
and we'd say okay each component just give us a We picked a scale one to five. So where one is low debt five is high debt. Yeah or maintainability Give us a number so in a group Everyone would sit around and they'd on their own write down a number and then everyone, you know
20:21
Like a like a poker planning session come up with okay, four five four five Why is it and then you'd kind of agree on a number and the cool thing about this is that people who didn't know So much about one component would be able to share their knowledge about that component and how well it's structured and so on and then You'd come up with you know a number for for that component to say this is how much debt then you could say
20:43
Okay, we've agreed that number now. What would we need to do to reduce the amount of debt in that component? Yeah, we've it's a before we want to bring it to a to what have we got to do? What work have we got to do? You know go crazy. Just tell us everything, you know, don't don't hold back It doesn't matter how big it is and then we could prioritize that list and say okay if we're going to start today
21:04
What would you work on to? You know reduce that debt what would be the first thing we're going to do so we come up with a prioritized list great So that's the process we've applied for all those things that we couldn't measure So in that model all those things that we couldn't measure with with a tool to get a number we use this process
21:24
So with the ops guys every quarter now we sit around in a room We stick up their infrastructure diagram and we go through all the components and we say okay from a security point of view Where is this out from? Their performance point of view where is this out and we go through that come up with a list of work
21:40
They need doing and then you know We've we've measured it and we figured out what we need to do to to reduce those issues so We measured in a ticker master international. We have five engineering teams across Europe and Canada We've got four and a half million lines of code. So this is roughly where we sit
22:02
You know, that's quite a lot less than Google's two billion So Still a significant amount and What that's helped us to do is to then Compare ourselves to you know industry standards and look at some more of those stats later on But it's it's good to get a handle on how much how much you've gotten how big the problem is
22:26
Okay, so What I realized as I was gathering all this data was sheesh we need like some kind of way of you know a strategy for dealing with this because We could figure out that there's a load of debt, you know in one platform or in a certain component this platform
22:43
But if that component is really stable and doesn't need to change then, you know, we don't care, right? We don't we don't need to change that so we don't care about the debt in it So we had to find figure out some way of saying okay. This is what we really care about This is what we need to change and so we came up or there are these are different strategies for managing debt
23:04
So this was a survey done By on developers as to how much time people spend on technical debt Maybe that reflects your experience So
23:21
Very few spend a lot of time working on debt and Obviously trying to prevent the debt in the first place is really important So what we came up with which I can't show you the similar reasons as the gentleman over there is We've got some greenfield projects and by applying that model and measuring those greenfield projects
23:41
We could start to see the beginning a lot sort of midway through last year that some of the debt was creeping up in The greenfield stuff right and if you think about a startup that's quite often the case a startup Just really needs to get prove that business model get those features out So they can prove that business works and then they go can go back in and deal with you know The debt that they've built up
24:01
But if you can bake those practices in from the start like test-driven development make sure your developers know their solid principles, right? It's amazing how many developers I interviewed not last year the year before I say, okay Tell me about solid principles and they'd be stuck, right? And that's that's people we've already screened from a good CV who couldn't tell me what solid code was
24:21
kind of scary so That's one way of doing it is is make sure it doesn't get in there in the first place Another nice approach is is this that you as you're going through your sprints through your retrospective is just collect You know what's built up in this sprint over these two weeks that we know is debt
24:40
Put it onto a card and then every few months just have a sprint of burning that debt down You know, that's that's a really healthy way of dealing with it. If you know, your your organization's in a good place And Then there's a strangler pattern then in head of the strangler pattern
25:00
Yeah, a couple of hands a few hands so Anybody now, here we go Chris. Sorry to pick on you So I work with Chris at open gi as an insurance company, which which had built software from the sort of mid 70s early 80s, I think 60s Yeah, so and they still had their core platform running didn't they the thing called tripods
25:26
Which and they still had the guy in the corner who was probably what 70-odd? You know just just a guy in the corner working on this core platform And they tried to replace that system three times and three times they'd failed with the Big Bang. Let's rewrite it
25:41
So and that's pretty much been my experience in every single company Ticketmaster years ago Started doing a big bang. Let's rewrite it all they've literally just chucked about two-thirds of that code away So, you know big bangs Tend not to work. So strangler is is is what's great
26:00
Because you can pick out those bits that you want and you can slowly rewrite them as new services. You're not changing the Services to your clients or your customers So Ah Microservices, so who's who's built a microservice. Yay
26:22
So the learning from microservices I believe is Don't make too many Yeah learn from experience Yeah because of the comms into play between Yeah
26:41
Yeah, yeah loads of network traffic and So don't go crazy with them Okay, is this a fair fight It's getting there. Isn't it? It's not one one against so The final part is really about trying to you know, overwhelm the enemy
27:01
like Bring in people from outside of engineering. So this is this is Really I think what made the difference for us was the fact that our VP of tech was not a technical person and he but he did know how to Sell ideas into the exec team to the people who make the big decisions and that's what we really need
27:27
And there was kind of this right? We'd gathered all this data But we only really need to present The you know the right data to the execs to say this is what data you need to be able to make decisions To say let's do some maintenance on this platform. This is what maintenance we need to do
27:43
This is why we need to do it. If you don't do it. This is what's going to happen You know, this is what bad is going to happen. And that's what we put on a report that we created and now This may look fairly dull to you There's a line. It's red at one end. It's green at another
28:02
There's a couple of dots in the middle, but apparently for people who aren't technical that this is very sexy I didn't realize this they love looking at this kind of thing because they can straightaway see where something is at So this is what we did. We got a report produced which
28:20
Basically, you can see straightaway how much debt From all these categories that we put together is in a platform With some text alongside to say this is what we need to do to fix it And if you don't do it, this is what is bad is going to happen Which is the sort of risk analysis? Then we put that into
28:42
One big report Which some people seem to love reading through massive reports But what the kind of cool thing was this rolled up score so basically we could give each platform a score out of a hundred to say this platforms are 60 this platforms are 30 this platforms an 80 and
29:02
So straightaway, it's really visible to the business their business, you know How much debt we're carrying is a business which you know is really helpful and In there as well, we could put a technical backlog to say now we've we understand all the stuff We need to do to reduce our technical debt. There you go. There's a technical backlog
29:22
so No, it's now a fair fight. We've got more people involved in in killing this monster So what happened for us? Because we did this over the course of last year we put out three of these reports we did a bunch of measuring So just a few interesting data points
29:44
This guy called Randall Jensen and when he come across Randall Jensen no, I read a really done this book about software estimation and He basically said that an engineer can handle about twenty five thousand lines of code This isn't you know lines of code per day because that's a stupid metric, right?
30:04
it's You know in terms of an overall code base How much can one engineer really handle before they get completely overwhelmed now worked out that Google with two billion lines of code about? 25,000 engineers they've got about 60 or 80. That's what's our eighty thousand line eighty thousand thousand lines of code for a developer
30:24
Now maybe a bunch that isn't used but you know, it's quite high this guy was saying 25,000 is about right. We're slightly over that but it does mean we can say to the business people Hey, you know if you if you're worried about stuff coming out slowly It's because we're trying to handle this much code compared to an industry average, right?
30:41
So that gives us a lever to say let's fix some of this stuff, right? Let's spend more time doing maintenance and less time on features just until we get this under control And what we measured over last year We could say the end of the year. We've reduced our debt by 10% Great, so purely on how we've measured it
31:03
But that's mostly on the application side. That was most of this stuff. We could fix quite easily We didn't really tackle any of the architecture side of things Which is what the developers are really complaining about, you know, like I've got three data sources instead of one You know just give me one data source And an API and I can test on that, you know make things easy make things simple and
31:23
It's that kind of thing that we're you know, now we're trying to They're gone
31:49
Yes, but not as part of this program. So we've got a separate KPI sort of program which yeah covers meantime to fail meantime to react and so on So yeah, that's in in a separate area
32:03
But you know for anybody else's purposes you could include whatever the hell you want right in your own model It's just a case of measuring stuff and trying to make better to give you know, make better decisions So, you know, it's that good is that bad With you know, we don't know but we can tell a story around it. We personally think we feel it's bad because
32:25
We need to be getting through it quicker. We need to make you know, get that dealt with better and Get it out the way so that you know, we can have more flexible platforms. We can deliver stuff more quickly and
32:42
That's me So any any other questions wait
33:09
Yeah, the waiting okay, do we wait the architecture stuff higher than the others no, we just kept it completely flat To start with but you know thinking about it now We you know arguably you could say definitely we'd want to wait the architecture stuff higher
33:23
But again for us we just feel it's a tool to be able to you know Get more time to work on our platforms instead of doing features So yeah, there's all sorts of things we could do like that
33:48
Removing people personnel Okay, so Okay, so the question is would you can did you look at removing actual personnel? Who are contributing debt to the system?
34:03
Not that I'm aware of but but as part of trying to You know Stop debt getting worse, you know, we do have a separate Basically sort of training program about making sure that we're doing best practices, right? So Yeah, can we fire people because they're writing crap code? No, I wish we could but
34:27
But we're working on you know, training people up to a certain level so that it doesn't get introduced yet. Yes Yeah, yeah, so the question is is there a
35:04
Balance around the cost-benefit analysis of addressing debt against and not addressing it Yeah, and for me that is absolutely one of the most relevant points because you know as engineers you could say yeah We totally need to re-engineer this thing But if if the outcome isn't, you know, if it's only saving time for a team of five developers
35:24
You know ten percent of their time and it's going to cost a year to do it Absolutely, it's it's it's got to be Have a you know, have have have value to doing it and that is a key part of making this work. Yeah Sorry, it's another one there
35:50
Yeah, okay. So the question was did we get pushback from developers who wanted to work on features rather than debt? I Haven't come across obviously these teams are scattered around Europe. So I haven't been to all the teams
36:02
So I haven't come across it What more the case is that because we've got more legacy systems is that the developers would rather be sorting the debt out then Contributing to it by adding more features to an already that written system if that makes sense But I totally appreciate that that could happen
36:21
So I'd probably be out rotating the work around so that no one's stuck on it for too long, I guess Right. So the question is do we do the developers support the code in production? So yeah, we do have a DevOps program
36:43
We've been running for a couple of years We are starting to move to that. So some platforms are better at it than others And we do have automation engineers DevOps engineers who would sort of trying to fill that that space But it's definitely a mind shift trying to get Developers to actually care about their code in production, which but that's yeah
37:03
Thankfully DevOps has come along to give us that awareness, right? Great. Well, have a nice weekend everyone and safe journeys and all that. Hope you had a good week