Live Long and Refactor

Video in TIB AV-Portal: Live Long and Refactor

Formal Metadata

Live Long and Refactor
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Refactoring major components of a live application with many users can be daunting. The stakes are even higher when the users are paying for your product. This talk covers how to approach building and incrementally deploying a complex refactor. Using a case study, I will walk through what makes major refactors so challenging, what you should avoid, and what can make them easier in the future.
Computer animation Code refactoring
Bit Series (mathematics) Mereology Entire function
Axiom of choice Computer animation Plotter Species Mereology Spacetime
Computer animation Calculation Online game Distortion (mathematics)
Calculation Software developer Figurate number
Web page Code Code Set (mathematics) Shape (magazine) Transformation (genetics) Software maintenance Mathematics Computer animation Computer configuration Series (mathematics) Single-precision floating-point format Code refactoring Codec Code refactoring Right angle Damping Quicksort Contrast (vision) Series (mathematics) Computer architecture
Point (geometry) Software developer Code Connectivity (graph theory) Multiplication sign Set (mathematics) Water vapor Software bug Product (business) Revision control Mathematics Authorization Software testing Code refactoring Series (mathematics) Form (programming) Authentication Scale (map) Software developer Closed set Paradox Bit Basis <Mathematik> Price index Cartesian coordinate system Software maintenance Entire function Product (business) Category of being Process (computing) Computer animation Code refactoring Self-organization Zeno of Elea Right angle Quicksort
Web page Server (computing) Service (economics) Code Multiplication sign Computer-generated imagery Ultraviolet photoelectron spectroscopy Similarity (geometry) Database Variable (mathematics) Product (business) Revision control Single-precision floating-point format Integrated development environment Software testing Code refactoring Computer-assisted translation Backup Position operator Condition number Dependent and independent variables Key (cryptography) Structural load Constructor (object-oriented programming) Database Bit Line (geometry) Group action Control flow Cartesian coordinate system Variable (mathematics) Checklist Human migration Process (computing) Computer animation Integrated development environment Password Revision control Software testing Right angle Key (cryptography) Quicksort Row (database)
Point (geometry) Trail Variety (linguistics) Code Connectivity (graph theory) Mereology Focus (optics) Element (mathematics) Power (physics) Revision control Writing Root Causality Blog Different (Kate Ryan album) Term (mathematics) Internetworking Analogy Program slicing Core dump Authorization Software testing Code refactoring Data structure Physical system Authentication Trail Validity (statistics) Cycle (graph theory) Content (media) Code Plastikkarte Bit Cartesian coordinate system Element (mathematics) Entire function Type theory Root Process (computing) Computer animation Personal digital assistant Phase transition Code refactoring Quicksort Cycle (graph theory)
Web page Connectivity (graph theory) 3 (number) Vertical direction Video game Software testing Complete metric space Quicksort Content (media) Complete metric space
Code Multiplication sign Workstation <Musikinstrument> Mereology Subset Mathematics Different (Kate Ryan album) Phase transition Pattern language Position operator Physical system Social class Software developer Constructor (object-oriented programming) Bit Instance (computer science) Variable (mathematics) Quantum entanglement Process (computing) Angle Phase transition Interface (computing) Code refactoring Software testing Pattern language Quicksort Physical system Web page Functional (mathematics) Existence Connectivity (graph theory) Online help Number Product (business) Revision control Writing Mixture model Goodness of fit String (computer science) Energy level Software testing Code refactoring Task (computing) Authentication Focus (optics) Forcing (mathematics) Schmelze <Betrieb> Interface (computing) Code Authoring system Division (mathematics) Line (geometry) Cartesian coordinate system Software maintenance Computer animation Integrated development environment Personal digital assistant Password Blog Vertex (graph theory) Social class
Software developer Code Multiplication sign Chaos (cosmogony) Bookmark (World Wide Web) Software bug Mathematics Goodness of fit Root Causality Different (Kate Ryan album) Analogy Videoconferencing Software testing Code refactoring System identification Rewriting Physical system Stability theory Arm Software developer Forcing (mathematics) Code Bit Maxima and minima Cartesian coordinate system Root Process (computing) Computer animation Function (mathematics) Revision control Code refactoring Cycle (graph theory) Physical system
Point (geometry) Focus (optics) Constraint (mathematics) Multiplication sign Basis <Mathematik> Cartesian coordinate system Number Revision control Newsletter Different (Kate Ryan album) Vertex (graph theory) Videoconferencing Self-organization Circle Code refactoring Quicksort Pressure Associative property Physical system Condition number
at the
end of the day and you know
I'm the high everybody I am I'm really excited and live in there is to talk about refactoring today but hopefully that will be fun and so he's been yet so I wanted to
start a little bit by talking about these 2 particular episodes that show up on inserted voyager and in the middle I think or the beginning of season 4 and if you're not familiar with voyager and it's with captain Janeway her crew and they are kind of accidentally and up
65 thousand light years away from earth like really come off course and part of the you know the entire series kind of and details their way back home and 1 of these 2 particular episodes they find that
if they take this particular path they can shave off a couple years of their journey back to Earth and add the dual of the research it looks like the you know on the folks that have inhabit the spread of space are generally non-confrontational they think it's going to be OK but but you know it would be called your health you so it's it kind of like a plot twist there and it turns out that actually run into this uh species called the crime and they're actually highly confrontational and I and you know they don't want to allow our voyager to be able to go through this part of space and and I think that the the interesting thing about this and especially but the cranium is that there certain the method of of of of war of choice is that they've actually figured out a way to rewrite history and change the entire timeline to prevent their enemies from ever actually existing and
I think what what's interesting about that even more than just that that the really cool thing that they can do is that and they
always get it wrong should so whenever and they they are trying to
get that this kind of Imperium as they call it back to this particular strict state of of
power and but whenever they do these calculations they think that they've accounted for everything every distortion that they know of they feel really confident they change the timeline and then something always goes wrong we never actually get it right and I think
at 1 point and in between the 2 episodes actually I captain when her crew figure out that they can create this like temporal shield to protect themselves from the crime and that also there is the
calculations off that actually puts their empire like way further behind than they anticipate and I think that the lesson in there for us as software developers on is that sometimes we try to do the entire rewrite there might be things that we don't account for that could lead to some unintended consequences and so I think that's if you get a chance to watch em I think they're on on Hulu on Netflix
and there's the super interesting episodes yeah
so yes so just to kind of get on on the same page about what is an actual refactor verses of full-scale rewrite Michael Fowler 1 of the most kind of common engineers who thinks about refactoring and clean quote a lot has His are definition but it it really boils down to sort of like small architectural changes to the code that aren't changing anything that at year end user can really tell us about but it's putting the codebase in or whatever feature that you're refactoring in a better shape for maintainability of making it easier to test may easier to adapt in the future on but you know this and when I talk about a large-scale refactor today what I'm referring to is a series of these smaller refactors to smaller features in the codebase to you know under kind of some broader goal like umbrella and this contrast obviously without a full-scale rewrite which is really building every single feature from scratch all the uh starting all over and not focusing on just 1 set of features or 1 feature on which would be a factor so and when and why then I guess is every factor of of a maybe a better option for you rather than a full-scale re right I mean if you the Cronin probably don't do that but but there's there's a couple of
reasons and I think I some of the most important things to think about when you're comparing every factor to an entire rewrite is that when you have the existing version of your application in your code based this is after you know you've got a lot of users you got a lot of features is not a time dedicated to this version of the application when you are starting that re right you've got a pretty tremendous gap between version was 1 . 0 and Version 2 . 0 right and back and you know for most businesses organizations that code base is still going to continue to evolve you've got users you've got on maybe new features may be bug fixes that you need to make the entire time I'll while you know a set of developers are maybe tasked with building up version 2 . 0 and our above Martin in his clean-cut series of says that this is sort of a classic example of the Achilles and waters paradox that Zeno talked about many many years ago and what's happening is that because there's that gap and because version 1 . 0 was sort of evolving continuously it's very difficult for the folks that are working on the 2 . 0 version of that application to ever actually catch up so what you're left with effectively I is the question when if ever is version 2 point of the applications are going to be done and then adding another great point that he makes a preferring above 1 is that if you're working with the code base that's difficult to maintain it has a lot of unintended consequences when you change something over here and application and it breaks something somewhere else you you probably don't have really great test coverage or you might have outdated test and 1 thing that makes closing that gap of even harder on top of that is that you probably don't have a requirements document so for the developers were going to be working on that we right they have to spend quite a bit of time and just understanding what are the requirements for this application and that if they're buried in the existing code base it takes time to kind of extract those out of there so that also kind of furthers that gap and makes it even harder to you know how to get to point where version 2 . 0 is actually going to be complete some and there are other things that are important to think about too is that even if you have a requirements document even if you know exactly what are the components that you need to re you know we have to think about the emergent properties that are in the code base so if there's ways that your users the using the application that you as a developer or as a you know product and I might not even be aware of that something to seriously take into consideration because if those are the features that you're on you're paying users really rely on for their jobs so there on you know whatever that they use your application for the depend on those and they're paying you for that product of full-scale rewrite might like accidently get rid of those and that could be a pretty bad user experience for those folks and and I think the other thing to note 2 is that depending on the size of your team sometimes it's just too expensive to save our entire dev team or a big portion of it is gonna work on this full-scale rewrite social we don't know when that's going to be complete when you know you've still got uses that you have to respond to kind of on a day-to-day out basis yeah great so I'm when should we do a lot still refractor and I've talked about this kind of hinted at a little bit but you know sometimes as developers we have to rapidly build out an application of our product get it out to market and that's not easy but easier to do then then also thinking simultaneously about concept like technical debt maintainability test coverage you know is this code in the best possible shape for for being able to adapt and change over time and it sometimes a we end up with is an application that's you know both functional functional and profitable but a code base it's very brittle it's very difficult for the developers to work and so when you have a scenario like that it might make sense to think about but every factor that would help them with the Commissioner more maintainable form and so on yeah so you know going back to the point earlier when you've got an application that has a lot of users you don't wanna risk you know messing anything up and losing them in you know they keep the lights on at the end of the day that's another thing about risk is something to take really seriously and yet when when the developers are not not just scared but a little bit apprehensive about making changes because they always have unintended fallout that's probably another indicator that maybe you need to consider a every factor yeah yeah so if you're at the
point where you know you're maybe going to consider refactor what are some of the steps especially that's going to be something fundamental to the application that you work on how do you actually begin that process and at National Journal for the past year we have been refactoring are entire I user authentication authorization process so I'm just sort of highlight the process that we follow the things that work for us and over the past year and hopefully that something that you're considering to it's helpful the so 1 of the
1st things of that we did was to really focus on but stabilizing the application and what I mean by that is can you on confidently answer that if the application were to go down if you you know all your servers will be white you had nothing would you be able to bring your application back up it seems like depending on where your particular applications art maybe you're in a great position on that anyway but but for us we really had to make sure of debris would answer that question things like can you deploy your application across different environments for this only work on the production service that you currently have on things like is everything in version control sometimes if you have a really chaotic base and that needs to get fixed right away there can be a habit of just as a staging into that server making that hotfix restarting application because it's sewage in you have you can't you know follow likable request process something so things like that can have a really big impact because if the version of the code base at some production looks very different from what's on staging or what's local that can you know before you change a single line of code you really wanna make sure that you know everything is in version control the but some of the stuff might seem obvious but are your or your secret keys here you know API API keys passwords are all those in environment variables on if you're working managing the application are your database migrations in version control and do you have database back ups that are generated regularly so that you're not connecting to prod the to the production database by accident and you know getting dummy data kind of infinite paying users that's really precursory work but it's important to kind of make sure all your ducks in a row before you change even even as a single line of code the yeah
on the next thing that's also another sort of preliminary step is too depending again on your application but if you have really acting intensity don't have any test it makes sense to kind of have some really basic test scaffolding and in DC where National Journal is based we had a cat Capitol building under repairs for quite some time and 1 of the 1st things that we did for that they did during the process was to really build out this construction scaffolding to give a little bit of more support and and coverage further repairs that were to come and I think you can think about your your codebase low and a little bit of a similar sense right the basic tests and what I mean by that is do I get a 200 response does that page load really really simple test thinkin prevent you from having unintended downtime especially if you have on a good a little bit chaotic a little bit volatile on and the thing is if again depending on the team that you're working on is odd there's not really a culture of testing if there's not I'm really if the condition that any kind of shaped even be brought under a test coverage that require a lot of work up front but that's something that as you continue to generate the test but every factor but you also see the return on that investment pretty quickly and and pretty on extensively as well so I would if we recommend that it great so once
you are you know you've got some kind of a little bit more of your application in a stable place you got some basic test scaffolding that the next thing that I recommend doing is really trying to understand what element of the application actually has to get refactored and it's easy sometimes to focus on the ways that the problems manifest especially for the end users in terms of bonds of things that you work on but I really recommend it tracing the problems back down to the root of the root cause as much as possible but it is going to go back to something pretty fundamental and application it's probably going to be scary and that means that you're probably on the right track on you know for us we had a variety of different little edge cases the you know something wrong with the user's permission and not being able to access the content them that they were supposed to a weird but with like a little blip where there was was to be a log but you know something else happened as far as we sort of trace that back down to well where these problems really originating from what's the root cause here Internet to be which is the way that our entire off of authentication and authorization process for structured on a of really great resource to help guide you want is the Mercado mention of really great book in the authors are are up there I would recommend that was super helpful for me this past year and
great so now I wanna dive a little bit more into this entire refactor of process the the so there's a couple points here that I wanted to make the 1st is that when you have cycles broken up into 3 kind of core components the 1st is the factor which includes you writing the test and writing the code for the actual component the next is deploying that's pretty quickly and then the 3rd is sort of validating that human eye validation that everything is sort of working correctly especially as you're test of your test so sweet is still evolving and I think the point that I really wanted to emphasize here is that you want every factor to use a value and and not to systems and so as I love this analogy the cake because it really shows that when only eat cake we don't really start would like the base layer and then the 2nd layer and then 8 but the frosting on top where we really take a slice and when you think about that in terms of user value it's it was super helpful for us to sort of structure work you know in this example it's a user should be able to pay with a credit card or user should be able to pay I with pay power or be able to pay manually and what that means is when you're working on or when you're going to 1 version of that of the 1st phase of that cycle you're doing all the work you that it would require for the user to be able to log right and then you would deployed that component would make sure that that's working and then he would move to OK now the user should be able to log out and then you do all the work that's related to that and the others those are those are fairly simple but you know once you get into other components like OK on a permission-based system a user with a certain permission should be able to access you know this particular type of content without it they shouldn't be able to access it it really helps you know when that part 1 that cycle is done like OK I can confirm that the tests verifying as a human is also verified this this piece is deployed let's move on to the next component and overall following the cycle really helps mitigate the risk that comes even with every factor and you don't want to really be in a place where you work on something 1st 6 7 8 months and then your finally deploying it and find out it doesn't work the something really basic deploying really frequently and to use a value helps to mitigate that risk and you know who
doesn't like take that that's a good way to think about it and and this is like just a quick
I mean a screenshot of like how we were sort of structuring Arctic tickets and get have is what we use for us and so just saying OK with this permission user should be able to access this this is gonna be all the work that kind of tied to that of sometimes test you know features and stuff for us it makes it that really clever ever really be on the same page of when this component is done in its life what should be you know how can we know that it's it's complete yeah and
then talking a little bit more about that refactor phase I think it's really important to and and the vertical stories sort of force you to do that but it's really important to understand but you know what is the acceptance criteria for that feature something like longing logging out obviously that's really obvious user getting around that password they should be able to log but if you've got a feature that maybe somebody worked on a while ago and there's a little bit less institutional knowledge around it a mixture and it's really helpful to make sure everybody is on the same page and they say well when they go to this page and click this button they should be able to download the CSP the CIA station these components getting that on that here kind of crime and and really clear among all the stakeholders is super helpful and that's it's helpful because once you have that it's really clear how you need to sort of right your test but something constructors process that's really important is that you've got current users using the application and you're trying to make it strange for them better in trying to make the string to the developers who work on the code base you know a little bit better to and so the part part 1 of the 1st things for fermions process was making sure that I had test for that existing feature on making sure that as I'm changing the code around as I am you know building out something new and not unintentionally or not intent or the unintentionally causing problems to the existing users so tests for the current and future super-important the next piece on that is writing tests for the new system hopefully those tests will look very similar so once you have that 1st hour test your kind and they're going to be a little bit more higher level you don't want them to be for super specific private methods but really kind of tying into that the user functionality and and and then I would write the code for the you know the refactored code the and then interplay 1 thing that really helped my thinking on this process was a Martin Fowler's blog posts on the string application pattern it's it's a little bit different because he's talking about that in reference to L a completely separate application overtaking existing 1 but in the concept is still applicable to the you know 1 feature that's gonna strangle an existing 1 kind of out of existence and I really would recommend of reading it but it was really helpful for me and you know extraction who on fighting when you know your code as an artist and there's always something from you think about the OK and so
on on his part about writing clean maintainable code and 1 in there and talk a little bit about this too because you know when you're following something like a string of application pattern where you can have a sort of 2 versions of of code for the same feature and as developers 1 of our I'm and biggest instances to make sure that our could is dry that were not repeating themselves but you have to sort of put that instant on hold a little bit at this process because but 1 drying up your code of too quickly in this process can lead to really tight coupling on and what you don't know and what you want at the end of the process is a really clear division between this was the code for the current system this is the code for the new 1 I know exactly what needs to get ripped out when the refractors over if you've got an you know about a method that sort of responding to both systems it can be really hard to kind of entangle which piece was using was being used by which 1 and it'll just can save you a lot of headache at the end so nothing that focusing on not repeat yourself is not important but at this stage in the process might be good to just say we want a little bit of duplication here so we know what we're what we're actually to get rid of found again I think it's I think it's really important to just a slow down in really solve the task at hand and a lot of times the reason why we're doing a refractories because we need to make code more maintainable or we have to get up an application a product out to market really quickly and if that's the case this is really a time to put a little bit of and you know polish on the code make sure that everything from really basic stuff too good environment variables stock strains of a number of a variable names of docstrings are in place but also a little more advanced stuff like are the hidden classes here that can be split up can we introduce an interface that will make this a little bit easier to use I'm adding those there a super important end of yeah I think where where didn't have the opportunity in the beginning to really have focus on Maintainability This is really opportunity to to do that and I think and there are no another really difficult thing to do is to hold the line and scope on and what I mean by that is you know when you're in the code base and you see something that has kind of been about for a long time and you're like I can just take a quick detour n and fix this and then come back to you know to what I was doing or it's is going to be 5 minutes let me just hammered out those d Torres as as distracting as they can be an and tempting to kind of hold off on and they actually will end up anchoring and end up adding up over time to hold along the refactors in take and when you're doing something like an entire oval overhaul of an authentication authorization system but you really wanna make sure that you're keeping your eyes on the prize and really getting better on the refactor done as as quickly as possible to the so called alignments enacting distracted by all the kind of millions of things that you just wish had like a 2nd to fix them and you know but you can have still take it into and you know really the angle is that the code base is going to be in a much better position for you to be able to make those changes at the end of the day so you know keeping that in mind just to focusing on on the yeah it yeah it
arm and this is not to say that this is not to say that writing clean maintainable code is easy it's actually very hard it's a process it's 1 that takes a lot of time it takes a lot of thought and when you used to you know things kind of on fire in a little bit of chaos and I'm working really quickly to like you know him out features it can be weird to search the pick time and on you know and and work on it well but I want to use the tools that are at your disposal there's our engineers and developers who have been working on and thinking about the steps suffered for decades some of my favorites are Uncle Bob Bob Martin and if you've ever watched his videos he is a really hilarious person and he wears a lot of different customs just that the spartans captain kirk he's really great resource and Michael Feathers has a fantastic book working with legacy code and he covers everything from you know I if you like it sort of like a resource book where I wanna change something I don't know how to bring this up under test coverage how do I start that's another great resource and I Martin Fowler's refactoring book is also really fantastic else mention them from method book earlier those are really great the yeah so just to
recap of its application is profitable if it's got a lot of users but it's really difficult as a developer to work on over to maintain or changing something in 1 place causes a problem in other places in your playing Whack-a-Mole you might really want consider of a factor 2 and maker could be so low that more maintainable and when you start to the fact that you want to ensure that your application is unstable stable when making introduce reducible it's got at the bare minimum basic you know smoke test and when you're thinking about what needs to get refractory you really want to not just focus on the way that the different issues are manifesting for the end users of the bugs that you're seeing maybe in your logging system but really looking at what is the root cause of all these different problems and really kind of thing that the force to the trees good and yet try to follow the the refactor deploy cycle as closely as possible to both mitigate risk but then also you know when that and that 1 of the feature is done and try to do it to User to User feature rather than the system remember that sort of like a cake analogy and the agony maintaining 2 versions of of the could recently actually done and not focusing on dry out too early if you have and so with
that go forth and and respected I think few so as to what point would you consider the
blue user you you could basis so terrible would be worth Drupal we're where that I don't think it's a matter of how terrible the codebases is I think it's a matter of whether or not you've got paying users that depend on application and you know for us we serve a lot of different government agencies and organizations associations and doing a full rewrite and the different that come with are just were just not feasible and so even though we were having a number of different problems can of crop up it made sense for us to say we're going refactor we know that the problems are sort of stemming from this 1 place we're going to allocate over time to refactoring that that I think about what makes a good point and in 1 of his videos and he says you just have to sort of face where the problem stemming from because on if you do the re write you might actually not making some of the same mistakes that were existing in the 1st countries to and I thought that was that was a really great point especially fear under that kind of time constrain the pressure to get that 2nd version out just as quickly what is the next the are drawn to take those detours even the system 15 minutes to a half hour so it's easy to see you don't do it do you want from the values as to how best to and to maintain focus now I think and 1 of the things that's that's been my good 2 is that if a node is something to at least find a 5 he ends are creating conditions tissue so that it is you think you need here so that I know that at some point I'll be able to come and go back and yes I wanted to circle back and focus on so not giving too much brainpower to to the things and Michael that's about the 1 thing and then I think I have what I like about the vertical stories of poetry that takes place is that I know exactly the piece that I have to work on so if it is OK you should be able to sign up for this newsletter and that's that's the piece that I'm working on and once that's done maybe 5 times or whatever they on I can come back to the piece that I sort of saw was problematic but can just finding kind of thing opinion it for myself to come back to leaders is going to the kind In this few and
js are