Split Up! Fighting the Monolith

Video thumbnail (Frame 0) Video thumbnail (Frame 7976) Video thumbnail (Frame 10382) Video thumbnail (Frame 14105) Video thumbnail (Frame 22527) Video thumbnail (Frame 25871)
Video in TIB AV-Portal: Split Up! Fighting the Monolith

Formal Metadata

Split Up! Fighting the Monolith
Title of Series
Part Number
Number of Parts
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Patrick Mühlbauer - Split Up! Fighting the Monolith Having to deal with a monolith, an application which became far to big over the time, can be quite bothersome. On the other hand if you split it up and have to deal with lots of smaller components, you might end up in dependency hell. But not only the splitting of the monolith and the management of the dependencies afterwards can be a problem, but also the packaging of you python components itself. ----- Do you know this situation, where you and your team are facing this big monolith? An application which has grown far too big over the years. Every time when you make a change, you have to fear the code might break at a totally different place, because lots of things are closely intertwined. But what to do if you are at such a point? Maybe you start thinking about microservices but then questions like "Are they really the right thing for us?" and "How do we get there?" arise. In my talk I will show you how we are dealing with our monolith. A collection of multiple python packages without clear boundaries, forming the actual application - all living in a single monorepo. I will talk about how we split up the whole thing, making it more flexible for us and also easier to use individual components by other teams. All this, of course, comes with a price: You have to think more about the dependencies between you components. You have to think about how you can efficiently test everything, making sure your final application is still working correctly. Don't loosing yourself in dependency hell and packaging all components correctly becomes quite a challenge. This talk will: - show you bad patterns to avoid, so that you don't end up in the above situation in the first place - give you ideas what to consider when tackling your monolith - explain how to package your python components and how to mange your dependencies
Point (geometry) Computer file Code Decision theory Multiplication sign Set (mathematics) Water vapor Parameter (computer programming) Mereology Theory Revision control Heegaard splitting Sign (mathematics) Set (mathematics) Software testing Lie group Data structure Computer architecture Standard deviation Software developer Gender Code Unit testing Cartesian coordinate system Computer animation Integrated development environment Repository (publishing) Structural equation modeling Library (computing)
Dependent and independent variables Digital electronics Constraint (mathematics) INTEGRAL Building Interior (topology) Branch (computer science) Branch (computer science) Cartesian coordinate system Heegaard splitting Mathematics Process (computing) Computer animation Commitment scheme Repository (publishing) Chain Repository (publishing) Process (computing) Library (computing) Library (computing)
Commutative property Installation art Code Direction (geometry) Multiplication sign Execution unit Workstation <Musikinstrument> Set (mathematics) Insertion loss Parameter (computer programming) Mathematics Computer configuration Semiconductor memory Process (computing) Library (computing) Pressure Exception handling Identity management Software developer Complex (psychology) Unit testing Entire function Connected space Process (computing) Vector space Repository (publishing) Order (biology) Software testing Quicksort Data structure Point (geometry) Cue sports Laptop Server (computing) Personal identification number Service (economics) Connectivity (graph theory) 3 (number) Continuous integration Revision control Software testing Data structure Metropolitan area network Execution unit Dependent and independent variables Server (computing) Code Core dump Cartesian coordinate system Error message Computer animation Network topology Library (computing)
Complex (psychology) Sheaf (mathematics) Branch (computer science) Product (business) Element (mathematics) Software bug Revision control Pi Different (Kate Ryan album) Forest Software testing Data structure Personal identification number Module (mathematics) Graph (mathematics) Weight Projective plane Feedback Data storage device Cartesian coordinate system Process (computing) Computer animation Repository (publishing) Personal digital assistant Right angle Cycle (graph theory) Metric system Resultant Library (computing)
so what did you tell which is the topic of Patrick move or release rain how to
split monitored on on application where the repository and the rest of so that like he thinks that the I welcome to my talk and Patrick and working as a software developer younger and if you came here because you only read the title of the abstract and like I did sometimes and I'm no expecting it will show to that's a monolithic application into a micro-services architecture and I have to disappoint you that's not what I'm backing of water are we want to talk about it read it with all application which was which consists of various Python packages and we had all Python packages and 1 repository and at 1 point he decided to everyone to that's 1 repository for every thank and we developed and yeah so let's start with you the example how upon packages should usually looks like you have for this structure have the actual package my library with this standard in the module and and maybe you have requirements file you probably have a set of pi and of course you will have test and and for us that looked like something like this so we had lots of these packages environment repository um what we also have what's this 1 requirements file for the whole application and this was probably the robust design decisions at the beginning so even if all the unit tests of theory of these libraries and we chat and thought everything and use that another version to run the test on we had actually a lot more than for of these packages and there's some yeah so why do you want to live developed various reasons 1 of the reasons for other teams of all companies started using some of the libraries and their own projects and they always complained about OK we want to contribute but I every time I have to look into this big repository it's such a pain gender it's just running whatever so you have this was 1 reason and another reason was that I called spaghetti code you that's this what I mean you crossed dependencies so you for example library 1 you import from lie rituals even if you don't like you wanted at the end because would not well structured um the year so as part of a team started to use all libraries for you I also have to release them and that's also use yesterday halftone repository and we also wanted to 3 you with something like that set of tools SEM to get automatic rationing so if you don't know what set to assess in it creates versions for a Python package although you you get Russian so macular versions also usually is that the isotope I file look something like this and you have this Russian keywords arguments that you have to manually adapts the Russians string every time you want to do when you release and set up to will discuss this for you automatically so you would just say yeah my set of requirements to so that was assumed to be available and I want to use this as the emergence and what such a reference that looks like this something like that so here we have 1 committee after the latest tech so before I said the text is 0 . 0 . 1 and this this 1 then says OK you are 1 commit after the latest 1 and at the end you have laughter laughs and at the start of today's accountant from OK so talking about get contentious so sign
outside and we want to split Our monolithic git repository M. but how do we do it if you just moved certain sub-packages some and then initialized the repository you from the start you would use all the history and the already have but there's a few users get I don't know if there's something in some of the material any other system but with this gets you have such a subtree in the another subcommand split and that's creates and new world history of commits for specified prefix so it uh half of as previously only 3 and if you also specify a branch name it we create a new branch which exactly has newly-generated history what you can then do this tritium new package and initializes the circuit and then you can put this branch you created before from the 1 Monosiga repository annual repository for you library new and like is 3 and with all the history the effective the library 3 so
this has to be done for all packages known as a good repository what then changed for us is how do we do in integration of fuel because before we had just this 1 repository of it didn't matter in which the package we made changes for next feature use Jenkins thank it's just checked ought to let commit every change in every of all packages the available and and was used that some of the responses that you have a monolithic applications that picture of our what we tended this OK we know we have lots of metropolitan so let's just cheque of every get repository at the beginning and create application artifact of this this ended up in a really messy chains jobs so if you don't know Jenkins constraints has lots of glycans 1 of those appliances the multiple bless like and then you can specify you know what it was repositories which should be checked out at the beginning of a job and yeah because this fall of all extract libraries and had a huge added this was horrible it was really horrible when we have to do back 6 reduces I mean you can imagine how the heart of this if you have to configure all it this because of the things you have to specify text branch or whatever has to be checked out so if you want to know what to do about things really we had to specify the text or commit edges which will you back of fall release so that we only M. changes the repository that we had to do but release and I think you can imagine that this this was really horrible so don't do that and
what you actually want to do a of courses just use your libraries in your application just like any other library and the problem that all this you will at your libraries to your application's requirements states the order every time you change something you would have to race depression in Europe I'm size because you in your requirements of course and yeah that's also and that's a good directional it happens quite often that implemented new features in all library packages then to the unit tests for this library past then this sort of carry can do released but when we actually used the new version and all applications is all note that not working at all and you want to have the if you make a lot fossils so what can we do do here um we came up with this workflow so run unit tests of all libraries if they passed the lecture and of to really real to our entire identity self and at the beginning of all applications a judge you just installed adults the minus minus tree option of the installed With this option you can installed pre-releases and so beta releases reduces all of those that releases you saw earlier created by so of as so we always had newest the Russians and continuous integration pipeline um also all libraries memories knew what do you need at least a unit has passed yeah and then we created another chalk for actually doing application reduces if you wanted to a release of applications we back to the you know have to do releases of all the libraries where know OK with this version of works and can then run the peaks from release job that this man's pre option is not used I mentioned test P and cool you knows that people use that In many hands and so that the server the the arguments over we use this in agree on the following tunnel packages and you but you can also use on your laptops so it's also just a mural for PI and for example if you on the train and want to hack but but don't have the connections and have installed and sort of package you could do it offline if it's already cash yeah you can whitelist vectors and to a loss of more things another thing is that to the requirements beginning and tools you will however this version conflict exceptions my colleagues and a few others the so what we agreed on research we don't pin the requirements in the set of 5 so you have the this and so require I because this for example like requires requests created no 1 . 0 and smaller than 2 . 0 and another library wants to use the next feature which came with and request 2 . 0 for example you get annoying Russian conflict over and most often does not make sensor even giving this Bain OK I want to have smaller than 2 points and yet the applications that responsibility you stick erect requirements and this will point lots of these exceptions and 1 other comments so we we don't kinda requirements and the set of 5 5 but the kingdom to have a special set of requirements for running the test so and another developer can then check text 5 . 0 0 for example and see you with which requirements the tests actually passed vector the OK so now we have
all those repositories what did we actually data so we can use that up to assess and repository don't have had lab contributed but on the other hand for us at the station you what the local developers that got a lot more complex all you if you develop a new feature sometimes you have to to changes 3 4 or 5 repositories you have to keep them updated on time and that can be really annoying sometimes but I I think I already and the structure of code and really improved so now is define the requirements which every library just as minimal set so it does not happen that you see that you have this ugly across imports that you don't actually want and so on and that's actually abhorrent and knowledge you have 15 structure it might be easier to see OK components and change in the same the same speed also Baghdad that and where might be an extra the man might be good to introduce a new service but only that library packages so maybe that's all I think what we think of the novel is step before actually getting to model services yeah and I think that's called and what if if OK so thank you
for to and his some questions for an old friend of so thank you for a job that was a nice experience but we we have what pretty the same problems in our project and so you you mentioned that during during the deployment you Jake his job um yeah did deployment is a simple deep style and it always install the latest version right so I mean it is the weights is the importance forest CI and the latest stable versions of for the production of these what you deploy some broken package to the production and you need to roll back to the previous version how do you do with that so is I don't think you've been versions you during the neural tube so how the dual role but to the previous version and not all of them that doesn't happen so be Don Finley what's that libraries want that um opinion on applications and we also then installed in the minds months snow depth so that we don't get any because we requirements I'm not sure if I understood your question correctly the so do you being versions of library somewhere during the production of wine yes of course so and the like that for in the libraries don't pin the in in storage sections of pie but we can all we have an extra requirements very primitive versions and we used this versions to run the tests for the last and an obligation we then in we want library displaced by 5 . 0 and so it cannot happen that something else comes in today we can talk later maybe thank you mentioned that you'll have that that the dependencies between the library so library 18 courses library the and the it to the so how would be the result that depends on whole it that depends sometimes um called deprecation of better than having this crossed dependencies you know that that's basically it's only depends on how whole reference looks like this I don't have a special example right no sorry they become a the depending on the 1 library on the other so sold for for Bolivar you you require the this that sometimes it's not a problem at all we noticed that for some In some cases the um had libraries each depended on our library and this 1 again required lots of other and they had a again and this big and structure things and the yeah I don't know the of this I did you consider using did submodules for your problem like for them to be pointing versions and being able to have a different kind of anomalies that just means that from the beginning we don't want you discuss modules no really didn't you uh just not that 2 containers to get so fucked up the book because we are going to now that's the way how we can employ different versions like the 1 you what fixed branches on elements of a repository this only data that that works also you a greater out any the hi how you mentioned something about the good quality like by following this feedback to constructed you want some improvement in the good quality and usually cycle you measure that you measure some graph cyclomatic complexity of something it's like what we're metrics to to see that your quality was improved I have to admit that just feeling I had and know and that came so it's difficult to explain maybe can talk after that no 1 knows OK thank you so much