MicroMonolith: Top anti-patterns of distributed systems
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 96 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/51823 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Computing platformProduct (business)NumberTransformation (genetics)Time domainFunctional (mathematics)Complex (psychology)PressureIterationDatabaseProduct (business)Computing platformPhysical systemShared memory1 (number)SynchronizationEnterprise architectureTransformation (genetics)Process (computing)Perfect groupControl flowMultiplication signDifferent (Kate Ryan album)Type theoryDomain nameSoftware developerWeb 2.0Mobile WebINTEGRALBitLatent heatSpacetimeCartesian coordinate systemPressureTwitterMUDPresentation of a groupProjective planeAdaptive behaviorFunctional (mathematics)ConcentricDependent and independent variablesCausalityScaling (geometry)IterationGoogolPower (physics)Visualization (computer graphics)Chemical equationSoftwareLimit (category theory)Decision theoryAxiom of choiceMereologyMathematicsRight angleWeb-DesignerBlogNumberClassical physicsArray data structureVolume (thermodynamics)Position operatorMoving averageMassOnline helpSingle-precision floating-point formatModel theoryIntegrated development environmentWeb servicePoint cloudCoefficient of determinationGoodness of fitWeb applicationMobile appWordFilter <Stochastik>Service (economics)Mathematical optimizationInternet service providerComputer animation
09:34
TelecommunicationData structureSystem programmingPhysical systemMathematicsProcess (computing)DisintegrationNetzwerkschichtKolmogorov complexityInformation securityConnectivity (graph theory)Computer networkMobile appDistribution (mathematics)Cache (computing)ScalabilityProduct (business)Mobile WebWeb pageWeb applicationFluid staticsMereologySpacetimeGateway (telecommunications)SummierbarkeitState of matterCASE <Informatik>Interactive televisionSoftwarePhysical systemCausalityEnterprise architectureMessage passingNatural numberSoftware developerDatabaseRoutingSelf-organizationINTEGRALMobile appWordConnected spaceSoftware maintenanceWeb serviceSheaf (mathematics)Cartesian coordinate systemSynchronizationWeb applicationCuboidPoint (geometry)View (database)Instance (computer science)Web pageData structureBus (computing)Service (economics)Process (computing)Green's functionScalabilityProduct (business)Model theoryMathematicsCodeComplex (psychology)Information securityWeb-DesignerSet (mathematics)Degree (graph theory)Communications protocolPattern languageGroup actionMobile WebTransport Layer SecurityMultiplication signSystem callDebuggerFunctional (mathematics)Fluid staticsHeegaard splittingDifferent (Kate Ryan album)Computer animation
19:04
Message passingScalabilityLogicDatabase transactionIterationData storage deviceService (economics)Model theoryCodeData structureDatabaseSoftware maintenanceProduct (business)Server (computing)Distribution (mathematics)Control flowMessage passingPhysical systemDatabaseINTEGRALData modelLogicComplex (psychology)Normal (geometry)Software developerCodeData storage deviceService (economics)Game controllerMereologyCartesian coordinate systemPattern languageMultiplication signGateway (telecommunications)Lattice (order)BitTable (information)Group actionPoint cloudDifferent (Kate Ryan album)Software maintenanceMultiplicationDatabase transactionTheory of relativityOrder (biology)Product (business)Single-precision floating-point formatPoint (geometry)Characteristic polynomialImplementationCustomer relationship managementEmailServer (computing)Web serviceSet (mathematics)10 (number)System callPlastikkarteBus (computing)MathematicsThread (computing)Relational databaseVariable (mathematics)Traffic reportingState of matterOperator (mathematics)Instance (computer science)Integrated development environmentLevel (video gaming)Streaming mediaComputing platformAnalytic setComputer animation
28:34
World Wide Web ConsortiumTotal S.A.Computer-generated imageryReading (process)Control flowProcess (computing)RoutingPresentation of a groupPattern languageEnterprise architectureWeb serviceProgramming paradigmDifferent (Kate Ryan album)Expert systemEvent horizonPoint cloudSoftware developerSoftware design patternSystem callMultiplication signPhase transitionMereologyCartesian coordinate systemBus (computing)Message passingState of matterComplex (psychology)LogicPhysical systemDatabaseLattice (order)Game theoryTraffic reportingBasis <Mathematik>MultiplicationLine (geometry)SynchronizationMathematicsScaling (geometry)Game controllerINTEGRALInstance (computer science)Product (business)Level (video gaming)Service (economics)Process (computing)Axiom of choiceDatabase transactionRight angleSoftware maintenanceData storage deviceSource codeWordInformation securityLatent heatAnglePerfect groupCurveBitService-oriented architectureSelf-organizationStreaming mediaSampling (statistics)Computer animation
38:04
Physical systemMathematicsPerspective (visual)DatabaseMultiplication signComputing platformPresentation of a groupProduct (business)Traffic reportingRight angleDifferent (Kate Ryan album)Erlang distributionProcess capability indexPublic key certificateService (economics)CASE <Informatik>Instance (computer science)Expert systemProgramming paradigmEvent horizonEnterprise architectureData storage deviceType theoryMereologyGoodness of fitOnline helpPower (physics)Database transactionSoftware developerChief information officerPoint cloudCartesian coordinate systemVolume (thermodynamics)BitComputer animation
Transcript: English(auto-generated)
00:06
It is 11.40, so let's start. I want to bring you on a journey from monolithic-based applications to something distributed to microservices-based applications.
00:22
My name is Mihal. I'm a lead developer at a company called Jeff Giving. I was born in Poland and moved to London two and a half years ago, and you can find my Twitter handle here, and you can check my blog, too, if you want to. You can ask this question, like, microservices, again, aren't we bored about this topic?
00:41
Yes, this presentation will be about microservices, but in a bit different way, because microservices are right now on the rise, and a lot more people right now are looking for them, and you can see this Google Trends chart that people are really looking for it, and people
01:01
So, people are looking for them, but we as a developer, we kind of mostly, I think, concentrate on the positive side of going with distributed systems, going to microservices and things like that, and this talk will be mostly about anti-patterns, and what we have found and what kind of mistakes we have made in the process of going from monolithic non-distributed
01:21
system to distributed one. So, this is my team, and you can see from the happy faces that my company is probably awesome. We haven't forced them to really smile, and those are the people responsible for our product, and we are a good company based in London, we are a social-giving platform, we have millions of users, we have helped to raise more than $4 billion so far, and
01:46
we have helped fund thousands of causes around the world. So we are helping people to raise money for things like disasters, like there was a disaster someone would like to help, or someone is really sick and needs help, so people are raising money for him.
02:01
So we have a platform like that, we are helping people to do that. And the reason I mention that, because we have kind of like a big numbers, and there are a lot of people using our platform, so the move from non-distributed system to the distributed one won't be that easy with such a big scale. And all of this was powered by this, like monolith and one single database.
02:21
And in numbers, we had one Visual Studio solution with 120-plus projects, there were multiple products in there, we had one big database which had a lot of data, our deploy time took us 60 minutes or more, and our release time was like two, three week releases. The platform was so big that we had to spend one week to basically make the system
02:41
stable and find it and make sure that it works still fine. And then we decided to do transformation, and we decided to do transformation because we kind of felt that we are reaching limits of our monolith after eight years of having this monolith in there, we kind of felt that, okay, maybe it's a good time to do something else, try something new, hoping that maybe by going to distributed world
03:04
with microservices approach, we will gain more agility, so this move was a bit also based on hype, but still, in the end, it was really a good idea, and it paid off for us in our environment, in our product. So we did it in a way that we took the monolith and slowly, piece by piece,
03:24
tried to cut it down to smaller pieces and move them to microservices. So you can see monolith is getting smaller, but microservices estate is getting bigger and bigger. In the end, we still do have some part of monolith which are kind of critical that we haven't yet moved, but we are slowly getting there, and we have a lot of
03:43
microservices in there, and this process took us two years, and it's still going, so it's not like finished. It's not yet finished, and right now, it looks like this, so we have microservices, a lot of them, consumers like web applications, mobile applications, APIs that are consuming
04:00
the microservices, databases providing the data, and all the different layers supporting this architecture and everything that is like layers like cloud, messaging, monitoring, logging, and things like that. In numbers, we have hundreds of microservices right now. Our deploy time was cut significantly to five to ten minutes, but that's kind of obvious
04:21
because microservice is supposed to be small, so deployment time of microservice should be smaller. And we kind of embraced polyglot persistence, trying to use different types of databases. It kind of failed in a way, but I'm going to talk about it later. And also, instead of having two to three week releases, we try to have right now daily
04:40
releases, and so far to this day, we probably have date like 400-plus releases so far, so instead of doing two, three week release, right now, we can do release daily, and this also has kind of like freed our developers, made them free, because right now, I'm not afraid to push stuff to production because I can just release stuff as soon as possible,
05:00
and even if something breaks and it's affecting users, I can just push hotfix very, very quickly. So maybe you heard about this, Stan, when you were looking into the microservices world. There is a concept called monolith first, and if you go to Martin Fowler's blog, he is mentioning pretty nicely why probably monolith first is a good idea, and our
05:21
product, our company was a perfect example of monolith first, so we have our monolith eight years of gaining the, finding our problem space, finding what we want to achieve, and then we took this monolith first thing and moved it to microservices. So what we can share today, what can I share today with you?
05:42
Of course, like I mentioned, anti-patterns, and I only mentioned the most important ones. I could do this talk for four to five hours, but sadly, I had to fit everything into one hour, so I will only talk about micro monolith, synchronous integration problem, one DB to rule them all, and top the best one, the underestimate and the cost of this move.
06:02
So let's start with micro monolith. Like I mentioned before, we have this monolith and this database, and we decided to move it on. Ideally, I would love to look at our monolith in that way, like a perfect hexagonal architecture with our application domain inside, all the different adapters, making sure all the different adapters
06:20
providing functionalities and external integration, right? But, in reality, we know how it looks like, and it mostly leads to something we call big bile of mud. Now, I don't like this name because even if this is kind of nasty and it looks bad, it's still kind of functional, right? Because it still provides business to value, it still works, and it still pays our paychecks and everything.
06:43
So we can't make it, it's really not that bad. It has to be functional. It's also a bit beautiful because you have to be an artist, really, to work with a system like that. And just not to be too positive, I'm going to add mess so that you guys won't go for this talk and tell everyone that Michael is telling everyone that big bile of mud is really functional and beautiful.
07:04
So let's just add mess in there. And how it happens that this big bile of mud is there in the first place, and I even think that most of the products will go in that way. It's because the IT industry is complex. Projects are complicated.
07:22
And we do development and exploration at the same time. So we don't have a perfect blueprint or perfect, I don't know, specification of our system that will fit into the market which will help the customers and will be perfect from the beginning. While developing products, we still are exploring our domain, our problem space and what we want to do.
07:42
So this might lead to suboptimal decisions and suboptimal choices. And of course there are also changes. The market can change, the product can change, the specification can change, people can change, everything can basically change. And there's also market pressure, like pressure from the market, which is something I was not liking when I was younger
08:04
because as a junior developer, I was looking as a tech, as a sacred. Tech is always right and tech is always good. But then with more experience, I have found out that tech and our products are built to provide value. So business is also important. And I believe in something like when business and tech, they are fighting together.
08:23
If there is a perfect balance of business and tech having the same political power, we will build something called good enough software that will fix our tech needs and our business needs too. But suddenly sometimes it happens that technical side has less political power and business side has more political power
08:41
and this leads to business basically handing developers and tells them what to do. And this leads to sometimes not perfect solutions, not perfect projects. On the other hand, if technical side has too many power and business side doesn't have much more power the developers will create over-architecture madness. I've seen it everywhere.
09:00
If developers have too many political power and too many influence, they will create over-architectured software. A little break. So our first iteration when we moved the monolith into microservices distributed world looked like this. And you can see that it kind of hasn't changed.
09:22
It looks very similar. And instead of having big ball of mud, right now we have distributed ball of mud. So we had already problems and not perfect solution in the first place but now we also added another problem which is distributed our problems. And that's how I call it like micro monolith
09:40
because it mixes two worlds together. And micro monolith was like a natural step because as developers, when we developed this product for eight years this other monolith for eight years we have kind of acquired prior knowledge and habits on how to do products. So I wasn't really surprised that we have gone that route
10:00
that we have kind of recreated the same thing we created at the beginning and just slowly explored the distributed world. So I wasn't expecting a perfect product from the beginning. Just like with Conway Low when structure of a team doesn't change the code will always stay the same so the code is driven by the organizational structure.
10:21
So if you want to create a distributed system you not only have to embrace the technical side you have to look into the organizational side of the thing. So distributed systems, they require distributed teams too. And changes are difficult. Changes in organization, in culture, in how we work in our habits, in our experience
10:41
this is difficult because the way the brains work they just create those connections inside and if you do this, if you create this connection it's really difficult to just reroute the brain like rewrite the brain. So the only way to do this is to experiment and try new things and even fail. And micro monolith was like a step
11:01
a step, a learning process for us. So we have did it, we have kind of failed and we took the lesson, we found out what's wrong in there and we just applied a new experience new knowledge to create maybe something better. And it's really important to fail fast but learn because failing is pretty simple, you just fail but learning out of it is much more difficult.
11:23
The other pattern that we have found which is tied into micro monolith in our previous experience is synchronous integration. So as a web developer we kind of working in non-distributed world we kind of look at the system from non-distributed world so we kind of have created software
11:41
that was working in one process so we haven't really worried about non-distributed system we didn't have any knowledge about it. So as a web developer because we like HTTP and REST we kind of integrated all of our microservices in that simple way. They were just calling each other and being dependent on each other. And because we were using HTTP
12:01
it created synchronous integration problem. And this is kind of problematic because if one microservice goes down all the dependent microservices will also go down. And it's not only tied into the microservice but also the HTTP protocol the transport protocol, connection everything can be problematic.
12:20
And because we had no experience with distributed systems we kind of underestimated the network layer. Which is a nasty beast because it adds problems like latency complexity, debugging problems because it's not that easy to debug distributed system like non-distributed system when you can just step into the debug and step into the process. With distributed system you have to have
12:42
token that is shared between messages and then based on this token you can find out what's happened with this message going through and passing through different microservices. And there is also security problem and of course connectivity issue that I showed you before. So we kind of like
13:03
forget about the network and you can't beat the network you have to embrace it. And network can be really nasty and you have to assume that there will be problems and you have to prepare your system to be able to cope and fight with those problems. So one way to do this is to introduce asynchronous integration between your services.
13:20
In that way when one service goes down the other services can kind of function on its own not being affected and then you create something called occasionally connected apps or occasionally connected microservices so that they don't have to be connected all the time. They can go offline they can lose connection to itself but they can still work fine.
13:40
So if HTTP is synchronous how can you make HTTP asynchronous? Well you have to add a middleman between the protocol between the two calls. And one of the middleman that you can use is message bus. So instead of tying the microservices to each other you can just tie microservices to message bus and then microservices can push messages
14:00
to the bus and it doesn't care it's not worried about who is listening there. And it's very simple for push. If I change some state, if I want to notify someone I just push a message and that's fine. But what about pool? What about scenario when blue service is dependent on the green service data? What happens
14:21
if green service goes down? Where will the blue service get the data from? So that's where you have to go into distributed data too. Which means like in this case when you have services and red service dependent on the other two, if one of them goes down the red one cannot operate because it needs the data from the gray database.
14:41
So what you can do? Well you have to distribute the data, you have to introduce another database in your red service and kind of keep at the normalized state of your other services in this microservice. So when the other service goes down this database can still function because it's kind of like synchronizing the data
15:01
with the other two. It's keeping the same state, not the same state. It doesn't have to keep the same copy but it can kind of normalize the data it needs and just keep the data it wants, right? So that's how you can create a connection and still maintain connection if services require the other service to function. The other problem
15:21
with synchronous communication that we have found is also related to mobile apps and web apps. So the old approach that we have used in our apps was based on having a monolithic app and if something went down the app the whole application went down. So going to distributed world we kind of also embraced distributed apps so instead of having one big monolithic
15:41
app we kind of split the app into different sections and then if a section that is dependent on some microservices goes down, like this microservice goes down, we can hide this section from the user. The user doesn't have to be even aware that there was a problem. Like an example, my product has some commenting system inside. If a commenting
16:00
microservice goes down, I can just hide the comment box. I can just hide it from the user. A user won't be even aware that there was a problem. You can also apply it in a different way and you can completely disable the feature. You can still show the comment box with old data, with the data that you cached and are keeping some stale data somewhere
16:20
but still you will disable interactions from the user point of view. User can't update, user can't delete comments, but at least he can see the state, the old state, the old comments. And this distributed app thing also provides better scalability because you can scale different parts differently. If you have some critical
16:40
part, you can add more instances in like AWS or Asia. Or you will even increase your reliability because if some of the microservices goes down, you can also still have microservices supporting the other part fine. Bringing it together, those synchronous problems,
17:01
this is a brief history of one app that we have made. We have started with mobile-first web app and it was based on Angular. Later on we moved to React and because it was client-side JavaScript, we had to have some public API. And this public API was just a gateway to microservices. And those microservices were keeping in one big database. So this is our
17:22
first step of what we did with one of the products. And you can see here there will be a problem because if the database goes down, the whole product goes down. So we had to distribute the data and move the data closer to the microservices. So right now microservices, they do own the data. So if one of them goes down, at least two
17:42
microservices can still support some part of the product. You can even go further away with that and add a caching layer into your public API. So with this approach, with this approach, if this goes down, you're still keeping some state, you're still keeping some data in there. And in my case, in my product,
18:00
we do display a static page which doesn't have any interactions. So if he goes to the page, he will at least see something. He won't be able to interact with the page, like add comments or do something about it, but still he will be able to have a static page visible. We even went further and in our critical system, the most important part of the business
18:21
is donations. So making sure that people can donate money to the causes they want to support. So even if our whole architecture goes down, we moved our donation system to a separate system completely that was maintained by different teams and that team supported a higher reliability, higher scalability. So even if everything went down in our product, we could still redirect
18:41
the user to make interaction in an external product. Then we also had to add some external dependency. And just to make things simple at the beginning, we have kind of integrated those dependencies to existing microservices. But this created a problem like those microservices right now are doing too many things at once.
19:02
So we had to introduce another layer of microservices just trying to be like a liaison, a gateway to those different external systems. And later on, other teams have joined in and kind of used our microservices. So we have kind of have right now shared microservices around the teams.
19:21
But to make the system even more reliable, you can see here clearly that this will be a synchronous integration. We also added message bars in between them. So now if this part of the system goes down, our microservices supporting my product, they can still push some stuff, some messages, some notification into the message bars so that when the system is back, the system can just
19:42
reply to the messages or just take the messages and do its own thing. And later on, we also had other integration to analytic system and in here we added EventStream using EventStore. Now EventStore adoption in our company was kind of funny because we kind of understood the event-solving concept.
20:01
We knew that you can't treat EventStream like a normal message bar or something like that. This is basically a database. But still, developers and I also did the same mistake. We kind of used this EventStream and treated it as a message bar. Don't do this. And if Greg is listening, yeah, Greg, we did it. So be pragmatic.
20:20
So you have to approach your application really simple. Start from the simplest possible solution and you have to find out which part of your system are really scalable, needs the most scalability and needs the most reliability. The other thing we have also noticed that is related to the monolith and our old ways of doing
20:41
things is the approach of 1DB to rule them all. So in our old monolithic system, we had the database which was monolithic database and monolithic database has some special characteristics like it has business logic. So after eight years of having a monolith, we have kind of built a business logic inside and I think it is
21:01
natural. We tried to forget about it, we tried to avoid it but still it happens because business needs reports, business needs some logic implemented. You're starting as a developer, then you add business intelligence team like that and they will do stuff on your database. They will create a complex business transaction and things like that.
21:21
So our first approach to still maintain it and still be able to cope with those requirements, existing requirements we have kind of created a system like that. So all the microservices that were dependent on one data and this is kind of problematic because it was a natural step, it felt right at the beginning but then it was a
21:41
single point of failure. If a database went down, we had a problem. It also creates a problem with data storage tie-up bias because if you are tied into one database, you will think about your problems. We have a SQL relational database, so all the problems for us look like relational and SQL-based.
22:01
Of course, your services are tightly coupled and the most important problem is that you have a non-distributed data model which is kind of problematic because as developers, we do sometimes or even most of the time follow this anti-pattern called that data model is affecting our code. This is anti-pattern and we shouldn't be doing that but still
22:21
it feels like most of the teams go into that route and if you want your teams to have distributed code, you have to distribute your data. In order to enhance and influence your team, you have to distribute your data because otherwise they might again fall in the same problem. If one database is problematic and complex
22:41
and it's not perfect, maybe we should do database-per-service. It looks good on paper but it also creates a different set of problems like data aggregation. Having one database with different tables, data aggregation is very simple. You just do a join in a database and that's fine. Doing a join across different databases
23:01
spread around and maybe even having different systems like NoSQL systems, things like that, it's getting really, really complicated. Supporting business transactions, distributed transactions is a complicated thing and this is one of the biggest problems of IT. Also, increase maintenance. This is very problematic because keeping one database
23:22
afloat and stable is much more simpler than having multiple tens of databases and making sure they are fine and they do work and they are stable and they are operational. We also started simple. Instead of going for a full-blown database-per-service distributed data,
23:40
we started with simple. We did baby steps. We still have one database but instead of tying into our own existing data model, we created table-per-service. This is like a first step to distribute the data. You can even go further if you need to support more complicated data model, you can add schema-per-service which is also very useful.
24:03
You have to be careful because monolithic database after eight years creates certain habits that are very painful and it's really difficult to change people and their behaviours, especially different teams, not maybe even tied to developer teams. I had a problem. We had a problem once that my team had owned the table and
24:21
we had the service that was basically owning this table. We had CRM team that is doing marketing actions and they need the data. They needed the data from our product to send some emails to our customers. In the old world, when they had one database, they just checked the database and yeah, there's a table, I will just use it, right? So they kind of tied into
24:41
this table like that and now the problem is they are coupled with this database without even telling us and they kind of are coupled by using the database system which is problematic because if I want to change the data right now or the data model or I want to move the table to maybe different database or I want to move the table to the
25:01
database-per-service approach, it will break their thing and we have this problem. We have changed our database, they basically all the marketing actions haven't went out and they were kind of not happy about it. The best approach is they should ask you about how can I access your data and then you can expose this data through some API.
25:20
So instead of tying yourself into the table and the database, you are exposing the API but it's really difficult to change people's behavior and like, implement them so they will ask you questions like because they used this old approach, they were so used to using the one database, data is there, I can just use it. But they kind of didn't felt that they need to do this, they don't need to
25:42
ask the team about how to access the data. So after this, after you slowly started with table-per-service or schema-per-service you can slowly start introducing things like database-per-service or even a bit of compromise like having a database-per-product which is like a compromise, you still have one database that will affect your
26:01
product but at least you have more control out of it and maybe you can move some business transactions or some data aggregation, also complex reports out of this big monolithic database. And it's also important that you have to, instead of relying on joins inside tables, you have to start doing application joins. So what application joins,
26:21
how it looks like, it's basically just one microservice aggregating the data from other services. Yes, and the last pattern we have found is underestimating the cost. So in the old monolith world we had been renting our servers and this was based on our prior experience. So we had the
26:41
ideas of acquired experience on how to work with rented servers. And we had it under control. So the cost over time chart looked kind of simple. It was linear and we could really always, we always knew where we are heading and what will be the cost, how much we will have to pay for it. And after embracing the distributed
27:01
new world that we weren't really aware of and kind of, we had no experience with. And we went to cloud, to AWS. We kind of had a spike of our cost increasing and then people got worried, like what the hell is happening? How are we going to handle it? How are we going to support it later on? It's not scalable because with more users and with more users going
27:21
into the platform, the cost will increase and increase and increase. So this was the time when we had to introduce spending control. And spending control in the old world was fine because we had an operational team that based on their knowledge and their experience they had been doing this by themselves. But going distributed and exposing the cloud to developers
27:40
we kind of showed the developers AWS and they started doing awesome things there like creating instances out of nowhere. So we were burning money and now we had to move the spending control out of the operational team and move it also and spread it out to the whole team. So we introduced things like planned downtime, which is like having
28:00
three different environments, production, staging, dev. We still keep production safe and find operational the whole day but if developers are sleeping we don't need dev and staging. And it's simple, it won't create a lot of savings but still it sends a clear signal to the team and people around the team that we are treating this seriously
28:20
right now and we are doing things about it. We want to do care about spending and cost control. Right now we are also looking into Docker which is a nice concept but we are just starting up and instead of having one instance holding the microservices like we do right now, so every microservice has four to three instances
28:40
right now, we want to go and have one instance with container that could hold different microservices. Now it won't be probably suitable for production because still in production we want to have ability to scale everything fast and nicely but for dev and staging I think it will be perfect. But the biggest thing that we have
29:00
we are introducing right now and we have introduced so far is that we had to introduce cost monitoring. And because AWS doesn't have any tool to really show you how much things cost we had to build our own tool. So we just built this tool that shows us the increased costs over the week time or the
29:20
whole increase. So this is a chart before we, after we have started monitoring the cost so you don't see this big spike at the beginning. But you can see that it's kind of like linearly increasing slowly over time. And you can have more reports like check which services are the most expensive and things like that. And also we have a report that shows
29:40
us what was the increase by week and weekly basis and which services are increasing the most and which services we need to look at. And this cost monitoring tool was really important for us because we wanted to make the cost control part of our development culture, part of our deployment pipeline. And it's really important because
30:00
if you give developers tools they will start using those tools. Then you have to teach developers how they should use those tools properly. So introducing cost control into development pipeline means that you have meetings and you have some you are basically checking how much does it cost. You're not only checking what's the performance of your application, you're also checking how each service costs, how much
30:22
does it cost. You can even create hackathons and have teams competing with each other trying to cut the cost most. I know that you can game the system and make a micro service that is very expensive and then it's very easy to just cut the cost but you get the idea. So after introducing those things we have kind of
30:40
slowly get back and our costs have decreased a bit and then they stabilized. And if you compare it to the old cost chart it's still bigger and the increase over time will be bigger because with scale it will get bigger because you can see that the line is different.
31:01
But still with increased up-cost we have kind of decreased development time costs. We have gained more agility and we have right now faster time to market with the distributed world because we can deploy really fast and we have daily releases. So still in the end this was very helpful for us.
31:23
So in a summary the road from monolithic application to distributed application should be simple but it's not really that simple, it's complicated. And you will on this route you will probably have we will be derailed by sometimes
31:41
and you will fall into the traps like we have to these traps like micro monolith, one database, synchronous integration and different org changes and things like that. And when you do this it was a learning process and when you also do this probably in your companies because cloud is not going away we will have to embrace distributed world. When you do this you will also have to
32:01
go with some maybe different route and probably you won't fall in the same problems like we have hopefully thanks to this presentation but you will probably find different routes and different ways how to do this so your route will be different completely different. And it's important to share this knowledge so that we will be able to find the perfect route. So it's like
32:22
with the book from Gang of Four, the design patterns they basically gathered the community knowledge in one book and showed everyone how to do patterns and design patterns. So right now we are slowly embracing the distributed world and distributed world becomes mainstream just like docker and containerization have been.
32:40
So distributed world was already there with service oriented architecture but not many companies were doing it because it was complicated. And with microservices I think it's getting into the mass public and more and more developers will do this. So it's really important to share the knowledge and gather all the patterns that are useful and to help everyone to be able to
33:00
move faster from monolithic to distributed application. Any questions?
33:24
Well that's a good question. So the question is if the BI team that we have in our company is so accustomed to the old ways of things and they want to look into the old way and they would like to get access to the database how can we make them happy so they can access this our database
33:40
this my specific database, right? So I think there is no easy answer to it and ideally they should also convince them that this is the new thing, this is the new world and you have to trust us. So we will expose the data that you really need and you have to tie in through ourselves because tying yourself into the database will create more problems so they have to understand it.
34:00
So you can't only change the culture across the development team you have to change the culture of distributed world across the whole organization, the whole team. So I don't know if this answers your question but there is no easy way to do this. Yeah. Can you expand on that?
34:20
So EventStream and EventSourcing is like a database kind of concept when you have events changing the state by state right? And you can then replay the events and you can get the snapshot of the database and this is how you should use it you should use it as a database with some events that are applying state changes so we use it as a message bus and we just basically stored messages like notification message which haven't been
34:41
even changing the state and this is not a good idea because those kind of messages you would like to replay and replaying events in an EventStore an EventSourcing concept is not possible it is possible but it's like it doesn't feel feel good, it doesn't feel right so using EventStore for message bus is a really really good
35:01
is a really really bad idea Does that answer your question? Yeah. You first. You mentioned doing application choice Did you ever see any performance? So there is a question about application application joins making performance problems. Yeah, there will be
35:21
performance problems but with multiple microservices there is a chance that you will be able to scale your microservices much and you can distribute the work kind of nicely, right? So if you even have to do like a complex join you can kind of distribute it it's much more complicated it's not that easy like doing it in a database but it's possible
35:41
but yes there will be an impact on your performance and some of the reports that we have tried to move we weren't able to move them and right now that's why we have still this little part of Monolith because we can't move everything into microservices world. There are still reports and some business logic that needs to be maintained because our business needs to be happy about it so we haven't moved most of the
36:01
things because there was a performance impact on it So the question is if you're exposing
36:26
your data and database access through some through the API not only using the database the question is will the API calls and different API calls expand to like a non-maintainable thing because
36:41
BI developers and SQL developers and people like that they do create complex reports, complex logic in there and sometimes your API won't be suitable to do this but then you have to add another API call and yes it has there was a problem with that and sometimes you have to create you have to work closely with your BI team to really
37:01
find out what you really need because sometimes they just ask questions for some new API endpoints and it leads to and then they just use it one time and they just stop using it later on so yeah but this will increase this will increase the burden so some of the parts you can't move out of your system Yeah
37:41
So how to deal with transactions in distributed microservices So I'm not an expert I don't know the perfect answer to it but I think that instead of working with transactions and looking at it from the old way of looking at things you have to embrace event driven architecture and maybe even eventually consistent event driven architecture and that's
38:01
I think you completely change the paradigm of how you treat the data so instead of looking at the problem from the old perspective you have to change completely the way how you look into transactions So making distributed transactions is complicated and probably there are books and better expert on it I won't be able to tell you how to do this I would
38:21
try to change the paradigm and completely try to move out of it and embrace eventual consistent event driven architecture or something similar Does it answer your question a bit? Ok, any questions? Yeah, sure So when you go to microservices were you able to do you have some services that is absolutely critical
38:41
you have to be sure that doesn't go down and in that case if you can find some of the occasion would you use something a different kind of technology that ensures that like Erlang for instance these things can't
39:01
go down so So the question is if there is a critical path in your system that has to be always online and it's really important would you use different technology like Erlang so I don't have any experience with Erlang I heard good things about it but I had no experience with it but yeah sure like we have our financial system is not distributed it's still being like a monolithic system
39:21
outside because we have to be PCI compliant so this system has to be too close to outside and you need a certificate if you want to access this system so yeah they are part of the system that we still haven't moved on and we will probably not move on because we can't just distribute like our PCI compliance system that was already certified a long long time ago and it won't be possible but yeah looking at
39:41
different technologies will be a good idea but yeah I think you have to do this you have to look at the different problems using different tools so you can't just microservices are not silver bullet and distributed systems are not silver bullet you have to look for different tools if you have different kinds of problems I don't know if that's answering the questions a bit
40:01
yeah okay any questions more yeah sure what type of event store do we use so I know that there is one that you can use using a SQL database and install the event in database but we use event store from Greg Yang which is like an application that you just install and you just
40:20
install it somewhere and we just use event store from Greg Yang company any other questions yeah sure did you face any challenges from the business when you decided to move to one question like how will business how will business like the move yes this will be costly move and this is a
40:42
significant investment and of course business won't like it but we had support from our excellent CIO and CEO and they were aware of the microservices and distributed world they were aware of that we have to move there slowly because the cloud is not going away so we will have to go there one day so with their support they were able
41:00
to communicate with the business and explain to them and always fight for us and always explain them why this is important so if you want to do this thing in your company you have to get support from someone that will support the idea it won't be possible just do it by yourself as one single developer or even as a team you need people higher up supporting the idea so without the CIO and CEO
41:23
and CTO help we won't be able to do this yeah let's answer your question a bit yeah because convincing business to it will be difficult like with everything like you know business they want they need value they need to increase the value they need to make the platform that provides the value they don't want to get into like what's the distributed thing we don't care we want you to develop stuff
41:42
that makes more value so this is when I mentioned this technical side and business side fighting together for resources and time because our technical side has a support from political people that do have a lot of political power we could fight for this idea and this move and in the end later on after some time we have prepared some reports
42:02
and presentations to the business that showed them that with microservices and distributed approach we have better time to market and you will be surprised but they kind of haven't at the beginning they haven't got the idea that this will be a huge thing but now they really appreciate it that when they have a request and they want to change the product we can just deploy something
42:21
in hours instead of waiting two to three weeks for a change right so at the beginning it was difficult but after some time when they've seen the benefits it really got to them and the business is really happy about it and right now they are supporting it fully more questions?
42:41
okay there are not more questions so thank you for joining my presentation don't forget to vote