Merken

Arrested Development - surviving the awkward adolescence of a microservices-based application

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
great thanks for joining everybody has so far today we're going be talking about my team we and the product that we build and basically how we bought into the promise of micro-services and admittedly maybe a little bit at the height of within the Pentagon going over the lessons we've learned over the 1st year 2 of working on that and tools for what you what you can do how you can approach problems after the shiny coating paint wears off and you left the remaining so 1st of legal background and make
sure you know what notice so Yelp if you're not aware it is an act or if you're feeling a little old-fashioned the website where
you can go and basically figure out what local businesses near you are really good at what they do and it's
not a small number of local businesses so we have about 70 million reviews and 140 million monthly unique users so really a lot of content a lot of depth and as well as that we've also got a fair bit of breath so most of you might associate Yelp with restaurant reviews and that is a very popular use case but we actually have a lot of other businesses on Yelp a lot of other contents to look at so shopping is a large category but we have a fairly widespread and in general our goal is to make sure that if you can find a business on the street and has a storefront we wanna make sure it's on you know things can find that information and find out of good things about so for
myself I'm Scott if you're inclined to use Twitter you can find me it's actually on there I've been at Yale for 4 years now working on back end systems spend a lot of that time on the search team but have worked on a variety of different applications of machine learning a lot of work with locations and geocoding and I've also still a lot of time bowls developing infrastructure for a service stack delta and being a consumer that service stack but the
team mean be talking to you about today is called the transaction platform team and it corresponds directly to a product that provides called the transaction platform for this talk through a little bit of what that means and in general if you hear me referring to just platform this is what I'm talking about alright so what is
platform and our goal in short is to make sure that for all of those great businesses that we have on you know as much as possible we can actually connect users directly to their goods or their services so if you're at home hungry after a long day and you're looking for some delivery you might search I to figure out which places are going to go and if you decide that what you're really in the mood for his crispy tacos and you go to that business we would give you the opportunity with the platform product in the platform team to go ahead and just click and get whatever that good or service that makes crispy tacos provides In this case hopefully next crispy tacos is going to be giving you something like a parasite taco or maybe so much it's in guacamole whatever is appropriate a however we
don't only have places that serve tacos on your so of course we also partner with a number of other 3rd parties to provide other types of goods and services so you can find some closing partner which optics or you can find a hotel booking we partner with it as well as a number of other partners and different types of products for articles and then the
goal is regardless you can go ahead and check out on Yelp pay with some stored information so you don't have to you know be re-entering and on 18 different websites in establishing all those different logins and get that thing you're looking for in the 1st place so that's a little bit about the
team and the only the only other point of order is making sure we're all on the same term micro-services I'd just a quick show of hands has anybody in this room not been to least 1 talk with the word micro-services involved this week as can whatever so you have to say it's a hot trend might be a bit understating it's pretty recent by I at least in this exact term if not the general concepts but honestly it's hard to avoid at this point you're probably familiar with that
said let's go over basic terms and nobody is a wildly off base so I like Maarten Valerie's definition and you can read words yourself but I want just point out a couple of them that I find particularly insightful so he focuses on mica services as being a suite of small services so you can have a few of them and you're hopefully aiming for them not to grow out of control he also wants them to be in their own processes and running communicating with lightly mechanisms so in our case it know this is things like HTTP J. signed a separate web servers and things like that so this is
mostly by contrast to model and this is sort of the traditional term for the single at all of your code in 1 place all of it being deployed together atomically and working from then I would say if you ever hear the word model with you hear people talking about their model with 3 or monolithic anything like this I would really encourage you to clarify what kind of monolithic they're talking about there's it's a very generic term for a lot of different kinds of systems if I do a little hobby you know rails out that's technically model and in jobs case if we have a million lines of code in a single codebase that's also model and there's some slight qualitative differences between those 2 so in your case are model with was big but like I said million lines of code million lines to test out we had hundreds of developers working on and you can imagine that this sort of problems you have when it comes to model of that size and amount of code of that size that much cooperation in a single codebase things get a little messy and that's basically what we signed up the middle what led us to go to a service-based action so what is
what is bad about these models and only just assume this is a given and it's a few brief points and then we'll move on on the assumption you've already heard this Putin's 1st off 1 big problem we had was monolithic Python code was really resisting the coupling we all know that we would like different parts of our application to stay in different parts of our application to not be tied together accidentally or on purpose but unfortunately despite the best efforts trust me out we found this was extremely hard and basically completely unrealistic to attain in a single codebase you might be objecting you might be saying well I know you start I about all these things OK interfaces and all sorts of great tools you can use to decouple your codebase trust me we thought of it that we in fact even thought we didn't need to go to micro-services at a service-based architecture at 1st and we basically found through a very expensive series of tests that we actually so it is surprising how much it resists de-coupling and best efforts in that direction In addition it has the bad habit of catering to the lowest common denominator so we had the case in our model it's because domain where there were a number of old libraries here we've been writing code for a decade in this application it's it's got some each to it and so you might imagine occasional upgrade needs happen to something like sequel alchemy or what have you and it turns out that the oldest and scariest and darkest corners of your codebase that nobody likes and nobody wants to talk about and certainly nobody wants to change are exactly the things that are going to hold back the parts that we actually need to move quickly so we found that we were being limited on libraries we're being limited on choice of languages but wouldn't it be nice to run in Python 3 well it turns out when you have to port million lines of code to do that but it's a significant undertaking in your being held back by all these parts of the code before you don't really want to be held back by and finally monolithic Python code is really the opposite of being agile uh we wanted to be pushing multiple times of data production we wanted to be running test quickly and then we we found ourselves doing neither of those things and moving in the opposite direction that you would want to be as far as whether those are getting better or worse and basically this was what really drove us to safe it's fine it's time so we started
doing services circa 2011 actually about 100 companies and this is just the cumulative graphic number of services over time you can basically my whole point here is it started pretty rapidly it continued rapidly and it hasn't stopped anytime soon this includes greenfield development this includes slicing pieces off of the model is and pulling out in services in this this fields or something like this this is this especially the 1st part of this graph is an amazing part of of sort of the best feelings of micro-services the best benefits all the things you see in a nice slide that when somebody's presenting why you should to this architecture that we feel in the 1st you know 6 months to a year for this and it was amazing fast food pushes actually having isolated systems and being free of these ancient dependencies that we care about
so my team 1 we've build our system said man that sounds like a pretty so we feel that we can have this horrible old creep the model with the idea that has been proven to be really painful and terrible working of this really nice new set of micro-services everybody's talking about it but you know that way it sounds kind of cool I like to write cool things so really body and this we had this idea of what it was going to be when we finished and it sounded pretty excellent so it was usable but surprisingly little bit concerning upsetting when we actually build that scale that out for a year and look back and and realize you know that thing that we build what we actually came up with it didn't really look a
lot like what we had been promised by liking Kennedy see where we were in the right direction we had like the rough concept uh you can even recognize individual features media pushes that slightly faster or maybe we had some some good luck with getting some isolation but like the whole picture having it all work together exactly as we were told that was not happening right now is pretty frustrating so will about outline that it's
a very briefly we saw a lot of problems as we grew grew larger and really got embedded in this this service-based architecture are API complexity increases so those nice isolated services that made sense and that 1 thing well started to do a few things kind of OK which is a lawless compelling it turns out we please perversely self-coupling right straight reason we do this in the 1st place was because I was tired of being coupled something I didn't care about but it turns out it's kind of really hard to build a product continued to evolve your product organically and not accidently couple yourself back to everything that you try to get away from the interactions between the systems get really murky and what I call a function I can read that function can begin to its contents it's running in my process I have some guarantees about what it's about to do but when I call an external API 3rd parties even worse who knows what it's going write it being able to say confidently what it does or does not do turns out to be extremely hard and finally all those processes that we talked about pushing deploying testing being confident that things are working in production but none of what we're used to doing with scaling up to the system of services and that her in fact that was actually exactly the opposite of what we're trying to do this whole idea so I can't brings to the topic of this talk which is great I bought into the hype right I want and what I was when I was told was going be awesome and I was convinced that it was gonna be awesome I implemented moral as told and a year later I'm finding myself left at the exact same problems why is my code coupling 2 things I don't care about again why my test that's that was the entire point of this thing right why can't I deploy regularly every day and so I want to give you some some hard-won tools that we've had some that we build some that we've learned to sort of attack these problems and to be able to leave you with the good parts of services and hopefully be able to work through an improved that part so do that we can talk about 4 main areas curiously you might know that all of these areas are kind of in the below list of benefits of services in the 1st place and it turns out there's a really nice initial spike when you move the services where these all get a lot better than they used to be in the monolithic and then there's a really nasty plateau or even data depending on the details of your system right after that so I really can't talk about how to avoid that plateau how to get over that deal with each 1 of them so 1st of all let's talk about a couple this is old boring problem right have a monolithic follow spaghetti code and what you can do about it while he's been paying any attention there's an easy answer here the solution is micro-services why what we're even doing like that this is so lowered this point not not worth going into more detail unfortunately from the perspective of of coupling we have other problems now my services are absolutely Ganesha concept you can't get away from this that's arguably the entire point so when they share concepts how can I do that in a maintainable a and when I now have all of these distributed systems some of which I control some of which may be my colleagues control what in the world do I do when I need to refactor something now you're in a little spot so we're kind walk through basically a case study of 1 particular problem we had and talk a little bit about where we ended up on and that concept is called a service type I will give you the convenient and uh mostly true definition which is a service that tells us what products a particular business provides and how they provide a and for a team that does fulfillment for a variety of different verticals is a pretty core concepts so when we want to be launched with food OK is with the early days were star oregon keep it simple get a nice MDP out and see if the idea has merit were going make it too complicated when not overgeneralized where service types well let's pick up and delivery right % for centuries your pick up food from the business so you can deliver and that's going to be the only definitions of service types that we OK the product well were growing in 1 expanded let's do something new so we're going do appointment looking at like pickup and delivery of these I have a couple options so we're going to name them looking at business and looking at customer we've we've learned you know we don't call it generic pickup and delivery not generic it's booking in business and looking customers so here's the learning curve for really accelerating and you know you go through another year of growth and success honestly and what you find is that this is a complete like what what's going on here and this just pick a couple of these if you look at hotel reservation and looking at business i'd be curious if we took a poll the audience anybody like this the majority could successfully tell me the difference between the 2 and why why do we have good a customer and delivery isn't delivery good that customarily the stuff is nonsense right and this is where we found ourselves and we honestly we kind of
got there because it was convenient right and it made sense to so when I digress briefly into an an architectural diagram just so that what I'm about to say makes a lot more sense in context for this is roughly what the high-level architecture with life of our system will quickly walk through it and then go ahead and explain how it relates back to the service types so we have front and right these are things that these users pages many views checkup pages it communicates with fulfillment in the sentence fulfillments essentially keeping track of order states are you ready to be charged are you ready to be delivered at so the important bit on the right is payments kind of essential processes and then we have a big moon in the system essentially fulfillment decides that something needs to be done to an order and it doesn't it notifies the partners 3rd parties which I will say this a few times we do not control these are external companies with opaque implementations that they're gonna come back to partner in eye and say cool your order's ready to be charged we use more and we can keep this loop going feedback between us and the apart so that's the rough architecture and a couple things of node in yellow on the right is systems we don't control uh these are internal systems we don't control partners external systems we don't control at the front end here is actually all back in the model is because dirty little secret of migrating your code out of a monolithic usually don't completely migrate your code at the model has so front end for reasons but is still locked in that could be so we have complications here right we have a lot of sort of history that you know it would take me an hour to explain you properly and uh this whole system is is messy it's messy from the start so highlighted in red here are all the pieces of the system that knew about service types and when I say know I I mean I saw the code and they all cared a lot right we're not talking about like I'm passing it along to my and my friend downstream we're talking about I'm making important functionality
breaking decisions based on the state of partners but that's just that I don't know what partners do realistically we have no control over what they do with service type and that's equally terrifying here it's a little different so this
concept in general was really confusing uh it was pervasive alarmingly so far more than we ever intended to be it was really convenient right this
was something that we used all over the place to make decisions to use if statements you know this is very useful to us but wasn't designed now is really the core problem so I want to talk about how
we approach this and and I wanna practices right off the bat was saying we needed to do a refactoring we needed to do it across all of these systems and we knew we couldn't completely eliminate concept those were the constraints again so it's not a great spot to start from in hard refactoring project and our
approach will basically we're gonna draw boundaries were introducing domain-specific concepts things that make more sense to the system there and then this they'd service type thing that I could even explained very well to you and we're going to make sure those concepts a functionality right the sort of the domain-driven design concept that if in system here using words vocabulary concepts that are consistent with the system the coastal kilometers so going
into detail of the refactoring because it was long painful and boring right this is 5 different services with interfaces across to all the other services service type was in an alarming number of them and it was just hard mechanical work I what I do wanna know is are eventual solution was to basically corral into the smallest space we could in the system and then rival adapter layers to concepts that made more sense for this so we don't control partners we had to keep service type for them although maybe someday we would deprecated but in the rest of the system we are able to basically transform this into something that made more sense locally and was more maintainable so what
lessons can we draw 1st off and
and this might be like the most enlightening thing we discovered interfaces are not just your API it's right interfaces are everything that's shared across system and it's alarming how many things are secretly shared across your systems and we got it not because service type was a resource on 5 services but because it was that little add on parameter on the end of 15 different service calls this was something that was convenient to us and we didn't really examine closely so be very aware of what you're sharing interest intentional about it and if 1 day after a factor know that it will be very expensive that's unfortunately what comes with a service
based architecture In addition sacrificing his dryness was a really good choice in this case but it is not always the best choice and I think by default most of us a pretty uncomfortable with repetition code but it's a tool you have in your told all be aware of where it could help you out and 1 of the places that
helped us out in this case was at service interface boundaries it's natural place for the coupling so if you're going to do something that's a pretty convenient way to do aaai let's
talk about defining and by defining I mean or interfaces in particular have you ever
need to understand something and then told to go read the source I say yes I is a pretty common report right I'm busy read the source OK maybe it has its place had that work in in a network of services as at work if I don't own the service and suppose to be reading the source of work I don't speak the language it's written in can get a little growth what happens if I wanna know your search services interface but I don't actually know if validated properly or at all by does that mean your interface is the 1 you wrote down or the when you told me or the when you actually validate in practice or all of them or none of them how does this work and the fact is coming from our Python monolith honestly are interfaces were bad this is this is a cultural things that had thing that Python is not the strictest language when it comes to defining interfaces clearly right if you want to write bad interfaces is thrilled to let you do so so what you will about that I think that's just a fact of life that we would then and we need to be very aware of that when we're moving in situations where the importance of interface increases and I like this example this looks like a kind of the kind of Python code you might show to somebody right this is like a pretty powerful glycolic it right we have we have some squad usage that's
good I'm getting points in my in my expert Python user of column we've got some validation everybody loves validation and we have very nicely named methods this is an important interface for us right we would be very clear about what is user is not happening here unfortunately I'm willing to bet nobody in this room can say with any certainty what's actually happening here what is although right it has attribute that's what we know the water in those squads and what in the world are they doing to the users uh and and really overall just what what we staying with this set fixed we don't we don't know and this is the habits that we had gotten into from the monolith from working in a single base so we attack this problem
with something called slack and swagger is an API Description Language it is
essentially a suspect for what you're API is high if you heard when roots lightning talk a couple days ago I think on remote swaggering RAM always sort of uh slight variations on each of the different implementations different design choices so we chose this uh with 3 main goals in mind we wanted to make sure that we documented our system for ourselves both for people on other teams and the people writing interfaces we want to make sure that we made our clients smarter inasmuch as was smart and wise to do so and we want make services so what is
required for slide In short you need to write a gigantic step right you need to describe your API what endpoints do you have what arguments did they take or
responses can they return maybe a docstring there to in a really scary stuff and you may become lasting yourself in thinking of this is a where you make a big deal out of as I was astonished at how hard this was this is really hard to do if you give your your best developers on a service the job of writing that aspect to this level for the service they won't know how to do it will have to look it up and cut off the back it out laughter reverse engineering because if you don't do this from the start you don't know your API and you might think you do and that's almost water so this is the work you have to do to know what a fun I wasn't a big fan of that when I did it for our API I better to do it from the 2nd for so we get out of it that's the real
question uh 1st of swagger has agreed set of tools uh I i essentially a pretty view on that same data which showed you encode around the shining colors you get to interact with the against the some sample responses on the client side we
have a library called bravado and the goal of bravado is to basically be again considering that API from some remote service learning about what that service is actually offering in in terms of its endpoints in terms of its types in doing all the annoying mechanical work for you that you don't wanna be doing that we maintained client libraries packed Python packages for a long time that did this themselves and they did it wrong and they did it in completely and they they didn't validate and they had many of the issues it's a hard job to do right this is aimed at basically making all the mechanical difficult parts go and on the server we also have
something of what so I have a library called pyramids swagger that essentially into the pyramid web framework that if you're using that if you're not their equivalents went further and frameworks and its goal is to do the equivalent on the server side so was that look like that looks like serving the swagger schema that something like slash so that other things can access and that looks like applying validation is you've defined your entire spec remember and you don't have to rewrite the validation a dude incorrectly and there's also a variety of other sort of smaller tricks and goodies what
lessons can we draw from this 1st off
interfaces should be intentional don't touch the monkeys by please don't build it halfway and then slapped together the rest of it I think of it as a whole because if you don't it will become a completeness I guarantee it we've we've seen as like clockwork for every API that doesn't get a regular basically redesign at least whether not you actually implemented thinking about it from top to bottom figuring out what the new concepts on what concepts are updated so important process your interfaces need to be specific uh this is the thing that sounds very attractive when I say it out loud and people don't really like doing when they have to actually implemented code but there's no shortcut here if you implement implementation is not explicit in what you're interfaces against all have an interface you just don't know what it is right you can't make smart decisions based on it you can eventually get bit and then hard when you accidentally break backwards compatibility because you have no idea what you're actually change
and finally find mechanical things about this process and automate them mercilessly OK and I a mechanical pointedly here because 1 thing that we didn't do was automate away in the network right we didn't hide the network in in that's why the client and that was Due to some hard lessons where we kind of automated away the network and we discovered that that was a dead end and a very dangerous area go but all the mechanical stuff all the stuff you can completely no and unabashedly automate away should use OK so let's talk about
production it can talk about design coupling what's it like getting this thing actually running in practice this is
a real customer among report your customers might be the bug reports look similarly but it says I was using your API I'm a good good bug reporters I tell you which API and when I was using and I saw 504 it happened yesterday there you go Gulf except for me this is your job might up this report the budget your job is fix the bugs looking looking you with the system like this working honestly go for the few approaches and In
in ancient times you might pull in your most experienced developer to in a room tell this also 5 force coming from that away and hope that they can just kind of grab whatever logs are lying around and and fill in the details right then
part of my my poor analogy here is that this is this can like picking up a random bookshelf and hoping it can help you on a research project like again might be yeah you might get lucky and find some things that are relevant but you're also going to pick up just whatever this person have lying around and you using whatever larger company your your your team happen create previously you're going to be getting stuff that was built for different purposes that doesn't really work very well OK fine but the learn
what's right with log all of our ideas we have a bunch of services within to every request that comes in every response that goes out Britain illness parade while
so there some obvious upside to this problem you no longer don't have the the data and there's some obvious downsides this is your job right go find me the pattern in all of this data and do it by tomorrow because I know I'm not paying you to just sit around with through data and this is technically possible if you have some you know command line gurus you might be able to without crazy 1 liner that involves by different users of Grafton and and manage but I don't really recommend but what
can some alignment is bold logging everything we know about that and getting into basically examined it efficiently and for us that
ended up being ElasticSearch in command this may be an old an old story to you it's certainly a very popular choice you often hear it in a connection with this guy here log stashed and collectively this is called the OK stack but if you're completely unfamiliar with it and I going really do a good job of explaining it but I would encourage you to go investigate accepts frankly pretty awesome so we we use it for a
while dashboards Elastic Search stores the log data that's the the long and the short of it so what we've mostly found is it's tremendous what you learn about your system when you log data in production and you actually look at it's not a lot more complicated than we did nothing more complicated than showing ourselves a graph over time of all the T I had just broken down by party that's all this so that big spike blue in the middle is a partner that is decided this is the pattern of hitting our API this kind of your money if you'd asked to solve for and we will said yeah they're all doing exactly or more or less what we told the MiniCad documentation I'm sure there's not anything too weird and their right we're really specific when we wrote that documentation that's totally clear what they're supposed to do well in practice no right documentation is less clear than we think and their implementation is subject to their implementation you can also learn
everything's things about errors so when you push out that bad code and you see a spike of I can't talk to your service errors and usually several in doing it now and this is definitely a step up from entailing log somewhere and biting its regret and making sure you don't know right the wrong regular expression so this concept is permanent and you can share links very but how does it help us solve that
mystery for a 5 4 so you
can read this but this is a dashboard you can create quite easily and the real trick here is even though by default these dashboards and you over all of our log data we can write queries to whittle that down we can add filters to clarify what we're looking for so in quite small text at the top we basically said I want only results that have been method path you know user info that they're getting that part of the API and I want everything that doesn't have a 200 status code in response because it's not
complicated but it turns out in this particular day we're doing pretty well and we got exactly 1 of those and knows a customer complaint so this the kind of thing that would have been but I can say impossible but it would been particularly unpleasant and inefficient under raw data formats and quite nicely In fact for this problem we
dug a little deeper and using that basically the same set up we're able to say hey if we split out timings by data center 1 of them looks a little slower than the others and that's the source of all this time outs so we really it's hard to over over state how much value you're going to get out of just logging what's happening in production and being able to view it efficiently In realistically we don't
want our customers have to be telling us
these things in the 1st place that will be inefficient and it leads to its own problems so we have a
monitoring tool that we've open-source they don't call the last and 1 last words essentially
does is it sits on top of ElasticSearch and it uh does 3 things you tell at 1st what is the ElasticSearch instance I'm looking for and what the index named where work basically the tell it what sort of constraints you want to apply so in our case we're saying once at least 20 years in 2 minutes although this language is rich and you can say many things and then finally we take some sort of action so what's going basically happen with this rule is if we see 20 years in 2 minutes for page on call reducing hey there more errors than we
like and maybe we can insert a nice graph link that says here's something that will help you understand more about the problem OK
lessons learned so 1st of logging it
honestly superpower this stuff is
also man and it's going to be amazing for you I can't really over state how important this has been for us and I would say it's 19 days since before we started using it extensively that raw is not enough either you do need to visualize it and ideally you need to be proactively monitoring on just so that you can be spending your time doing what you care about rather than digging through mountains and mountains of raw text logs and over all these approaches made a world of difference of these 2 currents their responses from days I mean we we had a way to get an e-mail from a customer for that 1 down the minutes right we're going page the moment the stuff happens and it to the investigations figuring out why something went wrong from basically arbitrarily long because maybe we just never could figure it out down to something much much quicker but if I can go ahead and skip the last link them straight at times but I wanna talk overall
about some sort of
lessons we've learned in and direct this I the area so overall how can you approach problems like this frankly the 1st step is you really need to understand where you're coming from you need understand the system you came from before in our case this big model a million lines of Python code and all that implies and you have to factor that into the decisions you're making if you're ignoring that you're going make mistakes based on those weaknesses and this isn't anything new but it's it's an important thing to keep in mind services are exempt from this they're not a magic cure all for all your problems in fact they just exacerbate some of 2nd the explicit you wanna be straightforward and be really clear about what the system is how it interfaces with the other parts of the world and with what you expect being explicit is going to help you in a lot of ways that are hard to quantify will you're just thinking of it abstractly but that is sort of the the 1st step toward any automation if you don't write something down if you don't documented ideally in a format that's you programmatically readable you never going to learn something you need to measure everything have you not measuring something you just don't know it and you certainly never going lot right this is the kind of stuff that is basic to say that has a really profound effect in practice and again it's taken us from feeling like we sort of knew what we were doing to feeling like were on top of any new issues that come up in that we can even now proactively respond things that are going to be a problem in the future and finally scaling uh micro-services are more complicated than model this is just a fact you're introducing overhead you get some nice benefits from it but you can't just pretend that the way you treated a model with this go work in the new world so automation is the obvious way to get around it feels really rewarding to pull off and and it's going to turn the team into a bunch of people that are able to really focus on what they care about building an application delivering whatever the values that you deliver rather than team sit around and and patches up after broken services a couple resources to just leave you with others get have read those I mentioned fractional projects bravado and pyramids library for swagger integration in last alert for working with ElasticSearch doing monitoring there are all on get have we have a long form article on our transition from a model with 2 services written by sort of the tech lead of our services team really well written has a lot of background information I haven't dove vintage even most of the issues that came up and that's very interesting reading and if you more in the mood for something size we have watercolor service principles so this is all of the senior members of the teams that have been working on services for a long time he out basically got together and said here's what we're going to write down here is a summary of what we know
and with that of you may have some time for questions and if you want to get in touch with you through those work and also the whole way albeit beyond the
few but this is time for a couple questions could high on militia in the eyes to be you know you need to get them right as much as possible from the start but you don't always do that and so how do you you with be version you do to have a tool for about 7 best practices for sure but yeah you're absolutely right obviously we did not get it right at the start and we continue diversionary as we've were still solving this I would say that the best suggestions I have are we really appreciate at least documenting the interface you have great for stocks will make you realize when you need diversionary and as far as interacting with them we've been mostly just treating them like any other points you know you need to make sure that you make a the 2 you can have clients and switch over to the user all these logs the year collecting let you monitor will be 1 is actually completely dead rather than you just don't know of any consumers right now so that those tools can can help you out I don't think I have any sort of plug-in solution for the overall problem so thank you for a talk of in the beginning you said that not only do you wanna address complexity but also want to interact with different languages and become a little the tooling everything is shown was in Python script because I still wondering about that you end up using other languages of the new system we do use other languages the I would say 95 per cent of the ballpark and we have data services for the more high for high performance stuff and search stack and we don't to my knowledge currently have any cross-language but the beauty of slider in particular is its data right it's not the scheme is written down in data schema can be read and J. Sun and there do exist plenty of Java base clients for slide so we definitely focused on making sure that were cross-language I I don't want this can say that we have here I think we're in the process of doing so did you look into the rest constraints and more specifically how the media formats in order to get more coupled system that that yeah I can say that we talked a lot about hypermedia uh I I can't say that we went much beyond that I think for our money and we've been uh busy with much more practical that sort of like all the way on the top of the pyramid of need to know that everything else nailed maybe hypermedia were were still working our way through some more fundamental issues so in these kinds of you know low-level issues that way before you get to the point of nice hypermedia API it can where we are in evolution but with the last the what's the lead time from the bad requests starting to hit your service to that being fed to the low process you know such than getting a lot which shortened theoretically a in our systems I mean in your system in our system so it doesn't lost or doesn't care about how the data gets to ElasticSearch so depending on what your data generation scheme is that matters a lot for us we use describe logging we have some bridges into Kafka for persistence and I think the over all the time to get into uh into elastic searches on the order of 30 seconds 10 seconds so like low seconds on times and making it delayed in practice we find we can rarely react faster than that so that's been pretty axle force and there's no reason you can't make it faster we have put in the effort to say on the how you have registration and government so the services on the ground and adaptation you know all know what and you know which fackins to talk too much more about how the have you might you have many theories inside guess by different teams and how you break of reinventing the wheel support reusing of the service and so on yes that's sort of how do we make sure that we are constantly reviewing each other's work and all that that we have a few ways of doing that we've sort of tried to we have a service in a that focuses on building tooling rates but inevitably there also sort of a lot of expertise so you're very much encouraged to go to them and discussed we have a a sort of by by similarity something that's like perhaps we have basically skiff service services reviews basically when you're designing something new you put it out for review you write a formal spec and say and see if anybody like center has problems with it to have a lot of consuming code review current processes like that and we've generally try to make sure that serves always have 1 coherent owner 15 and we have 1 service sort of attached on the side of the model with that let us access that data we felt like trouble in the years that it didn't have explicit team with it just sort of being a tragedy of the commons so making sure someone owns it is reported but but don't have the little will signed helpful questions now so I'm just going to
Bit
Fünfeck
Vorlesung/Konferenz
Biprodukt
Baum <Mathematik>
Assoziativgesetz
Managementinformationssystem
Bit
Web Site
Kategorie <Mathematik>
Güte der Anpassung
Stellenring
Zahlenbereich
Information
Inhalt <Mathematik>
Figurierte Zahl
Computeranimation
Web Services
Bit
Subtraktion
Kartesische Koordinaten
Physikalisches System
Biprodukt
Systemplattform
Computeranimation
Arithmetisches Mittel
Virtuelle Maschine
Transaktionsverwaltung
Web Services
Twitter <Softwareplattform>
Front-End <Software>
Biprodukt
URL
Persönliche Identifikationsnummer
Varietät <Mathematik>
Metropolitan area network
Subtraktion
Dienst <Informatik>
Web Services
Datentyp
Güte der Anpassung
Zahlenbereich
Abgeschlossene Menge
Biprodukt
Drei
Systemplattform
Service provider
Computeranimation
Bit
Web Site
Punkt
Twitter <Softwareplattform>
Wort <Informatik>
Information
Ordnung <Mathematik>
Störungstheorie
Term
Computeranimation
Subtraktion
Prozess <Physik>
Gruppenoperation
Term
Code
Computeranimation
RFID
Benutzerbeteiligung
Prozess <Informatik>
Modelltheorie
Kontrast <Statistik>
Softwareentwickler
Gerade
Trennungsaxiom
Web Services
Suite <Programmpaket>
Kraftfahrzeugmechatroniker
Einfache Genauigkeit
Physikalisches System
Quick-Sort
Einfache Genauigkeit
Generizität
Dienst <Informatik>
Suite <Programmpaket>
Server
Gamecontroller
Wort <Informatik>
Punkt
Kumulante
Formale Sprache
Zahlenbereich
Kartesische Koordinaten
Fortsetzung <Mathematik>
Code
Richtung
Multiplikation
Domain-Name
Code
Programmbibliothek
Modelltheorie
Softwareentwickler
Auswahlaxiom
Gerade
Schnittstelle
Web Services
Softwaretest
Addition
Bruchrechnung
Graph
Reihe
Physikalisches System
Biprodukt
Quick-Sort
Rechenschieber
Dienst <Informatik>
Datenfeld
Mereologie
Unternehmensarchitektur
Bit
Zustandsmaschine
Menge
Hypermedia
Physikalisches System
Modelltheorie
Richtung
Metropolitan area network
Bit
Prozess <Physik>
Punkt
Komplex <Algebra>
Computeranimation
Homepage
Metropolitan area network
Web Services
Kurvenanpassung
Softwaretest
Lineares Funktional
Sichtenkonzept
Güte der Anpassung
Biprodukt
Kontextbezogenes System
Konfiguration <Informatik>
Dienst <Informatik>
Rechter Winkel
Ordnung <Mathematik>
Varietät <Mathematik>
Aggregatzustand
Rückkopplung
Subtraktion
Interaktives Fernsehen
Implementierung
SI-Einheiten
Code
Loop
Knotenmenge
Weg <Topologie>
Perspektive
Datentyp
Inhalt <Mathematik>
Modelltheorie
Biprodukt
Plateau-Problem
Drei
Beobachtungsstudie
Videospiel
Mailing-Liste
Physikalisches System
Quick-Sort
Generizität
Diagramm
Flächeninhalt
Mereologie
Debugging
Gamecontroller
Unternehmensarchitektur
Metropolitan area network
Befehl <Informatik>
Web Services
Rechter Winkel
Datentyp
Gamecontroller
Speicherabzug
Computeranimation
Aggregatzustand
Entscheidungstheorie
NP-hartes Problem
Metropolitan area network
Lineares Funktional
Randwert
Nebenbedingung
Web Services
Datentyp
Wort <Informatik>
Projektive Ebene
Physikalisches System
Refactoring
Quick-Sort
Computeranimation
Subtraktion
Zahlenbereich
Physikalisches System
Raum-Zeit
Computeranimation
Metropolitan area network
Dienst <Informatik>
Web Services
Rechter Winkel
Datentyp
Gamecontroller
Refactoring
Schnittstelle
Parametersystem
Addition
Schnittstelle
Physikalisches System
Teilbarkeit
Code
Computeranimation
Portscanner
Web Services
Suite <Programmpaket>
Datentyp
Default
Unternehmensarchitektur
Chi-Quadrat-Verteilung
Auswahlaxiom
Schnittstelle
Schnittstelle
Videospiel
Dean-Zahl
Datennetz
Formale Sprache
Quellcode
Code
Computeranimation
Physikalisches System
Randwert
Systemprogrammierung
Dienst <Informatik>
Programmfehler
Web Services
Rechter Winkel
Ordnung <Mathematik>
Verkehrsinformation
Schnittstelle
Expertensystem
Punkt
Menge
Wasserdampftafel
Validität
Ordnung <Mathematik>
Computeranimation
Attributierte Grammatik
Schnittstelle
TVD-Verfahren
Parametersystem
Subtraktion
Implementierung
Physikalisches System
Quick-Sort
Computeranimation
Metropolitan area network
Dienst <Informatik>
Client
Rechter Winkel
Wurzel <Mathematik>
Auswahlaxiom
Schnittstelle
Binärdaten
Sichtenkonzept
Wasserdampftafel
Ausgleichsrechnung
Computeranimation
Datensichtgerät
Übergang
Metropolitan area network
Client
Web Services
Menge
Reverse Engineering
Prozess <Informatik>
Fächer <Mathematik>
Endogene Variable
Stichprobenumfang
Kantenfärbung
Softwareentwickler
Gammafunktion
Server
Validität
Äquivalenzklasse
Term
Framework <Informatik>
Quick-Sort
Computeranimation
Benutzerbeteiligung
Client
Web Services
Prozess <Informatik>
Datentyp
Mereologie
Client
Server
Programmbibliothek
Varietät <Mathematik>
Schnittstelle
Maschinenschreiben
Schnelltaste
Vervollständigung <Mathematik>
Prozess <Physik>
Single Sign-On
Implementierung
Code
Computeranimation
Entscheidungstheorie
Schnittstelle
Client
Datennetz
Flächeninhalt
Rechter Winkel
Biprodukt
Computeranimation
Managementinformationssystem
Explosion <Stochastik>
Güte der Anpassung
Ausnahmebehandlung
Physikalisches System
Login
Computeranimation
Programmfehler
Forcing
Prozess <Informatik>
Reelle Zahl
Reelle Zahl
Softwareentwickler
Verkehrsinformation
Web Services
Dienst <Informatik>
Fächer <Mathematik>
Endogene Variable
Mereologie
Projektive Ebene
Analogieschluss
Computeranimation
Web Services
Subtraktion
Datenmanagement
Prozess <Informatik>
Mustersprache
Computeranimation
Einfach zusammenhängender Raum
Graph
Prozess <Informatik>
Mustersprache
Implementierung
Physikalisches System
Elastische Deformation
Speicher <Informatik>
Biprodukt
Auswahlaxiom
Computeranimation
Resultante
Regulärer Ausdruck
Filter <Stochastik>
Web Services
Mereologie
Endogene Variable
Abfrage
Information
Binder <Informatik>
Default
Code
Fehlermeldung
Rechenzentrum
Metropolitan area network
Rohdaten
Dateiformat
Quellcode
Biprodukt
Computeranimation
Aggregatzustand
Nebenbedingung
Explosion <Stochastik>
Datentyp
Open Source
Gruppenoperation
Formale Sprache
Indexberechnung
Systemaufruf
Schlussregel
Quick-Sort
Computeranimation
Homepage
Homepage
Metropolitan area network
Modul <Datentyp>
Verschlingung
Automatische Indexierung
Wort <Informatik>
Biprodukt
Fehlermeldung
Instantiierung
Subtraktion
Datentyp
Graph
Momentenproblem
Vererbungshierarchie
sinc-Funktion
Strömungsrichtung
Binder <Informatik>
Login
Computeranimation
Endogene Variable
Homepage
Homepage
Rohdaten
Modul <Datentyp>
Konstante
Endogene Variable
Leistung <Physik>
Biprodukt
E-Mail
Metropolitan area network
Aggregatzustand
Gruppenoperation
Kartesische Koordinaten
Code
Computeranimation
Bildschirmmaske
Web Services
Programmbibliothek
Modelltheorie
Booten
Gerade
Soundverarbeitung
Web Services
Bruchrechnung
Physikalisches System
Quick-Sort
Entscheidungstheorie
Integral
Dienst <Informatik>
Flächeninhalt
Softwareschwachstelle
Mereologie
Dateiformat
Projektive Ebene
Information
Overhead <Kommunikationstechnik>
Nebenbedingung
Maschinenschreiben
Subtraktion
Prozess <Physik>
Punkt
Formale Sprache
Adressraum
Formale Grammatik
Versionsverwaltung
Bridge <Kommunikationstechnik>
Komplex <Algebra>
Login
Code
Physikalische Theorie
Client
Web Services
Existenzsatz
Skript <Programm>
Modelltheorie
Schnittstelle
Zwei
Plug in
Ähnlichkeitsgeometrie
Strömungsrichtung
Nummerung
Physikalisches System
Bitrate
Quick-Sort
Rechenschieber
Dienst <Informatik>
Generator <Informatik>
Forcing
Rechter Winkel
Registrierung <Bildverarbeitung>
Hypermedia
Anpassung <Mathematik>
Dateiformat
Ordnung <Mathematik>
Baum <Mathematik>

Metadaten

Formale Metadaten

Titel Arrested Development - surviving the awkward adolescence of a microservices-based application
Serientitel EuroPython 2015
Teil 147
Anzahl der Teile 173
Autor Triglia, Scott
Lizenz CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
DOI 10.5446/20087
Herausgeber EuroPython
Erscheinungsjahr 2015
Sprache Englisch
Produktionsort Bilbao, Euskadi, Spain

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Scott Triglia - Arrested Development - surviving the awkward adolescence of a microservices-based application The potential upside of microservices is significant and exciting. So much so that Yelp's Transaction Platform committed from the start to an architecture of small, cooperative microservices. This talk explores the inevitable complications that arise for Python developers in as the services grow larger and stretch both their own architecture and the developers responsible for them. Come hear tales of terror (tight coupling! low test coverage!), stories which will warm your heart (agility! strong interfaces!), and everything in between as we follow the adventures of our plucky team. The talk will be focused on the functional, cultural, and reliability challenges which occur as a microservices-based project evolves and expands over time. Particular attention will be paid to where these diverge from the utopian way microservices are often described, and to the particular difficulties faced by Python developers trying to implement such systems. My goal is to share with attendees some mistakes we've made, some successful methods for growing gracefully, and Python-specific tools/libraries which can help with these problems. To enjoy this talk, you should be aware of the basic vocabulary and concepts of HTTP-based services. Any additional awareness of distributed systems (and their failure modes) will be helpful.
Schlagwörter EuroPython Conference
EP 2015
EuroPython 2015

Ähnliche Filme

Loading...