Add to Watchlist

Nameko for Microservices


Citation of segment
Embed Code
Purchasing a DVD Cite video

Formal Metadata

Title Nameko for Microservices
Title of Series EuroPython 2015
Part Number 77
Number of Parts 173
Author Bennett, Matt
License CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
DOI 10.5446/20159
Publisher EuroPython
Release Date 2015
Language English
Production Place Bilbao, Euskadi, Spain

Content Metadata

Subject Area Computer Science
Abstract Matt Bennett - Nameko for Microservices Microservices are popping up everywhere. This talk will explain what this fashionable new architecture is, including the pros and cons of adopting it, and then discuss an open-source framework that can help you do so. Nameko assists you in writing services with well-defined boundaries that are easy to test. By leveraging some neat design patterns and providing test helpers, it also encourages good service structure and clean code.
Keywords EuroPython Conference
EP 2015
EuroPython 2015
and thank you and welcome everybody thanks for coming this talk is about market services and and make which
is an open-source library you can use to write them in Python my name is matt then it's time that apart from engineering at company that's currently still in stealth mode so I can't talk about it 1 camera and and previously I was a senior engineer 1 find state which is where the maker was was born say how many people in the room know the phrase processes can
describe and how many of you knew about it 2 years ago not that many Marcus services is is the hot new buzzword and suddenly they seem to be figure history lesson in
November 14 months Fowler dangerous publish this paper micro-services i which I think is considered to be the
seminal paper on the topic so I highly recommend it it's very accessible as number along and a lot of information and Susan very recent so they didn't invent the term micro-services but they gave it a concrete definition and really propelled it into uh of cavalry so that 1 finds that we discovered this paper that when it was published and realize that it describes what we've been building for some time and that was really exciting because suddenly we had a common language well with which to share ideas about stuff so for the uninitiated what are market services or more correctly what is the micro-services architecture so this is Martin Fowler's definition it's an approach to developing a single application as a suite of some
crucially each running in the same process and communicating with likely mechanisms so I think it's helpful to contrast like the services to model which is probably your default way of building in applications and as a single process 0 typical gender sites is a good example you would probably compartmentalize your logic into different acts in gender patents but ultimately they would run in the same process and memory space as each other and where in micro-services your apps become entirely separate programs so in a sense this is a this is good old-fashioned de-coupling and encapsulation spot applied and process and so what this forces you today is considered to be the boundaries of the services of the scenes that run through your 1 3 replications a common response to the height around micro-services is you should be doing this anyway and that's just good design which is trading but with my processes you you can't be lazy and they say across component imports because it's not that input and there are other benefits to using 2 separate processes as well so 2 reasons for document is that the
private of any software architecture is scale or rather maintainability at scale so I mean skeletons of several hundreds of millions of requests a 2nd but rather in the complexity of the problem you are trying to solve and the team that is charged with solving so there's an analogy for this that Alan Kay is inventor of small talk and object-oriented programming use in a 1997 he notes which I watch a video of because I was 13 in because isn't so somebody ask you to build a doghouse out of
wooden planks and nails you probably be able to do a reasonably good job recently sound structure if they are seated in scale up 200 times the size using the same equipment and tools it you you couldn't do it the thing would collapse under its own weight and so you when society started building massive structures like the ones we use studying objects to support the weight of the structure and I had this light moment where I realize that the etymology of the word architecture is literally the application of launches so happen markers moccasins is help you achieve maintainability at scale we've already said it's about the coupling and
encapsulation but what else so as separate programs that independently deployable which means you can have a separate release cycles and set the deployment processes the each part of your application so the Guardian newspaper have written about how they've embraced microservices and is allowed and start using continue delivery in in and it's really very
quickly in to deliver 1 part of the application but without getting at dates and slower-moving World War legacy or more risky parts at risk several programs are
also independent scale so now I am talking about sending hundreds of millions of requests so you to scale monolayer you have no choice but to replicated and deploy another instance yet replicate the whole thing but micro-services a much more granular and therefore more composable so if you have a service that is very highly CPU-bound you can deploy more of those across more CPU's without having to drag along the rest of the application as well and is also a freedom of technology so being good I finish this I'm
sure we all really want to use Python 3 where we can but sometimes we get stuck using anode library there hasn't been updated in and therefore we cannot stuck in Python with in a monolith you have to use the lowest common but like sensors individually free to use the most suitable interpreter for that supplied to pi 3 pi pi sup and I've perhaps say this
too loudly at Python conference but this extends to your like your choice of language as well so if you want to experiment something functional like Haskell and you can write a service in the language forgive this sector reference on this 1 but Marcus are not monolith sorry outside the realms of Software Architecture a monolithic
something as big and imposing and impenetrable think that the from 2001 space outwards markers is a small and nimble and quick they have a smaller codebase which means it's it's easier to bring a new developer on board that and understand the whole thing is a lower cognitive overhead to understanding how it works which is inherently more maintainable and his Conway's Law say how many people felt Conway's Law so this is something that thought
let's talk about a lot and in in 1968 a chapter called Melvin Conway said this organisations which design systems are constrained to produce designs which are copies of the communication structures of these organisations the 1968 doesn't seem like there any new ideas and so architecture so your did you have a little regular treated with application you have you have a database layer and you have an application logic layer and you have user interface and you likely employ specialists that the work in this area John Watson the migrants and as a member of the of the middle I would be able to see to talk and talk to my application developer appears every day and it would be really easy for us to communicate and then we went to speak to the UI flights we use the subtly slightly different language and it's this layer of labor friction and then it demands that we made mistakes in it was harder for us to communicate with that's commas been actions so what bullet-ridden instead is
you build a small multidisciplinary teams and then you separate them based on the natural divisions that exist within the organization that yourself and as a result you get an application that better reflects the organization rather than these somewhat arbitrary technical so
these wonderful benefits away and what is it cost well it's kind of a growing up architecture there a a lot of things that you have to have in place before you can make it work for you if you want to avoid the architecture Duncan so there's a double ahead if you're increasing by 10 or 20 times for 100 times the number of things that need to be built and deployed and looked after doesn't massive headed for operations
and anyways tight really is the leverage automation so it for for your testing deployment for your machine management and another insight from works is that Marcus services part CDs architecture what means is that it's enabled by automation and without automating the testing deployment machine management this bird would make micro-services completely as so I think this is why in megacities is all of a sudden seemed to surrounded it is the same good ideas of the decoupling and encapsulation but with this
new dimension being enabled by by Dempster he was a demonstrator ahead you you also have to embrace the domain in which law operates so you could argue that for sufficiently complex application should really be doing it anyway but I
certainly replaces the didn't so what I mean and by domain knowledge you have to really understand the business requirements i.e. the problem that you are trying to solve for your organization and you have to do that so that you know where to draw the lines between your services how to divide your application of you can't just build the weather and then attacked things on as they become apparent micro-services force you to do to do this up front to and it the
decentralized aspect so you you know don't have a single source source of trees like the traditional database where we have to relinquish and to relinquish acid guarantees and instead embrace based which stands for
basically available soft soft-state eventually consistent
which is really awkward backronym but a good chemistry J and so this means that is that you can't applying transactions across calls to multiple services you have to apply in 1 place and then wait for those changes to be propagated and reflected in all the other players presidential consistency so i want wednesday we made we made a mistake in this in this row we built an abstract calendering service that can the data calendar and data and calls for several other services and because there there in separate the calling service in the calendar service separate we can apply transactions across the so it we kind a rookie error really but you know what what ended up happening was the calling service with right to become called write something to the calendar which would succeed or fail and if it succeeded the calling service would then do something else if that's something else fails we had to capture it explicitly and then calendar services to say are just please and do the things just done and in we couldn't use a role that the transaction transactions which achieve that present unnecessary leads to
it forces us to jump through and of course there's also the race conditions where we have while the calendar has something in it that we end at end up removing later something else that's the calendar and they see this free full but it should actually for through decentralized
aspect means you have to think about these things and you have to be aware that you are building your introducing complexity so a collection of micro-services is fundamentally more complex than a model the
molding parts and those moving parts are connected by a network which is inherently less reliable than in memory so it in a complex system it in a in a common system failures rarely happens for exactly 1 reason is usually a cumulative effect of various soft failures adding so you in your network slows down in 1 area of your infrastructure which causes a backlog of requests which combined with the recent code change means writing some more to this which means that you run disperse and
in and it's only when you get to the 4th and the 5th of the end soft failure you actually fall over OK to mitigate this you need monitoring and telemetry and you need an analysis of the data that that produces so that when something goes wrong you can figure out what it was what caused it will preferably figuring out the the 4 that's right so by now you may be asking yourself when my percentages right and if so here's some questions to consider is a code base large enough that no 1 person understands all your
dad and release was slowed because of chains of dependent changes that need to be made do tests take forever to run besides you might be fighting among and if that's the case are
you ready to support a distributed system do you buy you leverage automation your tests deployment of machine learning and do you have sufficient monitoring and analysis in place to figure out what's going on inside is if your answer to these questions yes and no respectively this not maybe you can build a multiple so this is a term that I came up to me yesterday and so much from other respects but it serves the purpose of the presentation there is a sliding scale between that tens or hundreds of my services of 1 and and a single monolayer of the other and this is a continuous continuous spectra
so you may choose to augment your existing models with 1 or 2 satellite micro-services the multi and this way you get some
of the benefits by you could choose to use a different interpreter or you could try out CD here without most of the costs so assuming were all involved and and ready
to embrace micro-services for multi left let talk about making so it's an open source pageantry projects and it's a framework is designed for writing my processes we named after the Japanese mushrooms which grows in clusters like this and we thought that you know that kind of look like micro-services with many individuals
making up the larger thing on the so I asked him about his friend of mine winding degree like that and he shrugged and said that is not much room the story so there is
a couple of important concepts I need to introduce have explain some of the design principles in America there are entry points which are how you interact with the service this is how you request something from it or otherwise get it to do something entry points of the interface of boundary of assessments and there are dependencies which is how the service talks to something external to it and they want to communicate so for example a database or service so we don't have some kind of I that the they carried in in the following examples in recovery so on galaxies right if you want uh and make service is written as a as a Python class so it has a name which is declared with the name entity and it has some methods that encapsulates the the business logic of this set and then methods are exposed by and checkpoints so this http decorator here will call the Greek methods if you think you get request to that to expand this example slightly and let's pretend for a minute that string
formatting is really expensive and we want to catch the greetings rather than generating return and also switched out the the entry point so now and it's a remote-procedure-call implementation as opposed to http so the 1st thing to notice is that the the business logic of the method is is unchanged by to challenge point we've we've added logic to deal with the cash box it's entirely isolated from having to to do with HTTP offices so in other words that it's a declarative change that has no impact on the procedural code in the method and the 2nd thing to point out is that the cash is is is added as a dependency so this line here can actually was cash client is the Declaration of Independence so dependencies are special in make you you
declare them on your Service class like this but the class actually is different to the instance-level attributes the method sees when it executes and that's the case the dependency provider which is our declaration the dependency provider at the instance level actually at runtime so if we if we hacked our our method here to to print these to actually it's when uh when it runs see that's we see that the date that they're different to the first one here the top 1 cash is dependency provider and the 2nd 1 is actually an instance of a memcached client objects and that's what the dependency provided objective so you're using depends injection like this and means only the relevant interface gets exposed to the search
methods and the service develop their all the plumbing of and managing the connection pool for handling reconnections is nicely hidden away inside the dependency provide so this emphasis on entry points and dependencies Rousseau makes you make a very extensive full and points and dependency providers are implemented as extensions to make even the ones that we share with the library which we include so that it's useful out the box the intention is that you are free to
encourage to build your writing all maybe 3 the 1 about so somebody would already built it for you so this is the list of
built-in extensions so often see decorated it we still area is an entropy-based obviously implementations that gives you a request response type
coal by the method press this use I publish subscribe the implementation of gives asynchronous messaging over and keeping it is time for crime like common things and this experimental WebSocket tuple so and is was explaining why we have this this entity stuff in here so an HTTP is a natural starting place for my perception there's a lot of great lightweight web frameworks out there there's uh this great tooling around around API exploring and caching Anais's is ubiquitous so and you probably gonna need http on the outside services so that clients can can interact with but for service the service interaction inside your cluster micro-services where you control both sides you probably want something other than http In particular pub is a killer app for micro-services they're all kinds of patterns for distributed systems that rely on asynchronous messaging with high fanout capabilities and stuff is really quite so that's why we included in and out of the box you have to use it and there was a really great test helpers in in making
so we've already seen how injecting dependencies keeps the service interface clean and simple but it also makes it really easy to plug those dependencies out during testing so error this snippet here we using a help cold weather factoring i which is very useful when unit testing services so you pass in your service class and it gives you back an instance of that service but with its dependencies replaced by my projects the say you then you remember tests the and you can exercise you methods by it by calling them and their finding Moscow called a purpose that would eventually has another mode of operation where you can you can instead provide an alternating dependency so In this case here were providing uncensored dependency we using the mock cash library which has for this for this test has a much nicer interface you need to set up return value thing sigh and there are there are other helpers in you might for the do this kind of thing for for integration testing to help you with integration tests now let you run services with locked out dependencies of disabled entry points so that you can limit the scope of your integration to the police service interaction to the things that you actually want to test so to summarize In March Services architecture he's very application services which run as a person and this is a way to achieve maintainability at scale so the you can build cathedrals of software and it comes with a host of other benefits the like freedom of technology the couple of release cycles even team structure if you want for each component part but it's a grown-up architecture it core you're building a complex distributed systems that which means you need to automate your devil you need to monitor and need to analyze the results of that monitoring and use of overall be aware that you're building a distributed system and when all of those distributed trade offs but you can also into adult incrementally you know by adding 1 or 2 satellite market services to your existing staff and if you want to go on services adventure this open-source library that can help you with it is made for writing services encourages you right clean highly testable carried and the several built-in extensions so it's useful out of the box but is designed to be extended to your so you want to know more redox for the recovery
that singular much be fit thanks for the talk on how we can but fairly so with there's lots of time for questions the
and so for i in my question is it seems that uh if there is some of the implications not you migrate stuff towards from 1 elliptic 2 or more and micro service kinds of architecture and but is it a good idea to actually start doing a major services from the beginning this is
so sound idea to try and so I ordered something small that's a that's a bold move I was saying uh so in that paper that lot of other published you at the variety and he still the total about Marx's services ready and says the provisions that are required services and I think that probably depends on your prior experience what you know what your you feel right that is in a way that you when you start in a blank slate or not the kind of title our sense for
the presentation of the and is there a big enough of open source project use code that we can look at as a real world example yeah that's a good question
so when in in a heavily in use at 1 time step which is closest and and there are a number of other smaller London starts at the starting to use it I don't think there any public open source applications that are using yes hi and
it seems to me that I believe that
this is going to be present at this and is very very much similar do I mean maybe it's a crazy idea but why not try to data the best ideas of both and build something from similar yes I'm excited to talk to you guys from the latest so we have high high act as you we've actually many changes that stuff and you know Europe Python is our opportunity to get together and talk about sharing some ideas so I have a 2nd order questions which are more technical but is the way I mean I've looked at the period that the condition this morning and this simply denies but there's 1 thing that missing that this just a simple XML-RPC of Python provides is the possibility to do introspection in the methods you want to know what arguments are expected by a certain method you want to have access to the doctrine of the method is the and I couldn't find it in the code on the documentation is there was a way to do this with nomical well so that the engine when decorators don't they don't mutates the service methods to should be have to take a service class and inspected like a like a regular as their regular class and look at the doctrines of the methods that you've implemented in all that is all the all yeah you for your service consonants that the long to so you don't want to import the on the client side want import the 7th place because sometimes as so screwed will depend on many things like I don't know database and efficiently you some the on on the train and you really do not want the service code to you wanna be introspecting on the client side yeah like at runtime that current today would examine up this year of the symbolism of purpose was you to do breed this uh this study is thought to assist them but methods for instance or what not I mean this is really useful for to develop a plan yet until the service so 1 thing that we have here bounce around for a while as the possibility of a client library where you can find service you can export something that the client can then interact with an otherwise you talk about shipping schemas over the y which is also a possibility having extraction obviously doesn't have this thing and I would prevent category of phone extensions the you can actually make is actually quite a quite young library certainly need it as an a as an open-source project being promoted say there's a whole bunch of possibilities this that and I had to be getting there are class that there is any ongoing efforts to make it a thereby should graphical interface is and this is about so again this is part of the accessibility so we we have what I say we use and keeping very heavily and say we build and the the built-in things we build because we needed them and membership in with 11 because we think a useful yeah of using Elton's semesters bustle using you know 0 and Q in any any of alternative communication mechanisms falls squarely into the category of is an extension of this building was built an entry point for uh and I hope that's what what happens notice of basically right now there is a kind of are building might anything beyond what's being on the slide or is no yeah but it's not just that it's not capital that's area of I think if a little kid urges the basic question so when you call for Microsoft architecture so you need to be sure that the 2 services will never need to share memory in the future or otherwise it can be quite a large amount of work to merge them together isn't it sure yeah thank you hi things like the talk and it looks like 1 of the hardest things to the wizard transactions you have any suggestions on how to approach the problem not really this transactions are 1 of the things that we've that we have got used to it and you don't you the ability to have have atomic transactions in my presence is but it's within the scope of 1 individual cell so if you need that's why dividing replication up is difficult because you need to make these decisions about you where do you put these boundaries so that you can have atomic transactions in places where it matters and fall back to eventual consistency for other the tumor and we talk about it and I'm sure it a few of us here in the room working on moments and you have any suggestions on how to approach the refactoring it into micro-services yeah I want to watch out for doesn't amount less so this is exactly what happened at 1 finds we FIL this uh we don't this January which is still out front end and they accumulated all this logic about in bookings and payments and financial stuff and and it just became unwieldy and so the journey started with the care let's take let it was actually the same piece that doesn't yet exist that we know it's going to be really hard to add into this gigantic current but let's just below that's a separate thing in say this is the 1st of a good candidate for the 1st month of services and using the need to date make that separate and then maybe you can identify another segments within your app that's been reasonably coupled already and you can you can leave that out it it is known that you can't really answer that question in in anything other than abstract hello there aren't there any situations where you wouldn't recommend to use a micro-services and the integration any situations which you wouldn't recommend wouldn't recommend you because I mean you can use it for use so you you probably don't need if your that if you're a developer
team is going to 3 people strong and you definitely don't want shouldn't there if you're not prepared to support distributed systems aspect of it but if you don't have automation in place we feel free that uh it's kind it's not a big commitment see you and you really wanna start going down this road when you know that you could the relevant things in place is unless he comes up pretty quickly hello there have tried doing using the mako on Platform as a Service that's 1 thing and then how do you approach like configuration I see that you uh declared our service but where is the services curation taken from right on on platforms in platforms and service industry many is symmetric museum had around with the above offer an but recognize that there on the on the on the in the context of this I didn't I didn't show you can provide on my life as a young follower to that contains a conflict and then dependencies have access to that conflict so you can make it excluded for simplicity that you would probably specified good the complicated lookup for sentiment cash locations when you declare dependency and then there was no to guarantee that element of the context of so if you if you look at the code on the hand and have done what do you maybe have a kind of learning environment variable positive like for 12 factor acts that are configured through an environment variable variables no no but and last question how do you run your services like was beginning be done according or with micro was we also think that there's a kind of in in Jamaica and then we just run that behind supervisor I'm so I've already been using nomical and was wondering if there's any interest in the spin this weekend on Monaco she has a son a flat on Saturday nite but all this Saturday at the time of proposal still I always wonder hallmark which should be the micro-services synthetic it's a bit it's a bit unfair to compare to 2 to compare only against the monolith and as you said before it's it's just it's a good Archie picture if you have components so if you said you should do market services if from a single person cannot keep the whole called in his mind but well if you spilled up into medium services that still fit into the human brain well there is a post to having basically every year out of your original configuration as its own service I find that I don't know where where's the limit so that doesn't that doesn't seem like you said at time micro-services actually kind and helpful because it implies a size which I don't think really had any really applies no 1 find say we perhaps didn't make all of the decisions correctly about how you know how to divide collection of but we had some services that we need to give you know just a couple of methods for me and others that were yeah could only just think somebody's bright thousands of and so it is not helpful classification from you is very unlikely he'd end up with lots of services that are all the same size in so have yet been keeping granularity deciding which that's that's the hard that really the you've told about the Bolton extensions can you extend cannot can I extend the hand write my own extension like if I want to put the political yes say and what's your was your that was the example of and was that was suggestion that the I want to so yes you can have you actually can serve as an entry point a hot as right because the emotional harm the dependencies a pretty easy so if the salaries environment if you want to different of data this example that is if you want to send a message of hope would messaging SQS it's relatively easy thing to do does anybody have any more questions 1 yeah you mentioned some of the key points of until all at the hop points to when you go into microservices and that the vaults and 1 of these he said you have their place really good monitoring and and you consider covering some support them namical for having some of approach to monitoring let's say you say you what the thing we used to that 1 finds stay which worked extremely well was uh was used session and elastic search so every time an entry point 5 we DispatchMessage signal there will be ingested by log stashed personalized search and we used to bind to explore the data to you could see you know what which must uphold and entry course that you can see which methods call them which arguments were which was generated areas and how monitor and what size the payload was and you can know whose 1st cool graph so that you get different you can see spikes in exploring now work really well so that didn't get source before I before I change jobs so I'm currently in the process of of the implementing that and that would become 1 of the 1st open source things a little thing here so if there are no more questions please thank be fed
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation


  473 ms - page object


AV-Portal 3.8.0 (dec2fe8b0ce2e718d55d6f23ab68f0b2424a1f3f)