Go and the modern enterprise

Video in TIB AV-Portal: Go and the modern enterprise

Formal Metadata

Go and the modern enterprise
Alternative Title
Go - Modern Enterprise
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Year

Content Metadata

Subject Area
Enterprise architecture Context awareness Enterprise architecture Multiplication sign Function (mathematics) Evolute Product (business) Spacetime Formal language
Enterprise architecture View (database) Decision theory 1 (number) Drop (liquid) Twitter Architecture Googol Cuboid Data conversion Social class Enterprise architecture Service (economics) Fitness function Electronic mailing list Drop (liquid) Digital signal Twitter Category of being Type theory Spring (hydrology) Point cloud Self-organization Quicksort Service-oriented architecture Square number Spacetime
Point (geometry) Service (economics) Enterprise architecture Architecture Self-organization Bit Spacetime
Web 2.0 Enterprise architecture Service (economics) Term (mathematics) Business model Bit Service-oriented architecture
Enterprise architecture Just-in-Time-Compiler Service (economics) Vector space Right angle Bit
Service (economics) Group action Latent heat Enterprise architecture Process (computing) Service (economics) Term (mathematics) Multitier architecture Hard disk drive Data structure
Enterprise architecture Service (economics) Programming paradigm Enterprise architecture Service (economics) Line (geometry) Food energy Twitter Particle system Structured programming Latent heat Personal digital assistant Buffer solution Remote procedure call Data structure
Mobile Web Android (robot) Greatest element Enterprise architecture Service (economics) Sequel Information Multitier architecture Virtual machine Set (mathematics) Web browser Function (mathematics) Client (computing) Instance (computer science) System call Web 2.0 Data model Logic Video game Right angle Quicksort Row (database)
Polar coordinate system Group action Digital electronics State of matter Multiplication sign 1 (number) Set (mathematics) Primitive (album) Client (computing) Stack (abstract data type) Perspective (visual) Food energy Order of magnitude Formal language Neuroinformatik Different (Kate Ryan album) Semiconductor memory Diagram Physical system Enterprise architecture Service (economics) Software developer Binary code Maxima and minima Food energy Instance (computer science) Benchmark Connected space Process (computing) Befehlsprozessor Architecture Telecommunication Order (biology) Right angle Figurate number Quicksort Metric system Resultant Point (geometry) Implementation Server (computing) Enterprise architecture Service (economics) Identifiability Twitter Time domain Goodness of fit Permanent Energy level Mathematical optimization Domain name Information Expression Interactive television Coma Berenices Line (geometry) Binary file Compiler Word Integrated development environment Personal digital assistant Blog Universe (mathematics) Library (computing)
Group action Context awareness Code View (database) Multiplication sign Design by contract Primitive (album) Mereology Formal language Single-precision floating-point format Software framework Circle Data conversion Monster group Physical system God Source code Enterprise architecture Service (economics) Data recovery Bit Parsing Connected space Category of being Process (computing) Software repository Boris (given name) Order (biology) Different (Kate Ryan album) Self-organization Right angle Pattern language Quicksort Metric system Writing Point (geometry) Server (computing) Histogram Enterprise architecture Service (economics) Open source Patch (Unix) Tape drive Canonical ensemble Event horizon Goodness of fit Term (mathematics) Business model Energy level Histogram Distribution (mathematics) Standard deviation Scaling (geometry) Information Interface (computing) Graph (mathematics) Projective plane Independence (probability theory) Line (geometry) Cartesian coordinate system System call Integrated development environment Personal digital assistant Blog Universe (mathematics) Pressure Library (computing)
Context awareness Group action Digital electronics INTEGRAL Multiplication sign Modal logic Decision theory Source code Set (mathematics) Client (computing) Stack (abstract data type) Formal language Direct numerical simulation CAN bus Strategy game Single-precision floating-point format Software framework Error message Physical system Enterprise architecture Service (economics) Programming paradigm Closed set Moment (mathematics) Flickr Message passing Arithmetic mean Interface (computing) Direct numerical simulation Self-organization Right angle Lastteilung Cycle (graph theory) Quicksort Physical system Fundamental theorem of algebra Spacetime Row (database) Implementation Server (computing) Enterprise architecture Service (economics) Motion capture Similarity (geometry) Control flow Mathematical analysis Machine vision Number Twitter Latent heat Term (mathematics) Bridging (networking) Touch typing Business model Energy level Form (programming) Module (mathematics) Standard deviation Information Server (computing) Interface (computing) Projective plane Physical law Mathematical analysis Client (computing) Limit (category theory) Configuration management Evolute Cartesian coordinate system Equivalence relation Visualization (computer graphics) Integrated development environment Personal digital assistant Communications protocol Abstraction Local ring
Due to due OK so I 1st of all data coming that's like goes been around for 5 years now I think the 5th birthday of uh an amazing language and its output we've been using it for a long time I think actually pretty 1 where release for some internal tools so today I wanna talk about basically go in the context of companies like SoundCloud which I'm calling the modern enterprise so 1st I want to talk about what I mean by that what is a modern enterprise wide it's important I am going to talk about my experience that SoundCloud specifically like our product evolution and infrastructure evolution I'm a talk about how I think that generalizes to other companies in our space and so I'm going to talk about what I think go needs to succeed in that space and so how we've kind of like in some ways missed the boat unfortunately in what we can do to make up ground so 1st of all the modern
enterprise I what I mean this is what I
think it is it's companies that are somehow tech-related I probably consumer-focused but they almost certainly have achieved hockey stick growth In our their their needs lives they probably have at least 100 engineers so a sizable company and probably they also have what we know is like a service-oriented architecture a company doesn't have to take all the boxes but probably most so what is in companies that kind of spring to mind anything of the sort of thing this is what it is for me so obviously Google and is quite large may be too big I would like to come across some of the lists that Amazon is kind of in maybe the same way like maybe there have been too big to be considered in this space and then there's a whole class of companies like Etsy which actually operates basically monolithically like not in SOA style so I'm going to cross them off the list as well and that leaves like companies in this space I think of they share a lot of properties they are pretty big Twitter's I think the biggest of these leading efforts is bigger now actually but I think they're like 15 hundred engineers or something like this but some cloud were much smaller were like 120 or something like that I but these are like what I consider to be the big players but it's not by any means an exclusive list if you kind of drop some of the requirements is a lot of companies and the ones that I know he's go the kind of fit into this space so this is a type of organization I'm talking about and I guess I would emphasize that there I kind of driving a lot of conversations in the tech world and in my view of a lot of technology decisions being made by these companies or these companies are can be replicated by other companies and that's the thing that I can I wanna talk about today Saloni talk
specifically about my experience at example of and this will be kind of a pretty common story if any of you of will a familiar with this kind of architecture in the beginning
there was the rails monolith right everybody knows about these things so at some point we realize this is unsustainable so we start our breaking a little services out of it right has anybody so far like has organization that work kind of like this you can the space of 4 OK so you follow this path a little bit more and you end up with like some
services trade models is still there because it will never die by its at this stage is what I'm gonna call like SOA
service-oriented architecture I um OK
great you keep going a bit longer and like this starts happening right and I guess is
what people mean when they say micro-services that's just like more of them in there may be a bit smaller maybe the bigger I don't know but now there's like this term of art called like a a likely Pico services has anybody heard that again on the web that would be like presumably
some terrible terrible come constellation or like even like manner services like the Holy shit
I yeah so it's a vector
this this is kind of where we were at an ally a year ago at 2 years ago you never know and and this is already getting out of hand right so we started to do is figure out a way to make these pieces make a bit more sense and so we
did something like this you group services by broad like role and you can arrange them into tears and you can assign this tier is kind of like uh jobs and some Broadway and I'm coin the term I haven't seen this before I can I call
this like structured services and so this is how we are today were kind of like a lot of services of the microsurfacing but they're arranged in these 2 years and they interoperates in a kind of a specific way so structured services what I mean by
that but I McClendon structures services or as a way in the modern enterprise that's kind of where I see the trend lines going and by structured
services based planning structure PC the services communicate with each other using energy to be days on or like 3 after particle buffers or whatever but fundamentally it's a request-response kind of like paradigm so in our specific case example of the way we have it cut up we
have the top tier of a set of specialized API is basically 1 API per client thing that we serve so we have an API for I was devices in MPI for Android devices for web of for a mobile web in all these sorts of things and there's exclusively concerned with producing uh output that is tailored to that device so if you have a web client you probably want a lot more detail than doing I was client we have different back and to support them the next layer down kind of the biggest layers are business logic layer and that's like producing all of this stuff you wanna see and then at the bottom we have a data model is actually the smallest of all these layers this is the data services that ceramic my sequel instances are Cassandra instances or row she or whatever so consider the
life of a single request you open up the SoundCloud happening that refresher whatever and it comes and maybe lucky that'll do it 1 of these things which has stated that machine and then it is to get some more information here and that does that and then you're here but this is actually totally typical right and just a single call we see in this
diagram we have 1 incoming requests that fed up to 9 services and ultimately 14 our PCs and actually minimal right are most things go up to an order of magnitude bigger than this we this is the universe that we can live in now so let's consider just 1 of these like little RPC things In all get to go on a minute I
promise and so is like a client server but what we care about and this interaction certainly we need metrics on client right the AP metrics since had the same stuff on a server we meta summers like connection on the like information itself both on the client perspective in the server perspective thank we need implementations of things like back request landing on the server side similarly we need like protection guarantees on the client side like circuit breakers at a high level we need of of an ability to cut through different transport layers in the communication channel are we the the higher order contemplate service discovery to like figure out what we're talking to what instances are alive the healthiness the for the whole system we need concepts like request tracing I guess is like d'après it can so we can see how request for the system identify problems this kind of thing and this is like I've I'm adding this like a bunch of stuff here but this is kind like bare minimum stuff right in order to operate this use distributed system we need all these things are but like like client-server libraries basically but so how do companies do it had that modern enterprise do it well as far as I can tell this is how it kind words of Google Amazon a big American like pioneers so the use of a set of languages is uh and sort of their own internal like libraries of not a lot of this is open source but in this middle here you kind of like a hum homogeneity but it seems to me through my lands this modern enterprises have settled on a day JVM stack and then of primarily finagle in Netflix they have the antenna like set of libraries I'm going to put stars nexus SoundCloud because we haven't really gotten all the way there but it's definitely the trend line and this issue really sad actually because when we started this process and here was definitely in the running and go lost right ends Ch the loss since and and scowl 1 and that's kind of sad to me obviously so how did how scholar managed to win this battle of it had a success story with Twitter Twitter to this lake yes for 5 years ago when they had that big of a transition from Ruby to rubyonrail select the current architecture the but it does give reasonable performance for a certain people's definition of reasonable not really mine but uh my voice is small but you get reasonable business domain expressiveness and so from these 1st 3 things you get this last mind sharing people talking about a conferences and you get developers excited about it but scholars obviously not a panacea but there's some really terrible things about the dollar of and there's 1 person in my company that like scholar is used to purity and he's all about it but literally every other developer uses it in anger right scholar has an objectively terrible to watching the compiler is awful like all the surrounding stuff is a really bad but the language design itself is confused by like this mishmash of like academia and industry like trying to reach some sort of consensus and as a result nobody actually understands the language like there's a time of like article copy stuff like a surprising amount even at the highest levels as far as I can tell so make this isn't like the permanent state right there's opportunity for improvement so go to some things right and from where I sit in this like enterprise environment this is what I see is like that goes strongest selling point spread this objectively all cool to in this uh extremely coherent unlike considered languages I I can't overemphasize how important the static native binaries are that's a really great for like for many dimensions of because really great efficiency in performance especially compared a scholar I've done in of basically benchmarks where you've computes the memory used and CPU burned per like request processed ends like OK we run some engine processes they when that's like bench mark level 1 go is like typically 10 x 1 order of magnitude greater by good optimize job is usually about 100 acts skulls at 150 X and then like the rubies and the pi bonds are around a thousand x rough order-of-magnitude as so go has a good case here we have really great micro conventions of put migrant reticence when that means and we have a lot of like micro success stories like companies who written blog post removed from X to go and it's like being a great great things of this like builds my this is great this is what we need but it's not all that sunshine and roses goes very young and that actually plays in a lot of people's minds but in my can we have an overemphasis on primitives especially in this environment but there are beholden caveats and I can explain them to other developers right but they still like it's kind of an unsatisfactory explanation I think we all think of the the elephant in the room there but the other ones to I'm in my opinion we don't really have higher order Conventions at the moment but and we don't have any big success stories in the sense of like choosing 0 has allowed my started to have 100 x growth and here the reasons why I'd love to see that sort of thing so what i want have advocate for and what I wanna say that we need is to embrace and extend the of the positive stuff that identified and spend some energy as a community on these negative things so now to like the meets what I think we actually
need I put a star here because this is definitely it is my opinion and you have a kind of a conversation about it but this is my perception of of things that I would make my job a lot easier in environment so
1st of all I think we view of blog posts about request much but I think you know what I mean right like and everybody has a great story about the writing of a request monster or maybe like uh a command line tool that that really cool and wrong it is really cool but we've we've had a lot of these now and I'm interested in something maybe a bit they did but like for events like this property be for example also let's let's spend more time thinking about that sort of thing but if you ever are able to patch together like a interesting a high level framework for the I don't know something you're doing your company that involves multiple go things that would be very interesting to hear about even if it's not like super clean I would love to read that sort of thing I love to read more blog posts about things like composable interface design or specifically the context back into it I think is a really cool thing and like all I really have to go on all I have to point of the people to is excellent but singular of post on the for undergoing langlog and if you're using this in a kind of an interesting way please likely get it out there the of I think we need more like contextual God maybe um operationally informants posts or discussions about things like application instrumentation like if you run go at scalar any language at scale you know you need this kind of stuff but as far as i'm aware there's not like a lot of conversation about this sort of thing in general I would argue we need more like blog was about Elliott patterns and the things that they enable and for me the canonical example is the 1 and Rob Pike gave about his his parser Alexa which I really enjoyed and which pretty much everybody I show 2 has the mind even just like for a minute and that's really great and I think goes on the languages where that the gap between what exists and like having a mind line just a little bit is very small and we can make a lot of like this but I don't think we're doing enough of it OK now community tape stuff but this is kind of near and dear to my heart and I think we need a more coherent give build story right now we exist in this like open source world and that's great and indeed in in my organization we have a saying like we operate as a sort of an enterprise open-source economy which means we have these teams which are responsible for like features and the kind of independent of each other they produce their own my kind of full-stack whatever and some they prove they pushed to their own repos and if other teams when you use that stuff they can but that relationship there is sort of an open source relationship in the sense that I they can be expected to Singapore request when they want new features they can be expected to adapt the code to their own users if they want to do that in this environment actually go model is pretty good um specifically in the context of a build story like I if you wanna go get somebody else's go library and then uh and use it in your own project probably but it's axonal you probably vendor it otherwise you can deduce use it directly that works of external libraries we have kind of circle the drain for a long time it looks like we're ending up is good up Dutch are which means bantering everything and rewriting the import past I think the rocket people do this as well it seems to be that the path of least resistance so this this works OK but like we have this single universe right we control of our own code for the most part I mean that we that that we don't we can treat as a dependency so we we have this universe and it would be great to be able to leverage it but I don't lean on the Google people but I know you also have like here strong known universe if you could give us like some stories about how you leverage that's too like get interesting things and 1st example and thinking specifically of like Kunst controlled lockstep upgrades um refactoring over the whole codebase like this sort of thing like the 1 I do the same thing in our environment um were maybe a little bit uh looser in terms of uh like the restrictions that we have so maybe that opens up some doors but we wanted do this sort of thing right now there's not a lot of clear guidance OK so a coherent build story going back to the other stuff I mentioned what I really wanna sorry for what I what I'm feeling the biggest need for is a higher order framework for what I called structured services so let me actually do a bit of a deep dive into this because I'm this is what I know what's really like moving a lot of my thoughts recently here is the same diagram and this is like a minimal thing but let's look at all these like pieces and let's think about what go packages to serve all these things might look like so but bestowed metrics right everybody needs good metrics I can imagine a package metrics that has primitives like gages Countess histograms of because we have such strong conventions are expected to be all that information be sourced from X Boris the canonical standard library way of putting out of estimating your application code would expects pluggable like exposition other systems graphite I I can't even think of another 1 where some other likely graphing and like metrics stats stearate OK yeah definitely influx here of for example yeah um some pluggable interfaces like that but I think for all these things is important to consider prior art and in this case prior means could held is like the metrics a group of we built actually a monitoring system at some part which is now open source called Prometheus are there's a lot of good thinking merits built by will former Googlers he 1 of the little experience in the open-source world so take a look blog search it it's good stuff but it's a pageant package metrics maybe but think of like the package server obviously wouldn't call it server maybe got something else but it would be responsible for things like declaring in enforcing these conventions that I described about back pressure in this sort of thing the connection pooling like how the handle but not only like the happy path for servers will also when things start to go wrong handling all that in a kind of understood and like non surprising way when you build a distributed system of like a billion services or wherever but you gain a lot of things and to be clear we never go back to the old way we gain so much by having his like independent channels of like distribution and and on deployments in like defining strict contracts and all the stuff we do lose some stuff right you lose like this ability to to understand what the system's doing overall so we need like conventionally strong conventions that enforce and a language layer the package where to make sure that we can
like you make sense of this distributed system that we built so I want I want ways to do all of these things I want multiple strategies for each of them so I can decide with some limits what's important for like my specific use case package assertion should probably published to a service discovery layer in fact it should definitely come issues service discovery layer what that means on into in a 2nd and again prior it's that finagle does a lot of this stuff and quite well I should add our sometimes a lot of places that corre on I think is the Netflix equivalents it's like the server side of the stack and it like sets up a lot of stuff for you basically there others does anybody is a giant carrot myself to them but you know of anything else that fits in this like space no competing company afterward so I'm really curious to learn more actually so what i want actually is to define my service in terms of basically go interface right off have a couple things you can do to it like front of something like this and if it means that if if if integration with this like framework that I'm describing means that I have to like throw some extra stuff and like pass context at the beginning or something I'm happy to do it right i wanna pay the tax I'm willing to lose the interface Puri unwilling to lose some performance cycles of it means I can extract all this extra value in the distributed system on the client side kind of the same stuff declaring enforcing inventions I want the same kind of modules for rate-limiting for circuit breaking is a common nouns that like and everybody is familiar with but I will never be able to do that did guy declaratively with lawful strategies the climb package should should subscribe to the same kind of abstract service discovery thing but then again there's priority and enable the can of capture both sides of the thing we're then is the client side of the Netflix stack of maybe there others as well the talk about service discovery that's like the big big topic in configuration management in clustering and love the law of this it's such a big topic there's so many competing like paradigms right now that I wouldn't deign said like that claim that 1 is the best so I think it needs to support multiple methods from like very simple primitive things like the idea necessary lookups the there is that the centralized model which is consul ready guess zookeeper of there's a hybrid model if you ever heard of this error being the basically has a single authoritative source of information will like as you keep I think it is but the push out information to every host in infrastructure where there are no local load balancer so all the applications to come to talk to other applications they just have to local host and then all the like the routing information is kind of handle for them I consider kind of a hybrid model these are all valid right it all depends on what you're trying to do where you in your organizations like evolution the the but we need somehow support for that in my in my vision and the thing bridging all this together is somehow like some transport package that will take all these idioms client-server service discovery and bridge them to the Flickr fundamental implementations using any number of things right because we need to make decisions at this level too to be brought about thrift Abba art that package RPC would be great if you're dealing appear go kind of world visual be interoperable maybe we can build this with go generate something like this of I'm sure this prior art and I'm sure there's inspiration we had a standard like standard live but I'm not sure exactly what it what it would be there and then when we have all these things what I would love to see personally also is something like a package Inspectorate where we have the system with all these conventions we pay the tax at all of like the lowest levels now we can reap the benefits of that we can get introspection of art like running distributed systems we get deep analysis at a cost and here there's a lot of prior and Google's dapper which I think the name escaped into the world but the implementation didn't I zip kinase is like Twitter is uh attempts at doing a dapper things and even something like JMX of the amazing to have in kind of like a controlled way and I think there's a huge amount of value that the modern enterprise needs and indeed expects that can be accomplished in this way so that's basically it but I was kind of like close on this idea that I think now were like 5 years into this project right and we have so many success stories but I think were kind of In some ways stuck were stuck at this layer of abstraction that we haven't ascended the on and I think it's time to start climbing this latter right this latter abstraction and I think there's a lot of benefits to be had here I don't know what they are I presented my vision maybe you have also a vision but I'm really really really happy and like excited and motivated to top each of you know if you have similar ideas from many form like some sort of working groups we can talk about the sort of thing that some minds together on it and push something like a standard out there and because I think go is like the perfect language for this modern Enterprise SOA kind of environments and I'm really excited to push it in a way that makes it a success there so I mean that's basically my talk so please get in touch and I'm happy to answer questions to FIL B anyone at all yes think of it's a service discovery is a topic of active from consideration we about a year ago pushed um service discovery to the DNS system so we used to necessary records but right now that's only taken so far and right now rolling out basically a consul based implementation and comes also this DNS to drop in replacement and we add more like publish-subscribe stuff on top of that the yeah so centralized model I guess as I described yes the make and so at the moment we don't break and that's yes sorry EU Interop between finagle services Ingo services and at the moment we don't unfortunately but that's simply a goal I mean ultimately they talk using a protocol like currently is almost exclusively http days on a problem but we don't get like the the can base request tracing and I want to see in a go framework exactly that kind of stuff exactly that kind of interop I wanna build it right but I need help ideas the that the OK cool thanks again poly hanging around the the