Bestand wählen
Merken

Go and the modern enterprise

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
Due to due OK so I 1st of all data coming that's like goes been around for 5 years now I think the 5th birthday of uh an amazing language and its output we've been using it for a long time I think actually pretty 1 where release for some internal tools so today I wanna talk about basically go in the context of companies like SoundCloud which I'm calling the modern enterprise so 1st I want to talk about what I mean by that what is a modern enterprise wide it's important I am going to talk about my experience that SoundCloud specifically like our product evolution and infrastructure evolution I'm a talk about how I think that generalizes to other companies in our space and so I'm going to talk about what I think go needs to succeed in that space and so how we've kind of like in some ways missed the boat unfortunately in what we can do to make up ground so 1st of all the modern
enterprise I what I mean this is what I
think it is it's companies that are somehow tech-related I probably consumer-focused but they almost certainly have achieved hockey stick growth In our their their needs lives they probably have at least 100 engineers so a sizable company and probably they also have what we know is like a service-oriented architecture a company doesn't have to take all the boxes but probably most so what is in companies that kind of spring to mind anything of the sort of thing this is what it is for me so obviously Google and is quite large may be too big I would like to come across some of the lists that Amazon is kind of in maybe the same way like maybe there have been too big to be considered in this space and then there's a whole class of companies like Etsy which actually operates basically monolithically like not in SOA style so I'm going to cross them off the list as well and that leaves like companies in this space I think of they share a lot of properties they are pretty big Twitter's I think the biggest of these leading efforts is bigger now actually but I think they're like 15 hundred engineers or something like this but some cloud were much smaller were like 120 or something like that I but these are like what I consider to be the big players but it's not by any means an exclusive list if you kind of drop some of the requirements is a lot of companies and the ones that I know he's go the kind of fit into this space so this is a type of organization I'm talking about and I guess I would emphasize that there I kind of driving a lot of conversations in the tech world and in my view of a lot of technology decisions being made by these companies or these companies are can be replicated by other companies and that's the thing that I can I wanna talk about today Saloni talk
specifically about my experience at example of and this will be kind of a pretty common story if any of you of will a familiar with this kind of architecture in the beginning
there was the rails monolith right everybody knows about these things so at some point we realize this is unsustainable so we start our breaking a little services out of it right has anybody so far like has organization that work kind of like this you can the space of 4 OK so you follow this path a little bit more and you end up with like some
services trade models is still there because it will never die by its at this stage is what I'm gonna call like SOA
service-oriented architecture I um OK
great you keep going a bit longer and like this starts happening right and I guess is
what people mean when they say micro-services that's just like more of them in there may be a bit smaller maybe the bigger I don't know but now there's like this term of art called like a a likely Pico services has anybody heard that again on the web that would be like presumably
some terrible terrible come constellation or like even like manner services like the Holy shit
I yeah so it's a vector
this this is kind of where we were at an ally a year ago at 2 years ago you never know and and this is already getting out of hand right so we started to do is figure out a way to make these pieces make a bit more sense and so we
did something like this you group services by broad like role and you can arrange them into tears and you can assign this tier is kind of like uh jobs and some Broadway and I'm coin the term I haven't seen this before I can I call
this like structured services and so this is how we are today were kind of like a lot of services of the microsurfacing but they're arranged in these 2 years and they interoperates in a kind of a specific way so structured services what I mean by
that but I McClendon structures services or as a way in the modern enterprise that's kind of where I see the trend lines going and by structured
services based planning structure PC the services communicate with each other using energy to be days on or like 3 after particle buffers or whatever but fundamentally it's a request-response kind of like paradigm so in our specific case example of the way we have it cut up we
have the top tier of a set of specialized API is basically 1 API per client thing that we serve so we have an API for I was devices in MPI for Android devices for web of for a mobile web in all these sorts of things and there's exclusively concerned with producing uh output that is tailored to that device so if you have a web client you probably want a lot more detail than doing I was client we have different back and to support them the next layer down kind of the biggest layers are business logic layer and that's like producing all of this stuff you wanna see and then at the bottom we have a data model is actually the smallest of all these layers this is the data services that ceramic my sequel instances are Cassandra instances or row she or whatever so consider the
life of a single request you open up the SoundCloud happening that refresher whatever and it comes and maybe lucky that'll do it 1 of these things which has stated that machine and then it is to get some more information here and that does that and then you're here but this is actually totally typical right and just a single call we see in this
diagram we have 1 incoming requests that fed up to 9 services and ultimately 14 our PCs and actually minimal right are most things go up to an order of magnitude bigger than this we this is the universe that we can live in now so let's consider just 1 of these like little RPC things In all get to go on a minute I
promise and so is like a client server but what we care about and this interaction certainly we need metrics on client right the AP metrics since had the same stuff on a server we meta summers like connection on the like information itself both on the client perspective in the server perspective thank we need implementations of things like back request landing on the server side similarly we need like protection guarantees on the client side like circuit breakers at a high level we need of of an ability to cut through different transport layers in the communication channel are we the the higher order contemplate service discovery to like figure out what we're talking to what instances are alive the healthiness the for the whole system we need concepts like request tracing I guess is like d'après it can so we can see how request for the system identify problems this kind of thing and this is like I've I'm adding this like a bunch of stuff here but this is kind like bare minimum stuff right in order to operate this use distributed system we need all these things are but like like client-server libraries basically but so how do companies do it had that modern enterprise do it well as far as I can tell this is how it kind words of Google Amazon a big American like pioneers so the use of a set of languages is uh and sort of their own internal like libraries of not a lot of this is open source but in this middle here you kind of like a hum homogeneity but it seems to me through my lands this modern enterprises have settled on a day JVM stack and then of primarily finagle in Netflix they have the antenna like set of libraries I'm going to put stars nexus SoundCloud because we haven't really gotten all the way there but it's definitely the trend line and this issue really sad actually because when we started this process and here was definitely in the running and go lost right ends Ch the loss since and and scowl 1 and that's kind of sad to me obviously so how did how scholar managed to win this battle of it had a success story with Twitter Twitter to this lake yes for 5 years ago when they had that big of a transition from Ruby to rubyonrail select the current architecture the but it does give reasonable performance for a certain people's definition of reasonable not really mine but uh my voice is small but you get reasonable business domain expressiveness and so from these 1st 3 things you get this last mind sharing people talking about a conferences and you get developers excited about it but scholars obviously not a panacea but there's some really terrible things about the dollar of and there's 1 person in my company that like scholar is used to purity and he's all about it but literally every other developer uses it in anger right scholar has an objectively terrible to watching the compiler is awful like all the surrounding stuff is a really bad but the language design itself is confused by like this mishmash of like academia and industry like trying to reach some sort of consensus and as a result nobody actually understands the language like there's a time of like article copy stuff like a surprising amount even at the highest levels as far as I can tell so make this isn't like the permanent state right there's opportunity for improvement so go to some things right and from where I sit in this like enterprise environment this is what I see is like that goes strongest selling point spread this objectively all cool to in this uh extremely coherent unlike considered languages I I can't overemphasize how important the static native binaries are that's a really great for like for many dimensions of because really great efficiency in performance especially compared a scholar I've done in of basically benchmarks where you've computes the memory used and CPU burned per like request processed ends like OK we run some engine processes they when that's like bench mark level 1 go is like typically 10 x 1 order of magnitude greater by good optimize job is usually about 100 acts skulls at 150 X and then like the rubies and the pi bonds are around a thousand x rough order-of-magnitude as so go has a good case here we have really great micro conventions of put migrant reticence when that means and we have a lot of like micro success stories like companies who written blog post removed from X to go and it's like being a great great things of this like builds my this is great this is what we need but it's not all that sunshine and roses goes very young and that actually plays in a lot of people's minds but in my can we have an overemphasis on primitives especially in this environment but there are beholden caveats and I can explain them to other developers right but they still like it's kind of an unsatisfactory explanation I think we all think of the the elephant in the room there but the other ones to I'm in my opinion we don't really have higher order Conventions at the moment but and we don't have any big success stories in the sense of like choosing 0 has allowed my started to have 100 x growth and here the reasons why I'd love to see that sort of thing so what i want have advocate for and what I wanna say that we need is to embrace and extend the of the positive stuff that identified and spend some energy as a community on these negative things so now to like the meets what I think we actually
need I put a star here because this is definitely it is my opinion and you have a kind of a conversation about it but this is my perception of of things that I would make my job a lot easier in environment so
1st of all I think we view of blog posts about request much but I think you know what I mean right like and everybody has a great story about the writing of a request monster or maybe like uh a command line tool that that really cool and wrong it is really cool but we've we've had a lot of these now and I'm interested in something maybe a bit they did but like for events like this property be for example also let's let's spend more time thinking about that sort of thing but if you ever are able to patch together like a interesting a high level framework for the I don't know something you're doing your company that involves multiple go things that would be very interesting to hear about even if it's not like super clean I would love to read that sort of thing I love to read more blog posts about things like composable interface design or specifically the context back into it I think is a really cool thing and like all I really have to go on all I have to point of the people to is excellent but singular of post on the for undergoing langlog and if you're using this in a kind of an interesting way please likely get it out there the of I think we need more like contextual God maybe um operationally informants posts or discussions about things like application instrumentation like if you run go at scalar any language at scale you know you need this kind of stuff but as far as i'm aware there's not like a lot of conversation about this sort of thing in general I would argue we need more like blog was about Elliott patterns and the things that they enable and for me the canonical example is the 1 and Rob Pike gave about his his parser Alexa which I really enjoyed and which pretty much everybody I show 2 has the mind even just like for a minute and that's really great and I think goes on the languages where that the gap between what exists and like having a mind line just a little bit is very small and we can make a lot of like this but I don't think we're doing enough of it OK now community tape stuff but this is kind of near and dear to my heart and I think we need a more coherent give build story right now we exist in this like open source world and that's great and indeed in in my organization we have a saying like we operate as a sort of an enterprise open-source economy which means we have these teams which are responsible for like features and the kind of independent of each other they produce their own my kind of full-stack whatever and some they prove they pushed to their own repos and if other teams when you use that stuff they can but that relationship there is sort of an open source relationship in the sense that I they can be expected to Singapore request when they want new features they can be expected to adapt the code to their own users if they want to do that in this environment actually go model is pretty good um specifically in the context of a build story like I if you wanna go get somebody else's go library and then uh and use it in your own project probably but it's axonal you probably vendor it otherwise you can deduce use it directly that works of external libraries we have kind of circle the drain for a long time it looks like we're ending up is good up Dutch are which means bantering everything and rewriting the import past I think the rocket people do this as well it seems to be that the path of least resistance so this this works OK but like we have this single universe right we control of our own code for the most part I mean that we that that we don't we can treat as a dependency so we we have this universe and it would be great to be able to leverage it but I don't lean on the Google people but I know you also have like here strong known universe if you could give us like some stories about how you leverage that's too like get interesting things and 1st example and thinking specifically of like Kunst controlled lockstep upgrades um refactoring over the whole codebase like this sort of thing like the 1 I do the same thing in our environment um were maybe a little bit uh looser in terms of uh like the restrictions that we have so maybe that opens up some doors but we wanted do this sort of thing right now there's not a lot of clear guidance OK so a coherent build story going back to the other stuff I mentioned what I really wanna sorry for what I what I'm feeling the biggest need for is a higher order framework for what I called structured services so let me actually do a bit of a deep dive into this because I'm this is what I know what's really like moving a lot of my thoughts recently here is the same diagram and this is like a minimal thing but let's look at all these like pieces and let's think about what go packages to serve all these things might look like so but bestowed metrics right everybody needs good metrics I can imagine a package metrics that has primitives like gages Countess histograms of because we have such strong conventions are expected to be all that information be sourced from X Boris the canonical standard library way of putting out of estimating your application code would expects pluggable like exposition other systems graphite I I can't even think of another 1 where some other likely graphing and like metrics stats stearate OK yeah definitely influx here of for example yeah um some pluggable interfaces like that but I think for all these things is important to consider prior art and in this case prior means could held is like the metrics a group of we built actually a monitoring system at some part which is now open source called Prometheus are there's a lot of good thinking merits built by will former Googlers he 1 of the little experience in the open-source world so take a look blog search it it's good stuff but it's a pageant package metrics maybe but think of like the package server obviously wouldn't call it server maybe got something else but it would be responsible for things like declaring in enforcing these conventions that I described about back pressure in this sort of thing the connection pooling like how the handle but not only like the happy path for servers will also when things start to go wrong handling all that in a kind of understood and like non surprising way when you build a distributed system of like a billion services or wherever but you gain a lot of things and to be clear we never go back to the old way we gain so much by having his like independent channels of like distribution and and on deployments in like defining strict contracts and all the stuff we do lose some stuff right you lose like this ability to to understand what the system's doing overall so we need like conventionally strong conventions that enforce and a language layer the package where to make sure that we can
like you make sense of this distributed system that we built so I want I want ways to do all of these things I want multiple strategies for each of them so I can decide with some limits what's important for like my specific use case package assertion should probably published to a service discovery layer in fact it should definitely come issues service discovery layer what that means on into in a 2nd and again prior it's that finagle does a lot of this stuff and quite well I should add our sometimes a lot of places that corre on I think is the Netflix equivalents it's like the server side of the stack and it like sets up a lot of stuff for you basically there others does anybody is a giant carrot myself to them but you know of anything else that fits in this like space no competing company afterward so I'm really curious to learn more actually so what i want actually is to define my service in terms of basically go interface right off have a couple things you can do to it like front of something like this and if it means that if if if integration with this like framework that I'm describing means that I have to like throw some extra stuff and like pass context at the beginning or something I'm happy to do it right i wanna pay the tax I'm willing to lose the interface Puri unwilling to lose some performance cycles of it means I can extract all this extra value in the distributed system on the client side kind of the same stuff declaring enforcing inventions I want the same kind of modules for rate-limiting for circuit breaking is a common nouns that like and everybody is familiar with but I will never be able to do that did guy declaratively with lawful strategies the climb package should should subscribe to the same kind of abstract service discovery thing but then again there's priority and enable the can of capture both sides of the thing we're then is the client side of the Netflix stack of maybe there others as well the talk about service discovery that's like the big big topic in configuration management in clustering and love the law of this it's such a big topic there's so many competing like paradigms right now that I wouldn't deign said like that claim that 1 is the best so I think it needs to support multiple methods from like very simple primitive things like the idea necessary lookups the there is that the centralized model which is consul ready guess zookeeper of there's a hybrid model if you ever heard of this error being the basically has a single authoritative source of information will like as you keep I think it is but the push out information to every host in infrastructure where there are no local load balancer so all the applications to come to talk to other applications they just have to local host and then all the like the routing information is kind of handle for them I consider kind of a hybrid model these are all valid right it all depends on what you're trying to do where you in your organizations like evolution the the but we need somehow support for that in my in my vision and the thing bridging all this together is somehow like some transport package that will take all these idioms client-server service discovery and bridge them to the Flickr fundamental implementations using any number of things right because we need to make decisions at this level too to be brought about thrift Abba art that package RPC would be great if you're dealing appear go kind of world visual be interoperable maybe we can build this with go generate something like this of I'm sure this prior art and I'm sure there's inspiration we had a standard like standard live but I'm not sure exactly what it what it would be there and then when we have all these things what I would love to see personally also is something like a package Inspectorate where we have the system with all these conventions we pay the tax at all of like the lowest levels now we can reap the benefits of that we can get introspection of art like running distributed systems we get deep analysis at a cost and here there's a lot of prior and Google's dapper which I think the name escaped into the world but the implementation didn't I zip kinase is like Twitter is uh attempts at doing a dapper things and even something like JMX of the amazing to have in kind of like a controlled way and I think there's a huge amount of value that the modern enterprise needs and indeed expects that can be accomplished in this way so that's basically it but I was kind of like close on this idea that I think now were like 5 years into this project right and we have so many success stories but I think were kind of In some ways stuck were stuck at this layer of abstraction that we haven't ascended the on and I think it's time to start climbing this latter right this latter abstraction and I think there's a lot of benefits to be had here I don't know what they are I presented my vision maybe you have also a vision but I'm really really really happy and like excited and motivated to top each of you know if you have similar ideas from many form like some sort of working groups we can talk about the sort of thing that some minds together on it and push something like a standard out there and because I think go is like the perfect language for this modern Enterprise SOA kind of environments and I'm really excited to push it in a way that makes it a success there so I mean that's basically my talk so please get in touch and I'm happy to answer questions to FIL B anyone at all yes think of it's a service discovery is a topic of active from consideration we about a year ago pushed um service discovery to the DNS system so we used to necessary records but right now that's only taken so far and right now rolling out basically a consul based implementation and comes also this DNS to drop in replacement and we add more like publish-subscribe stuff on top of that the yeah so centralized model I guess as I described yes the make and so at the moment we don't break and that's yes sorry EU Interop between finagle services Ingo services and at the moment we don't unfortunately but that's simply a goal I mean ultimately they talk using a protocol like currently is almost exclusively http days on a problem but we don't get like the the can base request tracing and I want to see in a go framework exactly that kind of stuff exactly that kind of interop I wanna build it right but I need help ideas the that the OK cool thanks again poly hanging around the the
Enterprise-Resource-Planning
Formale Sprache
Evolute
Kontextbezogenes System
Biprodukt
Enterprise-Resource-Planning
Raum-Zeit
Funktion <Mathematik>
Quelle <Physik>
Tropfen
Umsetzung <Informatik>
Architektur <Informatik>
Sichtenkonzept
Quader
Kategorie <Mathematik>
Selbst organisierendes System
Klasse <Mathematik>
Mailing-Liste
Dienst <Informatik>
Twitter <Softwareplattform>
Quick-Sort
Raum-Zeit
Eins
Entscheidungstheorie
Digitalsignal
Twitter <Softwareplattform>
Enterprise-Resource-Planning
Datentyp
Quadratzahl
Tropfen
Enterprise-Resource-Planning
Streuungsdiagramm
Serviceorientierte Architektur
Fitnessfunktion
Bit
Punkt
Selbst organisierendes System
Enterprise-Resource-Planning
Dienst <Informatik>
Computerarchitektur
Raum-Zeit
Benutzerbeteiligung
Bit
Dienst <Informatik>
Informationsmodellierung
Enterprise-Resource-Planning
Term
Serviceorientierte Architektur
Bit
Dienst <Informatik>
Rechter Winkel
Enterprise-Resource-Planning
Just-in-Time-Compiler
Vektorraum
Umwandlungsenthalpie
Dienst <Informatik>
Prozess <Informatik>
Enterprise-Resource-Planning
Multi-Tier-Architektur
Gruppenkeim
Dienst <Informatik>
Festplattenlaufwerk
Datenstruktur
Term
Umwandlungsenthalpie
RPC
Dienst <Informatik>
Strukturierte Programmierung
Energiedichte
Puffer <Netzplantechnik>
Dienst <Informatik>
Twitter <Softwareplattform>
Enterprise-Resource-Planning
Programmierparadigma
Partikelsystem
Datenstruktur
Enterprise-Resource-Planning
Gerade
Videospiel
Browser
Mobiles Internet
Datenmodell
Systemaufruf
Fortsetzung <Mathematik>
Humanoider Roboter
Mathematische Logik
Quick-Sort
Virtuelle Maschine
Benutzerbeteiligung
Client
Dienst <Informatik>
Datensatz
Menge
Rechter Winkel
Enterprise-Resource-Planning
Minimum
Multi-Tier-Architektur
Information
Instantiierung
Funktion <Mathematik>
Resultante
Domain <Netzwerk>
Punkt
Prozess <Physik>
Web log
Extrempunkt
Compiler
Minimierung
Formale Sprache
Computerunterstütztes Verfahren
Binärcode
Übergang
Eins
Arithmetischer Ausdruck
Client
Prozess <Informatik>
Enterprise-Resource-Planning
Permanente
Figurierte Zahl
Gerade
Benchmark
Güte der Anpassung
Dienst <Informatik>
Menge
Twitter <Softwareplattform>
COM
Rechter Winkel
Festspeicher
Server
Identifizierbarkeit
Information
Ordnung <Mathematik>
Programmierumgebung
Enterprise-Resource-Planning
Aggregatzustand
Instantiierung
Telekommunikation
Subtraktion
Gruppenoperation
Implementierung
Keller <Informatik>
Interaktives Fernsehen
Dienst <Informatik>
Zentraleinheit
Domain-Name
Polarkoordinaten
Perspektive
Programmbibliothek
Primitive <Informatik>
Softwareentwickler
Grundraum
Binärdaten
Einfach zusammenhängender Raum
Linienelement
Physikalisches System
Quick-Sort
Energiedichte
Diagramm
Digitaltechnik
Energiedichte
Wort <Informatik>
Größenordnung
Computerarchitektur
Distributionstheorie
Umsetzung <Informatik>
Bit
Punkt
Web log
Formale Sprache
Gruppenkeim
Schreiben <Datenverarbeitung>
Kartesische Koordinaten
Ungerichteter Graph
Fastring
Übergang
Monster-Gruppe
Prozess <Informatik>
Enterprise-Resource-Planning
Mustersprache
Gerade
Schnittstelle
Zentrische Streckung
Sichtenkonzept
Kategorie <Mathematik>
Magnetbandlaufwerk
Güte der Anpassung
Systemaufruf
Kontextbezogenes System
Ereignishorizont
Druckverlauf
Dienst <Informatik>
Histogramm
Rechter Winkel
Grundsätze ordnungsmäßiger Datenverarbeitung
Server
BORIS <Programm>
Projektive Ebene
Information
Ordnung <Mathematik>
Enterprise-Resource-Planning
Repository <Informatik>
Programmierumgebung
Standardabweichung
Selbst organisierendes System
Dienst <Informatik>
Term
Framework <Informatik>
Code
Wiederherstellung <Informatik>
Histogramm
Open Source
Informationsmodellierung
Programmbibliothek
Primitive <Informatik>
Grundraum
Einfach zusammenhängender Raum
Kreisfläche
Linienelement
Stochastische Abhängigkeit
Open Source
Einfache Genauigkeit
Gibbs-Verteilung
Physikalisches System
Parser
Quick-Sort
Design by Contract
Patch <Software>
Diagramm
Mereologie
Momentenproblem
Formale Sprache
Gruppenkeim
Bridge <Kommunikationstechnik>
Kartesische Koordinaten
Gesetz <Physik>
Analysis
Raum-Zeit
Übergang
Client
Enterprise-Resource-Planning
Programmierparadigma
Visualisierung
Kontrollstruktur
Maschinelles Sehen
Schnittstelle
Umwandlungsenthalpie
Abstraktionsebene
Stellenring
Ähnlichkeitsgeometrie
Quellcode
Kontextbezogenes System
Entscheidungstheorie
Arithmetisches Mittel
Motion Capturing
Dienst <Informatik>
Konfigurationsverwaltung
Menge
Twitter <Softwareplattform>
Rechter Winkel
Evolute
Client
Server
Strategisches Spiel
Projektive Ebene
Information
Programmierumgebung
Enterprise-Resource-Planning
Message-Passing
Fehlermeldung
Standardabweichung
Schnittstelle
Maschinenschreiben
Server
Selbst organisierendes System
Keller <Informatik>
Zahlenbereich
Abgeschlossene Menge
Implementierung
Dienst <Informatik>
Äquivalenzklasse
Term
Framework <Informatik>
Lastteilung
Physikalisches System
Bildschirmmaske
Informationsmodellierung
Datensatz
Direkte numerische Simulation
Inverser Limes
Analysis
Fundamentalsatz der Algebra
Protokoll <Datenverarbeitungssystem>
Einfache Genauigkeit
Physikalisches System
Flickr
Modul
Quick-Sort
Integral
Modallogik
Dreiecksfreier Graph
Digitaltechnik
Direkte numerische Simulation

Metadaten

Formale Metadaten

Titel Go and the modern enterprise
Alternativer Titel Go - Modern Enterprise
Serientitel FOSDEM 2015
Autor Bourgon, Peter
Lizenz CC-Namensnennung 2.0 Belgien:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/34405
Herausgeber FOSDEM VZW
Erscheinungsjahr 2016
Sprache Englisch
Produktionsjahr 2015

Inhaltliche Metadaten

Fachgebiet Informatik

Ähnliche Filme

Loading...
Feedback