Go at CoreOS

Video in TIB AV-Portal: Go at CoreOS

Formal Metadata

Go at CoreOS
This session will discuss using Go to build products that make distributed computing as stress-free as installing a Linux distribution
Alternative Title
Go - CoreOS
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Year

Content Metadata

Subject Area
Axiom of choice Building Context awareness Run time (program lifecycle phase) Serial port User interface Distribution (mathematics) Code State of matter Multiplication sign Source code Coroutine 1 (number) Stress (mechanics) Mereology Public key certificate Proper map Neuroinformatik Formal language Core dump Flag Endliche Modelltheorie Data conversion Descriptive statistics Physical system Scripting language Overlay-Netz Wrapper (data mining) Binary code Data storage device Maxima and minima Bit ACID Unit testing Lattice (order) Measurement Product (business) Hand fan Tensor Type theory Data management Arithmetic mean Repository (publishing) Self-organization Right angle Pattern language Writing Point (geometry) Ocean current Slide rule Implementation Server (computing) Mobile app Freeware Service (economics) Computer file Link (knot theory) Cone penetration test Gene cluster Mass Product (business) Revision control Centralizer and normalizer Goodness of fit Authorization Operating system Software testing Standard deviation Distribution (mathematics) Demo (music) Key (cryptography) Gender Cellular automaton Projective plane Counting Line (geometry) Cartesian coordinate system Word Maize Algebraic closure Blog Factory (trading post) Table (information) Communications protocol Resolvent formalism Library (computing)
Distribution (mathematics) Electronic mailing list Insertion loss Term (mathematics) Product (business) Message passing Error message Software Source code Pattern language Communications protocol Routing Library (computing) Physical system
Polar coordinate system Freeware Distribution (mathematics) Streaming media Client (computing) Stress (mechanics) Term (mathematics) Product (business) Voting Population density Source code Quicksort Message passing
Server (computing) Run time (program lifecycle phase) Distribution (mathematics) Multiplication sign Demo (music) Similarity (geometry) Price index Function (mathematics) Mereology Semantics (computer science) Revision control Voting Goodness of fit Mathematics Very-high-bit-rate digital subscriber line Energy level Message passing Proxy server File format Sampling (statistics) Electronic mailing list Streaming media Client (computing) Instance (computer science) Term (mathematics) Entire function Product (business) Software Right angle Pattern language Table (information) Communications protocol Spacetime
Freeware Service (economics) Distribution (mathematics) State of matter Demo (music) Execution unit Virtual machine Stress (mechanics) Client (computing) Correlation and dependence Centralizer and normalizer Operator (mathematics) Flag Curvature Condition number Physical system Demo (music) Cross-platform Projective plane Variable (mathematics) Product (business) Radical (chemistry) Word Data management Integrated development environment Right angle Freeware Sinc function Library (computing)
Freeware Software Distribution (mathematics) State of matter Aliasing Demo (music) Virtual machine Right angle Stress (mechanics) Product (business) Library (computing)
Scheduling (computing) Distribution (mathematics) Multiplication sign Decision theory Demo (music) Execution unit Stress (mechanics) Client (computing) Mereology Coefficient of determination Mathematics Uniformer Raum Bus (computing) Source code Information security Physical system Overlay-Netz Service (economics) Logical constant Building Software developer System administrator Bit Instance (computer science) Statistics Thread (computing) Product (business) Radical (chemistry) Message passing Process (computing) Right angle Physical system Alpha (investment) Row (database) Server (computing) Service (economics) Supremum Computer file Link (knot theory) Virtual machine Gene cluster Power (physics) Cylinder (geometry) Telnet Lie group Message passing Maß <Mathematik> Execution unit Forcing (mathematics) Weight Client (computing) Login System call Word Film editing Software Fingerprint Window Library (computing)
Boss Corporation Distribution (mathematics) Direction (geometry) Virtual machine IP address Theory Product (business) Medical imaging Software Semiconductor memory Personal digital assistant Telecommunication Right angle Table (information) Computer-assisted translation Fingerprint
Suite (music) Freeware Table (information) Distribution (mathematics) Virtual machine Stress (mechanics) Open set Kernel (computing) Series (mathematics) Physical system Routing Window Overlay-Netz Default (computer science) Gateway (telecommunications) Demo (music) Data storage device Instance (computer science) Cartesian coordinate system Product (business) Exterior algebra Kernel (computing) Computer science MiniDisc Figurate number Table (information) Arithmetic progression Library (computing)
Frame problem Link (knot theory) Distribution (mathematics) Computer-generated imagery Virtual machine Code Bit Line (geometry) Product (business) Error message Broadcasting (networking) Source code Configuration space
Frame problem Server (computing) Distribution (mathematics) Server (computing) Computer-generated imagery Binary code Code Streaming media Bildschirmtext Term (mathematics) Measurement Product (business) Error message Broadcasting (networking) Source code Collision Message passing
Building Distribution (mathematics) Code Multiplication sign Stress (mechanics) Parameter (computer programming) Mereology IP address Formal language Medical imaging Bit rate Analogy File system Vulnerability (computing) Namespace Software developer Structural load Binary code Product (business) Electronic signature Arithmetic mean Repository (publishing) Right angle Domain name Trail Game controller Freeware Service (economics) Ripping Open source Patch (Unix) Computer-generated imagery Virtual machine Mass Computer icon Product (business) Hacker (term) Googol Data structure Traffic reporting Alpha (investment) Default (computer science) Distribution (mathematics) Standard deviation Key (cryptography) Interface (computing) Projective plane Interactive television Code Directory service Line (geometry) Cartesian coordinate system System call Kernel (computing) Pointer (computer programming) Blog Factory (trading post) Limit set Video game Routing
today good to go the right so welcome I'm Kelsey Hightower and I would get growers everyone do a big bro short demonstration of less well actually go anchorless and so really quickly when its current was core chorus Inc the company that attempts to make money for venture capitalist so we toss projects actually free and clear and farming projects are the operating system of was itself which is a minimal Linux distribution and then from containers right so there's no package manager and we have a couple of problem projects at the D N rocket that's are of newest project and our goals are to bring distribu computing to the masses using when I explain some art are companies role in the IT Eagle system is to provide global like infrastructure that you can download it right so that's the that's the goal so today I'm going to talk about a couple of our key projects and a little bit about how we write go code from the good thing core OS all Rocco based on and go so we do a little bit of C C + + on the operating system sigh that every other project every single project is written go there is no debate no choice no meetings is going to be go whether you like it or not so many teachers they're missing try to contribute to the stand live or some other goal package so kind of makes a nice knowing that everything that you open up is going to be in a single language when we do write code we stick to the most external libraries of usual poor request bringing in like for new dependencies that's gonna be a long conversation we want to make sure there really bring in things that we need there is a mean we don't use any third-party dependencies and we'll see the was that we take advantage of here some everything we do is on get hard so we used it for everything issue-tracking pull requests and we make things simple by just using Travis EI and going testing packages for all of our unit testing so really quick some of things that we do since everything is running go in our OS distribution slammed there's no package manager so we try to start doing link all of our go binaries to the point where we use the go native resolver and we don't get that take advantage of things like SSL certificates being loaded by other go run time so sometimes we stuff the men in for shipping containers will buy are SSL search so that way as a cell tensor work so other people into parsing the stars they're going linking the go binary Federalists server they realize they can validate as a selling 1 for the couple church there but we solve by by mountain in our research that ROS or actually packing certificates in the go binaries that we need the next thing we do that do a lot of people crazy we don't we have a lot to make files but we do have bills scripts and test scripts and this is mainly for people there new ago the project and they don't have go past so we tried to account for that by having a wrapper script so how build it would do things like go get things that they need before I'm actually done project but 1 of our roles is you at the Scripps it has to be go get a book right you can't count on weakening go gettable so go get this to work the standard to willingness to work and then you go I wrapper scripts around that if someone was just type for slash spilled and another thing is we've gender all of our dependencies using go right so we check in all the source and not just the description father got provides but all the actual dependencies that those parties that depend on this is important too because sometimes people get tired of been open-source contrarily delete the general repository and you find out a deployment 1 day you will be sent when we use a dashed our flag went to rewrite imports I personally am not a fan of rewriting the import Urals because what's really ugly in within AD like always go a past and is really tricky when you have multiple projects using gold at new dependencies also use go that year and is is it's a maze of who actually is authority for the important so tend to stick to the role of your main package uses go depth in all your library just assume that the caller or the the main binary will resolve the dependencies in the ones that you depend on and right now the legal system is young so most of these libraries are at just 1 Nadeau so maybe a couple years from now when we start seeing version 2 of things wall dependency resolving actually be a pain from so for now to recommend using go debt that was the offline if you can help so I said all of our projects in Rn in go and we actually try to stick to where we go from tells us we don't write jobless thou go code we try to write things the way you see in the standard library and we have to spend a lot of time making sure that were writing code of the best way the community you wanna see it as a reason for that is when people contribute to go project you can expect the co-based so look a certain way and you see a factory pattern would like the word factory that user you closure text immediately and move on to another project contribute to an we were prevent as much as possible on so here's some of our key projects and you go to that our our or organization get help you the rest of our projects to argue now a stepped through are projects measure shows small demo of each of them the show how they work and what she libraries we depend on to build which projects by something a lot of people and new to go you hear people using go for everything we'll actually see the thing that the building right to try to deliver that today so the 1st projects um 1 or more proper project and at C so at C D is slide the guess diverse winners persuaded the chubby cone so it's a key-value store de-centralized has built-in in AJ use in the rap protocol and provides a distributor or centralize lock service for your apps and all our applications take advantage of STD in some way for we have a overlay network told them and some that's for cluster restore this routing table and NCD this simplifies the applications that we have to build the key packages in NCD is version the new version we just release of last week actually to 0 and we leverage from poor about heavily for on the RPC serialization between our the wrath nodes in the active clusters all we use a context model which actually print the co-based quite a bit if you have a at the blog post on the context modelling should check it out easy way of passing you no shared state context and things like that lines in cancelation to other go routines actually do the heavy lifting to the entry point of the the and go raft so we we implement another wrath implementation we start with go rest want a lot lot NCD servers caught fire in production and we found that our that implementation was not of 2 parts we wrote it and we have the standard RAF library that's being used by other projects as well so what I do is give people a quick demo of what this looks like if you have any questions feel free to interrupt I don't guarantee them as the answer them but you surely welcome to ask so at normally run and they're like
a 3 no consideration and use the rest protocol to basically oral until a reader have 1 mission goes down it will automatically hold an election and find the next node so using those key libraries there and we can bootstrap on an entity node to understand which dropped and what's happening
here is this listening RBC messages to a list of its appears in the network but it can't find the other 2 so right now is going to spend these messages which is use a simple go warning package weakening gave up on complicated patterns like you all just long to stand there and we were tools like system the rout those loss to a central place so no more debate you know how long should be formatted
just as the next node and what we end up
here is um now that the rat has established we have our PCs concerning could confirming that we have a quorum in the cluster and now are some 3 node clustering those only to those are present are actually raised start taking work and will start the 3rd
node Haiti have actually heard density I you are similar to do it at the the there's sort of thought would you like entities cool and we do that by our largest install show my friends
like that's a good thing and it's very impressive right so on the sample a proxy and similar things that you can do that C D some of the patterns in a lot of people like to do are that's the we should is actually working the price a found so we can actually do is watch for cheese this and watch for key food so what happened is in this pattern if you're doing 1st is like and finally we have a new node that come onto the network and allocates the self some that need to update the routing table for all the instances in the cluster so what they'll do is come up and watch particular key what we can do is write to that key 2 TAO and then we'll set the food to borrow and we should see also output on the other hand that's typically what we see a lot of people do be 1 the 1st now does have a way of basic communicating onto the cluster having consensus means that all the other servers will agree on actual value this semantic also works for doing locks or leader election you have like a PostgreSQL master-slave you counted out which 1 is going to be the master at runtime they see reality race a lot to that key right the value become amassing of along with stuff them and then 1 thing is to be tries to
handle for you if Amazon deletes when your ancestors without telling you and it will go through in elect a new master and ideally if you're using the entire list of nodes inside your array of tools to connect to them then a year of right agrees Wardrop you'll attempt to roll over to the next node and once it comes back and recover so that's SED at a higher level we using part above space need to we at Berkeley last time by using just typical rest Jason format and then the API Version change and that's really painfully in the basic packing B 1 B 2 and carrying along for robustness that much easier to deal with backward compatibility of protocols stories Postbus going forward we also saw some performs increasing on the summarisation you can imagine doing with Jason at high speeds is culture of right so that's
entity by the next project that we have this fleet and fleets since in the in corr was you don't have a package manager there's no human FT which refundable Leicestershire so expect people to interact with a cluster of from central place and have a centralized them and so what fleet is it's are distributed system the mainly because you interact with the with unit files that you assume typical Martin Fowler Institute and fully also provides in access to that that particular cross international use SSH to happen to all the nodes based on his machine ID and here the key library that we use here so for it is another tool is built on top of NCD to synchronize the state of the cluster and global cough and go I and I think of those are really powerful libraries that you can run command line tools and you have the same flags a service environment variables for people that don't like scene along with the flags and terminal and India crypto SSH package is a godsend especially the China support do shows while operations like when those words really a pain guesses edge of ingoing using their economies fleet cross-platform and where you have to ask you that it will install SSH we can use a built and for the client API and fleet we actually use a goal goals light API spec we you kind of finding Jason and then generate the client libraries from that so that's been really helpful even though were all using it to generate our go class libraries so on to do of free library in Ruby they conditions the same tools so I'll give it quick
demo of a fleet really quick and you have heard of fleet yes use I just heard of it in a way that works the right so and he gets Chen
right so we can talk to the other
entity cluster here so I have a multi-node cluster on that whole the does not rely on the network is that's going to go badly so when we do now is interact with the actual things up ISO fleet is actually running and what I can do is use lead to interact with the state of the cluster the so these are all the machines in the cluster so using the ego SSH library we can actually SSH into these notes by the machine ID and and fleet tries to do the right
thing 0 look at that security and what do people
do when faced with this message I was appointed this all everyone does lies like yeah
whatever like still my bank account records I so we can as decision to the server using fleet but for each row powers the scheduling work into the clusters so the really simple scheduler that allows you to give a unified out and tries to do the right thing to find the right server to run your resources so do that I can give it a unit file so here's memcached anyone that have worked it is sucks there so here you can actually blame Ali yesterday that 1 out so what we can do if we reduce fleet to our large memcached cylinders just stop this unit the very thing works and don't can should actually start running on 1 the nodes in the cluster so flee actually figured out that a node and 140 is actually capable of rains particular unit so what this unit looks like is a Michigan download a memcached Dr. container wanted binding to the same port is the host and can you sleep to list all the units running on all the machines in the cluster In my terminals that but you can kind of see that and then cashes actually running and if that's actually true we should be able to use my trusty dusty non-cash client to login a star interacting with memcached 1 1 2 1 1 the yes and then yes so when cash is actually running and the chasing do the right thing if I'm 1 of the nodes goes down and it should move that particular job force right so we have it running on 140 of when you're new hires kind gets mad that he didn't get a raise and he he let you know about its rights and and that's down on your page in the bill goes up a little bit and then the tries to do is find a new home for that instance and it seemed to have done so quite quickly and ideally we can verify really quick that that actually works in yes it does to Leslie out and not shown and we basically do leader election because the actually want all the machines that we only need 1 master uses entity on the back in to do that why semantic if 1 the nose dies in those needs to take word's work the is right so flannel so finals are other part that we built and this 1 cutting gets into some low-level stuff we use a syscall package quite a bit the and final homage overlap people crazy look we found in that Link Library that pretty much solved a lot the problems that we want to do 1st is you wanted to have the x land to do our overlay and most the work was already done in that and net Link Library we used to go system the library mainly because we need other services to wait until files actually up and running before they start and 1 way to do that is by using something like system the and using St notified the supremum message on the bus to communicate to the other services that you're actually up and running and prepared to do work I'm having people known overlay network is and why you need it has a window could demonstration were filed does so in that it came out everyone got really really excited special developers on a single machine it was like the world change and they rented abstinence and put back on every machine so we did the and once we did that become we we had a harsh reality that was to nodes across machines was not thing that actually worked really easily so well and you here show the reason that our final exist so we actually have final running on the servers right now and were willing to attempt to do here is into 2 machines and they straight cross container networking so that 1 can tell can talk to another container and other sound really really simple but try song that is using dogs for the 1st time the travel will go nowhere in the lab as the hearer's we try to figure out how to use TCP them right so here we go well on into a 2nd container at this ID so again these 2 machines yes security does anyone actually stop when they see our printed they don't have
committed to memory short you remember your fingerprints fishery issue rebuild deserves more often I don't really know what the guy suck a doctor runs or in do really quick is we're going to Austin busy boss container and warranted do here is just give a quick show and they were going to do Bashkir Jennifer remember how I want to do i have configured ETA 0 and if that works in O if that were start netcat a listening on everything that you can find on 4 80 there's a lot about right show the ship it right so this container got this IP address so basically we go to the other node and we
do the same thing and you wanna you BusyBox and I'm cheating here because I have all my doctor images protest the thing that's like really we should be doing if you don't have greater than 1 or your and hotel the right so in theory 1 show again and then will attempt to the ground that cat the and we want to know a command the people who to too complicated there we go in and this works very well right we have this bike directional communication now it's simple but trust me that is golden for a lot of people that is were for containers about a bunch of us and a nice thing about this is I'm using them in this case I'm using found to manage these Democrats and the machines so whenever a new holes would join this network we automatically get a table update it so we don't have to do any more
consideration and this about to start rusty all the other holes for Petipa subnets and the routing table story on NCDs of the machines crashed start final back up everything that can happen so that's final and it deals with the low-level
kernel stuff to make all of your overlay networks just do what they're supposed to do this would be and the last thing to talk about is rocket so Rock is a command-line tool and go up we promise you the standard library for that as well and we leverage the open tgp package pretty extensively like that packages pretty great you know can build a whole CA with that some then we use it made to build a tool or system that we can actually verify the application instances the containers that rock will actually run so easily turns signing on by default so we attempt to download your application of container it is not signed we blow up and we built a nice little key stored as high based on the actual name of the in which that were downloading reason national talk I'll progress so if you're sitting there waiting for something big the download is really handy package to make you US people smile and the disk the package from Peter is fantastic and we use it for content-addressable store if you want to give computer science on go to Wikipedia and figure out what that is and then do a quick demo of our what racketeers had this people know Iraq is Suite is our attempt to devalue doctor and make the the series of turn to alternatives right so I'm going to larger sick pick
a machine so what I wanna do here is just have rockets and all of a remember these commands so I have a rocket run a container that's hosted on my Mac 1 and His name is hello were released at 0 . 1 the Linux and the 64 ACI who has line that was the 1st no confidence in either
it's yellow then search artist eschew little bit here and see what actually
there has to be in static so measure using a go server to actually holds all the standard binary so this is the gun in was
enough which was an you but close and the can't that the the yes year
aren't happy again the understood this is your paying attention so where did you guys those following rate attentive the this there you go so my little hello applications running I'm using the go PGP package to actually validate the signature and the basic turned a file system into a big teaching to move into this how call magic and Ophuls ship a vendor trusted keys that you can override some by looking at the story this control see this the so in all of our own the just a keys can live in a directory structure looks like this in the oxygen to them by domain name or IP addresses to trust only certain keys for images stored in certain places but using those packages and took like a day to build this Petipa feature interact and enabled by default the right so that's rocket the so I think about the developers at the Croats allowed them are alike kernel hackers back in guys so when they won't go for the 1st time the benefits are you don't see a lot of functioning factory we do see things like that mass being used for function arguments right that is why not thing doing goes values of the and see what In the goal world bills of these run into but usually not that have to do with you know we use the dual nice review and they move on with it but most of the developers at that grow pretty much come from these particular languages so picking up go from them is user like a week worth the effort innovative the rights code a couple call reviews and they're off on their own a preacher we can cut a few days off of that by pairing 1 day the syntax straight simple that you is not much to learn none of crazy styles in our company chooses news but we do have a couple of gripes so there's like 1 guy will mention his name he has like this blog post is been writing for like 3 months about white girls sucks and as all of good of that you see in Hacker News whenever there's a new release of go right inevitably generic but things like shadowing bite the hell out obtained by Kiyoko continues the work and then someone finds out that it doesn't have to behave in the way you think because your showing a variable come and sometimes people are as productive but that's with a new language with that you write your master hacker and see you move on to go and then you find out you can do all that funny pointers stuff that you're doing passion the world service and then the large part growers as coulombs go like all limited to the challenges so distributions it to lag rip all your dependencies outs and repackage them as separate packages and then reassemble everything on the fly which doesn't work well at all in the go well and we've actually was that of now I'm getting you 1 that also you guys I wasn't stop that and is a new worlds and out for so what is it because of the time so our edges have essentially by years and when talking as part of this and other because I was they all of our partners got really quick adoption because like a standard binary Rachel user download this thing and just use it and then in the demining world is life at India load that all package that an icon of grateful for years and the your stuff so that's 1 thing they were Janet we're trying to be nice about a by restocking meaning with our dependencies are so good that helps their given that Jason manifests and what hasn't been this is we have but I'm not sure how that'll work out I need this particular I want this other packages as other shall 1 and have a huge repository of stranger than try to repackage all those commits and then again a new language for developers and finally edited then we love go so most of us don't even think about it right we start a new project little side things were just trying experiments go is makes a pretty easy even to stuff of think aloud where c doesn't values to the Cisco interface don't seem get tripped up at any time we have a major bond that we went to we did pretty good turn around for the next release of go patching it in 1 of the main features like of user namespaces for from the can work that we want to do of fusion over the required by Dr. analog people using go out to build a lot of this tooling and that patch landed and were able to move on so that's cross-set go and if you have any questions on hypertensive thank you if it would be and so I'm and you talk about that and right to the the it can be used to find a the 1st thing that you all I think that the so if you use it you yeah I mean using Lemma my their machine ideas you say ignore all that host checking their production yes you're right we should pride really doing no validating those things especially your mind because understanding the you know and you know the because of I'm think all of us are free and I have a few patches that landed and we don't were not prohibited from doing during company time and adding is because of our open-source route all of our part open source so any time we have something affects just go fix it properly upstream and yeah so that on the time the the the question the what I think for the poppy the report so it really is very limited set of dependencies to begin with so that makes it easy to come to keep track of things and there are any issues like an additional we discover new alpha release that rebuild all those packages so nicely about building go packages is straight simple to do so if we need data dependencies using go that will block of the particular commit that we need to go that save push build and then do a new release and then doing an announcement on but I haven't seen a lot of security vulnerabilities in some of these third-party packages the price there is about cool I will thank yes