Ephemeral Apps With Chef, Terraform, Nomad, and Habitat
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 45 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/34579 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
ChefConf 201729 / 45
1
4
5
9
12
16
18
20
21
22
24
25
27
28
35
37
39
42
00:00
Mobile appBinary codeService (economics)RandomizationConfiguration spaceComputer fileState observerComplex (psychology)NumberAbstractionOrder (biology)CASE <Informatik>Message passingFile archiverSource codeHydraulic jumpShift operatorServer (computing)MyspaceDefault (computer science)Local ringComputer configurationYouTubeKeyboard shortcutBinary fileOpen sourcePoint (geometry)Computer programmingIP addressHookingBitImplementationLine (geometry)CodeUniform resource locatorMereology2 (number)Ocean currentLoginCartesian coordinate systemRadical (chemistry)Level (video gaming)Slide ruleDemo (music)Scripting languageLink (knot theory)Software developerDirectory serviceGoodness of fitFunction (mathematics)Integrated development environmentPerspective (visual)HoaxPlanningWater vaporSemiconductor memorySupremumWeb 2.0Control flow graphComputer fontSupport vector machineFluid staticsCompilation albumSummierbarkeitSingle-precision floating-point formatInstallation artMaxima and minimaInstance (computer science)JSONXMLUML
08:19
Radical (chemistry)Connected spaceStructural loadDifferent (Kate Ryan album)Demo (music)Core dumpCheat <Computerspiel>Interrupt <Informatik>Multiplication signIntegrated development environmentContext awarenessConfiguration spaceCodeDefault (computer science)Server (computing)Point (geometry)Scaling (geometry)2 (number)Dependent and independent variablesRight angleSource codeCASE <Informatik>MyspaceBitComputer fileExtension (kinesiology)Revision controlFerry CorstenFlow separation1 (number)Order (biology)Streaming mediaService (economics)Term (mathematics)MathematicsPlanningFrequencyCartesian coordinate systemKernel (computing)Moment (mathematics)LoginLastteilungProcess (computing)Line (geometry)Directory serviceSheaf (mathematics)Group actionIntegerMereologyVideo gameBit rateBinary codeError messageHookingInsertion lossBuildingSingle-precision floating-point formatCache (computing)Perfect groupCompilation albumInterior (topology)Mobile app
16:33
Hydraulic jumpGroup actionRight angleConfiguration spaceService (economics)Bit rateClosed setMultiplication signMedical imagingComa BerenicesOnline helpCartesian coordinate systemDefault (computer science)Natural languageCASE <Informatik>Communications protocolSeries (mathematics)Keyboard shortcutImplementationPeer-to-peerAddress spaceLastteilungComputer fileInstance (computer science)Message passingFlagPattern languageParameter (computer programming)Expert systemTouchscreenIP addressAsynchronous Transfer ModeCoprocessorError messageMyspaceOrder (biology)Ring (mathematics)Web browserElement (mathematics)Local ringLaptopRoundness (object)AlgorithmMoment (mathematics)LoginHoaxGreatest elementProcess (computing)Function (mathematics)MathematicsMaxima and minimaData managementType theoryScaling (geometry)Intelligent NetworkTraffic reportingDifferent (Kate Ryan album)IntegerOpen sourceEqualiser (mathematics)OvalContext awarenessProduct (business)Structural loadSubject indexingPrice indexINTEGRALArithmetic meanDynamical systemBoom (sailing)Ferry CorstenCompilation albumFront and back endsDemonRoutingComputer animation
24:48
Binary fileClient (computing)Spacetime2 (number)Service (economics)TrailVideo game consoleServer (computing)Task (computing)Sheaf (mathematics)Scaling (geometry)Product (business)BefehlsprozessorResolvent formalismParameter (computer programming)Different (Kate Ryan album)Variable (mathematics)Performance appraisalDynamical systemType theoryConfiguration spaceWindows RegistryMiniDiscResource allocationCartesian coordinate systemScheduling (computing)MereologyCodeComputer fileOrder (biology)FlagRevision controlSoftware repositoryInstance (computer science)DigitizingEqualiser (mathematics)Integrated development environmentFluid staticsSoftwareOpen sourceMyspaceCommunications protocolMixed realityProcess (computing)Maxima and minimaLatent heatMetadataOverlay-NetzInterface (computing)LastteilungDirect numerical simulationIP addressPeer-to-peerPartition (number theory)Multiplication signLine (geometry)Web serviceImplementationData managementData centerINTEGRALImage resolutionPoint cloudKernel (computing)Default (computer science)Key (cryptography)Data storage deviceFirewall (computing)LoginMappingSupremumNumberRun time (program lifecycle phase)Demo (music)CASE <Informatik>Keyboard shortcutQuicksortFreewareDisk read-and-write headCellular automatonSource codeMedical imagingWordMusical ensembleComa BerenicesBitData conversionPrimitive (album)Right angleGene clusterResultantPhysicalismPoint (geometry)InternetworkingComputer animation
33:02
Client (computing)Resource allocationLink (knot theory)Instance (computer science)Service (economics)Hydraulic jumpProcess (computing)2 (number)MyspaceInformationDemo (music)Point (geometry)Different (Kate Ryan album)WordCASE <Informatik>IP addressBefehlsprozessorSemiconductor memoryAddress spaceBitDynamical systemLastteilungScheduling (computing)Right angleTraffic reportingServer (computing)Natural languageLoginAuthorizationOrder (biology)Subject indexingCartesian coordinate systemLocal ringMereologyMultiplication signRoundness (object)Client (computing)Type theoryPhysical systemLine (geometry)Software testingMathematicsScaling (geometry)Structural loadNumberBinary codePerformance appraisalKeyboard shortcutLaptopHash functionMiniDiscSpacetimeWeb browserState of matterElement (mathematics)CountingMassObject-oriented programmingPerfect groupFunction (mathematics)Matching (graph theory)Configuration spaceTwitterComputer animation
41:16
JSONXML
Transcript: English(auto-generated)
00:05
Hello everyone, how are we doing today? That was really pathetic How are we doing today? All right there we go. I have four slides for you here today. You've seen them They're they're rotating so this is entirely live demo
00:23
That means that at the end of this talk I will also open source all of the code and the script for this so check out my github I'll tweet out a link at the end, but this is a lot of terminal and a lot of text I'm going to assume that you went to either or both of Jamie and Fletcher's talk
00:41
If you didn't I'm going to quickly talk about habitat, but at a very high level So for those of you watching at home on the YouTube channel stop Go watch Jamie's talk go watch Fletcher's talk now come back here Cool, so let me to stop this I wasn't kidding there were four slides And let's go ahead and get started
01:01
Who's excited all right? I was that was okay All right, so is this a good font size for everyone No, okay. I can make it bigger. That's fine Okay, so what I'm going to be doing today is I'm going to be using a demo application and
01:23
No, habitat. No chef. No terraform. No packer. No anything. I want to show you what that application does It's a go binary, so I'm just going to go ahead and build that go binary right now and I run go build and Voila I have a go binary and that go binary is called HTTP echo
01:40
It's a really tiny in memory HTTP server that echoes what I give it so I'm gonna give it some text of like hi and It's gonna start a server listening on a configurable port five six seven eight and if I curl that port Localhost five six seven eight I get back that text okay, and I get a log message that says like hey
02:04
You got that so that's what we're dealing with here today. That was like no extra tooling This is just a tiny little web server This could be a really complex application like like rails or WordPress, but I wanted to limit the number of abstractions So this is what we're working with here today So in order to build this I needed like a local go and Bellman environment
02:23
And I have all that set up because I'm an engineer by trade But you might not be a go developer, or you might not need all of this, so let's jump into habitat So I'm gonna stop this Chef calm zero Okay, so the very first thing I have is we're starting on a journey so this HTTP echo server This is going to be the next myspace, and I'm Tom here to guide you through the journey
02:45
So we have to start out somewhere right we have to start out with our tiny little Application in order for it to be big and be the next big myspace But we need local development first, but for now Let's just pretend that I'm running some off-the-shelf myspace solution Where the static binary is already precompiled, and I just need my own config on top of that, and that's the first habitat scenario
03:06
I want to walk you through so let me take a look at what I have here Inside my directory. I have a habitat folder and inside that habitat folder I have basically two things a plan file and a default dot Toml or Toml depending on how you pronounce it
03:21
And I have a hooks directory, so let's take a look at the plan first So inside this habitat plan I have a number of configurable things in here I'm specifying like my name the packet name and where to download it from What you should get out of this the important part here is this source on line 7 This is telling habitat to go out to in this case github releases and download this precompiled tarball so in advance
03:46
I've gone ahead and compiled this so I don't need go. I don't need Ruby or any nonsense It's already compiled a single static elf binary with no dynamic dependencies And then I'm using the shaw sum on line 9 here just to guarantee that I get the right thing so for some reason the Wi-Fi craps out or if I like I'm subject to some minim attack like
04:06
This will prevent that from happening I'm adding the resulting thing to my bin so this will make it available as an executable binary And then I'm exporting and exposing port and I'll show you what that is in a second But if you remember when I started the service it ran on port five six seven eight
04:22
but I might want to run it on port one two three four or sixty seven thousand one hundred and twenty three and The binary lets me do that and we can actually configure that via configuration file in habitat. I Also have a few things here like the build step and in the build step. You'll notice that it's really complex return zero
04:40
Really really tough stuff there That's because by default habitat wants to build my package, but I'm using a precompiled binary in this example So I just overwrite that and say hey, don't worry about it. I already built it and then in the install step here I am simply moving that compiled binary into the path habitat automatically unarchives it for me So you noticed above it was a tarball
05:01
Habitat will automatically unarchive for it for me put it in the source path and I just need to move it into bin And give it executable permission So what about that port thing? Well if we take a look at the default tommel You'll notice that there's two configuration options here There's the port to bind the service to and the text to render and these are the default values five six seven
05:23
It's the default value for the port and text is hello chef comp with an exclamation point at the end So the last thing that I have to show you then is this run hook The run hook is the thing that actually runs the program inside the supervisor. So this is the thing the supervisor is going to run
05:40
HTTP echoes already in the path as part of the do install step and I'm telling it to listen on a particular port Which is from the configuration and I'm telling it to render some text which is coming from configuration So there's things between those curly braces there the little squiggly lines Those are coming dynamically and being filled in by habitat and the habitat supervisor. So sys.ip is the current IP address
06:03
so I'm binding to the IP address of that instance and Config dot port or CFG dot port and config dot text are coming from that default tommel file And as you might imagine since it's called default default tommel We're actually over able to overwrite those things which we'll get to in a little bit But for now, let's just see what it looks like to run this thing under habitat
06:22
so I'm gonna go ahead and enter my studio and Implementation detail this is gonna like start a docker thing and do some magic darker darker darker And I'm going to build This binary and build is a little bit of an overstatement here because it's actually downloading a tar ball
06:40
Unzipping it and putting it inside of a container done that fast took about six seconds Really really really tough stuff here and there's a bunch of output here, but it's not important What's important is that you think about the high level is I had a pre-built binary on an archive service in this case GitHub releases I downloaded it and now it's in a habitat package and we can run it. So I'm going to go ahead and run
07:04
Habs SVC Any my little cheat sheet here start and it's called Seth of Argo slash HTTP echo and It says the supervisor is starting and we can see that it's running by running sub log or SL for short We can see okay. There's some output there and great. There's the log line from my server
07:23
says it's listening on sys dot IP, which was interpolated to 1702 and it's running on port five six seven eight on the new line there. Thank you line wrapping There now it's on the same line. So this is running and we can curl this URL here
07:41
So we can actually copy this IP and we can curl that and we get hello chef comp and that came from that default Toml file so that text was interpolated in that run hook. It's listening on the port and IP that was configured So let's go ahead and stop the service now, I'm done with it we're moving on with a demo So I'm gonna go ahead and stop the HTTP echo service. That was pretty fast. So let's verify in the logs
08:06
Okay, we got a stopping signal. But what is this? There's this kind of scary log line here unknown signal terminated well, and then it says that it shut it down and I don't know about you but Jamie Windsor and Fletcher told me that SV stands for supervisor, but I believe they named it after me Seth of Argo
08:24
So every time you see SV just know that that's actually named after me So right now notice that that second last log line there error that's coming from my application my go binary Well, what's happening is after some research? I found out that habitat sends term to ERM as the signal to gracefully terminate an application
08:44
Well, my app doesn't respond to term it responds to int in order to shut down gracefully Which is just another Unix signal like usr1 or quit or whatever So I actually have to update my application my go binary to also listen to term Because since this is an HTTP server and it's gonna be as popular as myspace
09:05
I can't just be dropping connections all the time So once we reach massive scale and we hit our critical mass We need to be able to gracefully drain connections and my code already does that I already wrote the code to do that But it only does that when it responds to the right signal in this case Int but habitat sending term. So let's go ahead and exit this studio and when I exit the studio everything's gone
09:26
So let's move on And what I have to do now is in order to change this behavior I'm gonna have to compile from source. So we need to make a few changes to our plan So again, you'll notice that I have that habitat folder, but I also have a source directory
09:41
So I downloaded as much ahead of time in case the Wi-Fi was terrible inside this source directory I have my actual go source. So I have the all the go files in there are the important ones We can actually remove some of these because I tested this earlier Okay, so we have four files in here
10:04
Server version handlers if you don't know go none of this actually matters What matters is I need to change the signals that my application response to and I've done this ahead of time so that you Don't have to watch me type but Down here in this section I added
10:22
This particular line of code or in this case half a line of code So I said in case I receive a SIG term also do a graceful shutdown You can see previously it was just SIG int So now when the habitat supervisor sends term my application will drain the connections and shut down But this is source code and and I let's just pretend that I don't have a go
10:45
Environment ready to go. So how do I build this? Well with habitat we can update our plan To remove all of that stuff for downloading it because we already have the source code here we don't need the tar ball anymore and Instead I'm adding a dependency on what's called core go. So you'll see here. This is a
11:06
package dependency on core go What this is going to do is it's going to build or bring down all the packages that I need to build this as A go binary and run it in a go environment So that means we also need to fill in our do build step So remember before the do build step was just return zero But now we're copying our source code into the cache and building
11:25
So same go build command as before and then our do install step is exactly the same no changes there So we actually net lost two lines of code out of the plan file We added one line of code to do build but we removed three from the top So we're actually net loss two, so this is actually a smaller plan
11:44
But we're pulling in a whole go context now But the process the workflow is still the same and the hooks and everything haven't changed So I'm gonna go ahead and enter my studio This is a different studio from before each time you enter the studio. It's different You'll notice that it downloaded some stuff at the beginning
12:00
It downloaded curl But when I run build it's downloading the go core So that came from the package depths and It downloads all the dependencies that gore cook the the go core needs because we need those to be able to compile from source
12:21
And now it's going to start building our application So it pulled everything down and now it built the application and this one took a little bit longer This took about 20 seconds because we had to compile the thing from source and we had to download some extra external dependencies But now we have the service and we can start it again so we can run have SVC start Seth Bargo HTTP echo
12:41
It's running we can run in the logs and see that it's running. There we go. It just started We can curl it to make sure that it's running so curl Great it's still running and then the moment of truth is did we compile the right thing? So I will have SVC stop
13:02
And we'll check the logs And there we go It got we got a note from our service that said received interrupt and we got a graceful termination Which means my application gracefully exited and it drained connections So if it was under really high load, right if we reach myspace scale And it was under a really high load. My application would drain connections before it terminated
13:25
So we have a well behaved application here and to a certain extent this also illustrates another point Which is habitats a really great tool just like, you know, Docker and rocket and all of these tools But at the end of the day, there are certain things that are application responsibilities This isn't something that habitat can solve for you if you have a server or a service
13:42
It should know how to drain connections when it receives a certain signal. That's just a property of like a well-behaved application so Let's move on to something a little bit more exciting I don't know about you, but if we're gonna be the next myspace we need more than one of these Right because it's go and it can handle but I don't think it can handle myspace scale
14:02
So we need to do some dynamic scaling. We need to really be able to build this thing out So let's check it out So go up a directory to dynamic and in here I have the same source the same habitat directory, but there's a few differences inside this habitat directory. The first is in
14:22
the run hook The run hook solely for demonstration purposes has added a new line So instead of just rendering the config that comes from the default Tomo file It's also printing out its hostname its IP and the port that it's running on And that'll become important because when we put this thing behind a load balancer
14:41
We want to actually be able to see that we're hitting different endpoints Now if this was a real application, you'd want it to behave the same But for the purposes of the demo, we're kind of cheating here a little bit I'm trying to make it look as interactive as possible That's the only real change here everything else is the same so we'll go ahead and enter the studio and
15:04
We'll build this thing You know notice that this workflow is always the same So I work at Hashicorp and we're a workflow company and we focus a lot on workflows and This habitat tool is also a workflow tool. I enter my studio. I run build It doesn't matter if I'm running Ruby or go or Python or PHP or COBOL
15:23
Like it's the same workflow every single time and I'm just talking to kill time while this compiles Perfect. There we go. You can cut that from the stream Great let's start one of these One of these things where's my little
15:42
Yeah, here we go. So we'll start one of these things start Okay, this thing is running Supplog you're starting to get the hang of this right? The thing is running. It's running on the right port and This time I'm gonna grab This IP to my clipboard because we're gonna need it in a little bit
16:02
But now I want to apply a config change. So what I'm going to do is on my The The thing here, I'm gonna run have config apply Then I give it the target group that I want the target service group, which is just going to be HTTP echo default I give it a monotonically incrementing integer
16:22
Which is just going to be two in this case and I'm gonna give it the thing that I want to change And in this case, I want it to change the text So I'm gonna say if you remember the current text is hello chef comp. So I'm gonna change the text to say Hello puppet calm awkward and
16:43
We'll run the log command. You'll notice there's that hooks recompiled there at the bottom and hopefully if we curl The right port five six seven eight. We'll get hello puppet comp. Well shit This is awkward. We should fix that so we can go back and we as long as we use a higher
17:04
Monotonic incrementing integer I can change this back and if we curl we're back to chef comp Pretty cool, right? I Didn't have to restart the service I didn't have to do any of that and it's gonna be even cooler when we put this thing behind a load balancer
17:21
So in order to run this at scale The habitat supervisor currently can't run multiple instances of the same package or same application So I'm gonna export this to docker which thankfully Habitat makes really easy for me so I can just say have package export docker And the name that I want it to export as
17:42
and this is going to download all the stuff that it needs to export as a docker container and Again, I'm going to attempt to talk my way through this but it's docker So it's gonna not only install all of those things rebuild my package But it's also then going to rely on that the whole docker build process. So in a second here, you should actually see the
18:05
Familiar docker output there. It just started. So we're sending the context of the docker daemon and here's your Standard kind of iterating over the docker file notice that I didn't write a docker file I didn't have to touch docker. It just happened to be installed and now I'm exporting a container So I don't actually have to think about the whole docker file process or any of that and we're done
18:27
So now I have a docker image on my local laptop and I'm gonna open a new tab To show you how this works So what I'm gonna do is I'm gonna start one of these Outside of the studio and the way I do that is with the familiar docker run command and I'm gonna run it in interactive mode
18:45
Just so you can see the logs, but it could start in a background process And it's just gonna be the name of the container. So here we go Notice the familiar output, but this is running in Docker and I'm on a Mac So it's running a Docker for Mac and notice that it's listening on one seven two seventeen dot three
19:02
And that's the way that doctors IP networking works. We already have something on dot o2 So now I want to start a second one of these Well, how do I do that in order to do that? I have to group them together into a pier or like a service ring. So I Can't actually do that in this scenario. So what I'm gonna do instead is
19:23
Start a new tab here and I'm gonna run the same command But this time I'm gonna specify the dash dash pier flag and I'm gonna pass in the IP address of my already running container Which in this case is the 1703 And when I run this it's gonna start up the same but you'll notice there's that hooks recompiled and if you look above it
19:45
You see another hooks recompiled that's your indication that those two services have found each other and we can do this a third time and I can join any one of the other IP addresses So in this case, I have two of them joined together But the way that the gossip protocol works under the hood is they all find each other
20:02
So all of these services are now gossiping they're communicating with each other and I'm gonna clear all these screens here and But now I need a way to address them So I have three of these things There are three instances of them running and a really common pattern is to put them behind a load balancer That way traffic is kind of round-robin or some type of routing algorithm to all of these services and we'll make that you know
20:23
Accessible to the users of our application on You know my myspace comm or whatever our application is going to be So the way I do that is by opening yet another new tab and I'm going to start a docker container, which is a habitat service that I built ahead of time This habitat service is called nginx LB. It's fully open source. It's on my github
20:44
What it does is it accepts a series of binds and binds to a backend and automatically creates an nginx load balancer Based off of that, so I'm not going to go into super detail there You're free to use it and look at it as you want, but it's just an implementation detail of this talk So I'm going to go ahead and run that again in interactive mode so you can see the the details there
21:04
But I'm also going to bind a port 80 and I'll show you why I'm gonna bind a port 80 on my host and on the container. So it is set of our go slash nginx LB and Then it needs a few things. It has to join the same peer group
21:21
So I'll have to pass in the dash dash peer flag, which was what 172 dot 17 dot 0 dot 3 But then it also needs a dash dash bind Argument and this is the backend service to bind to that I want to route traffic to and in this case, that's just HTTP echo dot default. That's the backend service group So I'm going to go ahead and start this up and you'll notice we get a couple errors. It's like hey
21:43
Scroll too fast. Hey, I don't see that service group yet. I don't see that service group yet. Oh, there we go I found it because the ring has completed I've propagated that gossip membership and there we go nginx is now running And it started the process and you'll notice that it compiled nginx.com and mime dot types Those are the configs that are being managed by this habitat supervisor. So now the moment of truth is if I open
22:07
We'll do side by side here If I open another tab over here and I curl localhost boom Not only am I hitting the load balancer because you can see the log message here on the left But if we run this a couple times
22:21
T P n 0.5 curl minus s localhost What this is doing is every half a second It's running that curl command and you'll notice that we're hitting different IP addresses three four five They're running on different nodes. These are actually behind a load balancer all monitored by habitat And all the backend upstream dynamic load balancing all that was handled by nginx based off of the habitat integration
22:46
Pretty cool, right? Y'all are scaring me Alright fine. I can see you're not impressed. So let's change that I'm gonna close this here. I'm gonna jump back over to my studio
23:02
So the studio is still running but this studio isn't connected to that peer group at all The studio is running its own supervisor, but it's not connected to those docker containers that are running So what I'm gonna do is over here. I'm gonna run that same curl Run that same curl command as before So same thing that's happening, but I want to push out a config change
23:23
So I want to have config apply with HTTP echo dot default I Want to give it a monotonically increasing integer. So this time it's going to be that The text is going to be hello world
23:42
But I Need to somehow target these other peers and I do that by passing in the dash dash peer flag up here and I can Give again any one of those peer addresses So in this case, I'm gonna give that zero dot four and I can't type Text equal I need an ending quote
24:01
There we go. So that config applied and now it's updated just like that So now this is hello world and I can come over here and I can say that the text is Banana and the text will update to a misspelled banana. Is this cool stuff now?
24:23
All right so The challenge I have now is that I don't know about you, but I don't run prod on my local laptop at least not anymore so Let's go ahead and quit this. So I'm gonna I'm gonna exit the studio. We'll quit these docker containers
24:40
no, we got time for that and We'll quit the nginx here. Okay, so now we're back to straight up Production so we're gonna go up to our production and nothing has changed about the habitat package I haven't added anything or changed anything. So it's the same dynamic one as before What I've added into the mix now are three new tools
25:02
Hashicorp terraform Hashicorp console and Hashicorp Nomad Sorry, we have so many tools All of these tools are free and open source and they integrate really well with the product as I'm about to show you So the first thing I did in advance was I spun up a cluster so inside this repo which again will be open source
25:21
I have a bunch of terraform configurations Terraform is an open source tool for provisioning infrastructure resources Amazon Google Azure or digital ocean It can manage not only just compute instances But it can also manage things like github teams and permissions or integration with your Oracle public cloud Basically anything that has an API in a declarative syntax So I have a bunch of terraform configs in here not trying to teach you terraform here
25:44
What I'm trying to show you is that this whole cluster provisions in about three minutes But I wasn't sure how long my talk would take so I did it in advance So what this is doing is making sure I'm all up-to-date And what I have here are five R3 XL's Running as clients and three C3 to XL's running as servers not important
26:03
If you don't know what those are Beefy instances that can run lots and lots of applications running in Amazon web services This demo has nothing to do with AWS You could run this anywhere, but just for illustration purposes. That's what I'm using These clients have software pre-installed Particularly, they have two HashiCorp tools nomad, which is an application scheduler
26:23
and console, which is a service discovery and distributed key value store Nomad is akin to something like mesos or Kubernetes if you're familiar with that terminology and then consoles similar to something like zookeeper or Other service discovery frameworks, it provides a registry and that's important and I'll talk about why in a second
26:43
So what I'm going to do right now is I'm just gonna jump on one of these clients here. So we will Log in here. Look at that. You get a nice. Have a great day MOTD configured that alright so One of the interesting things about schedulers is the way most schedulers work is they run multiple applications on the same physical host
27:08
But if you have applications that bind to a specific port in this case five six seven eight You either have to use like overlay networking and then do some type of firewall routing or you can do what's called port mapping Where you use really high port allocation numbers like forty three thousand five hundred seventy or sixty seven thousand one hundred twenty three
27:25
To dynamically allocate these resources on different ports Then you have some type of registry that keeps track of what ports these services are running on You'll also recall that in the previous sections in order to join those things together those three
27:40
HTTP echo instances and that one nginx load balancer. I had to copy and paste an IP address around That's not really automation friendly and I want to be able to scale this very rapidly again We're the next myspace. We have to be able to hit that maximum scale You guys are joking you think I'm joking but it's it's it's for real So in order to fix or alleviate this problem I'm leveraging console
28:04
What console is going to allow me to do is I'm going to start a service I'm going to register that service with console particularly that service registers itself with console Console that knows its IP address the ports is listening on a bunch of metadata about it And then I can query consoles DNS interface So I don't have to pass around IP addresses I can pass around well-known DNS entries and let me show you what I mean
28:28
Jobs, okay. So this is a nomad job. I haven't told you a lot about nomad yet, but that's okay We'll talk about it in a second This is a nomad job specification. What I'm doing here is I'm starting a service
28:41
You can see that's type equal service with a priority of 80, which is just an arbitrary priority related to the other Jobs that are running. I'm running it inside Docker and I'm running the official habitat hab management supervisor and I'm marking it as a permanent peer which means this helps protect against like network partitions and Transient peers again habitat implementation detail. The important part here are these four lines of code
29:06
I'm registering this service as Hab dash sup so that name equals hab dash sup and I'm saying that it has an HTTP port Let's not actually port 80 as you'll see in a second. It just has an HTTP port what this says is that this service
29:22
Will be available at hab dash sup dot service console Just like dot myspace comm dot service console is a TLD that console resolves within your data center I'm specifying the resources that I need. So I need one CPU and One gig of RAM, which is a little bit of overkill, but whatever
29:43
I need 20 megabits of network and I'm allocating a static HTTP port 96 31 That's the port that habitat runs its HTTP server on and a static port 96 38 Which is the port that the gossip protocol works on so pretty easy. This is basically encapsulating the docker run command into a text file and the advantage of doing that is like now we get code review we get peer
30:04
review we can version this text file and We can very easily check it in and out of source control For now, I'm just gonna go ahead and run this and I do that via the nomad run command and I pass it In the path to that and what this does is this triggers what's called an evaluation and allocation So it inspects the cluster
30:21
So nomad looks at your cluster as like one terabyte of RAM and one petabyte of disk space and it does bin packing To schedule the application in the most appropriate place on the host We don't really care where it went. It might have gone on this host that I'm on we can check and see did it It did not it's not running on this host. It's running on one of the other clients in this cluster. It doesn't matter
30:45
But then how do we find it? We don't know its IP address Well, that's where console comes in because what I can do is I can hit have sub service console And that will resolve to the IP address that this is running on But what's nice is unlike copying and pasting this IP address around I can hard-code have sub service console because that's actually a dynamically
31:06
Resolved endpoint. So that means when I peer when I pass in that Dash-dash peer flag instead of one dot two dot three dot four I just pass in have sub service console and we rely on the kernel to resolve that
31:23
The resolution actually happens to console which exposes itself in a DNS like interface and it resolves to the service discovery layer So let's check out what that looks like. So let's look at another job here the HTTP echo server So this is going to look very similar I'm creating a service. I'm creating three of them this time
31:41
So I only have one supervisor, but I have three instances of this server So again, very similar to what I ran locally a minute ago I'm giving it a really tiny ephemeral disk because my service doesn't actually need a disk at all. It's just for logs That's the only thing I need disk space for I'm starting it as a server and then here's where things get a little bit different than what you might have seen before
32:02
I'm passing in this environment variable have HTTP echo, which is the environment variable that habitat looks for for its configuration and The port that I'm giving the service to listen on is Nomad port HTTP Nomad at runtime Dynamically populates the values between those curly braces with a really high numbered port that it's allocated for this task
32:27
So that means my server instead of running on five six seven eight like it has been it's going to run on 23,406 or whatever available port is decided and the scheduler does that for us? We don't have to think about it
32:40
Down here. I'm passing in some arguments to my container Remember my container is running the habitat supervisor and the habitat supervisor by default runs its HTTP port and gossip ports on 96 31 and 96 38 But if I build those ports on my host, that means I can't scale my application
33:00
There's only one port that is nine six three one. I can only bind that one time per host So what we do here is we actually allow nomad to dynamically assign not only the port that my service listens on but the port that habitat listens and gossips on and each instance of our HTTP echo application will be Assigned a different dynamic high port number and habitat handles this seamlessly
33:26
Since this is a tiny little go binary. We can actually give it very minimal resources here It's getting 1 20th of a CPU 128 megs of RAM, which almost all of that is just for Docker Five megabits of memory and I'm allocating three dynamic ports and you'll notice they're dynamic because there's nothing inside of them
33:45
What that's telling nomad is just pick me a port that's available. That isn't privileged So like not 80 not 443 something like less than you know Say a thousand or higher than a thousand that I can bind to very easily And I think that's it
34:00
Oh, and then I'm telling it to do minimal logging so that I can keep the disk space low I'll show you why in a minute So let me go ahead and run this now, so I'm gonna run HTTP echo job It's running It was that easy Let's jump back when we run the nomad status command. We can see we have two jobs or two services running
34:23
Have an HTTP echo we can get more information about a particular service by passing in its name So here I can ask for the status of hab in particular We can see we have one running and it has an allocation each instance of the application or service will have an allocation so I can get the
34:41
status of this allocation and here I can see that This particular habitat supervisor is using two of its thousand megahertz of CPU 66 of its gig of memory 30 megabits of disk and it's bound to two addresses the 10.1.1 to 26 and The two different ports there and we can actually get the logs for this service as well
35:03
So instead of alex status we can ask for the logs and that log output should look really familiar That's what we saw in the studio and what we saw in the Docker container So we can verify this thing is running Same thing is true for HTTP echo so we can ask for the status of HTTP echo And you'll notice that I didn't fill the whole thing in. It's a quick greedy match. We have three of these running
35:23
They've all terminated to the running status I'll just grab the first one here I'll check the alex status and here's where things are gonna look a little bit different Notice that that the HTTP port is fixed fifty six thousand eight hundred and seventy six I'm running on a really high gossip port and a really high have HTTP port and these were ports that the scheduler assigned to me and notice that it's a different IP address. This is running on
35:45
2.175 I Can also get the logs for this service the same way we did before and we can see it's running and I can grab this IP and I can curl that IP and we get a thing. So it's actually running like it's not BS
36:04
The thing is actually running that's a different host in the cluster and I just hit it via curl And there's three of these running so just like before I need to put these in a load balancer Because I don't want to copy and paste IP addresses around so we'll take a look at the Nginx load balancer job again, it's pulling down the same docker container
36:23
But the one difference here is on line three. The type is no longer service. The type is now system This is a special type of job in nomad that instead of specifying a count This tells nomad to make sure that one of these is running on every host in The cluster and as new hosts join they automatically get this and I'm binding this to port 80 on the cluster
36:44
So again dynamic port allocations. The important part is down here. I'm binding on port 80 So what that means is every host in my cluster in this case Every AWS instance is going to have an nginx service listening on port 80 bound to the host Which means if you hit the public IP of this AWS instance, you would be hitting this nginx load balancer
37:04
This is important in a second For now I'm gonna run this I have five clients how many nginx servers are gonna get scheduled Five let's see if we're right. I Hope we're right. It would be awkward
37:22
one two three four five right we got five scheduled and Nomads handling like the downloading of this container. It didn't exist in advance. It's handling all of that background network traffic and placement It's making sure that there's one of these on every host every client in the cluster So we can check the status of the nginx. Perfect. All five of them are running and if I run
37:43
Docker ps you can see. Oh, look, that's one of the nginx containers and I can Nomad status nginx. I can grab one of these allocations and I can ask for the logs of this allocation
38:02
And we can see that it's actually getting requests and oops. I'm giving something away here There's an ELB health checker coming in which I'll show you in a second. But those are the logs so Now what I can do is I can curl localhost because this thing runs on localhost it runs on localhost port 80, but it also runs on localhost and
38:26
Oca L host It runs on localhost and each time I hit this I'm getting round-robin to a different thing. So watch dash TP and 0.5 Crow minus s localhost. Okay, and you can see that that's changing continuously, but like I said before
38:48
We like we need scale right so how many of these do you think we can reasonably run how many people think we could? hit 50 50. All right. Let's check it out So one of the differences between nomad and other schedulers on the market is its speed
39:04
So with nomad we scheduled a million containers in under five minutes So I'm about to schedule 50 so it should be pretty darn impressive so I'm gonna time this nomad run jobs HTTP echo and I didn't nomad. There we go. Nomad run and
39:22
It's gonna do the evaluation and that took a little bit over a second so we just scheduled 50 we scheduled 47 containers because there were already three running and And What's interesting is that these are actually public facing So I'm gonna jump over to my laptop here and I'm gonna fire up a web browser and I'm gonna hit
39:42
nomad.hashicorp.rocks you can hit this on your phone if you'd like This is a real URL that is actually load balancing all those clients and I'm just hitting refresh So all 50 of those clients are being load balanced. You're free to try this out if you'd like This is a public facing thing So what we have is there's a load balancer an Amazon load balancer and ALB
40:03
That is hitting port 80 on all of our clients and port 80 is listening to nginx and nginx is routing traffic to these Dynamically bound back-end services. They're dynamically bound and registered with the habitat supervisor. I have two seconds, but I'm just gonna keep going Y'all free to leave but I have one more thing to show you so
40:23
How many people think we can do 250 All right, so let's check it out so we'll time this nomad run Jobs HTTP echo how many people how long do people think this is going to take? Five seconds. Let's check it out. Remember we're only scheduling 200. It took 1.14 seconds
40:41
It's a schedule 200 containers. So we have now reached critical myspace mass and if we jump over here We'll start seeing more and more and more of these instances running And I don't have time for it But the end of my demo was pushing out a config change that said thank you But I only have negative 44 seconds left. So that's the end of my talk. Everything will be public on github
41:05
I'll tweet out the link. I'm Seth Vargo. Thank you Bye