Modern tooling to assist with developing applications on FreeBSD
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 42 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45511 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
3
5
9
11
12
13
15
17
18
22
28
29
31
35
36
37
39
41
42
00:00
Demo (music)Software developerSound effectNumberReal numberDemo (music)Point cloudTerm (mathematics)XMLUMLComputer animation
01:00
Virtual machineView (database)WindowGastropod shellScripting languageBuildingCache (computing)outputMedical imagingDemo (music)Computer fileOrder (biology)Scripting languageGastropod shellSeries (mathematics)Parameter (computer programming)Forcing (mathematics)Open setComputer animationSource code
03:45
Gastropod shellView (database)WindowSupersonic speedKeyboard shortcutBridging (networking)Virtual machineComputer fileVirtual realityFlagGame controllerFloppy diskParallel portData modelGeneric programmingInterrupt <Informatik>EmulationLink (knot theory)Computer configurationNumberWeightNormed vector spaceDirected setHeat transferIntegrated development environmentSCSIFrequencyRootCore dumpPhysical systemInterface (computing)Virtual machineConnected spaceContext awarenessShooting method2 (number)BootingType theoryInternet service providerMultiplication signKeyboard shortcutTouchscreenSource codeJSONComputer animation
04:31
Virtual realityComputer fileVirtual machineView (database)WindowSystem administratorComputer networkKernel (computing)Physical systemComponent-based software engineeringNetwork topologySource codeSuite (music)Software testingDistribution (mathematics)Partition (number theory)MiniDiscOpen setGastropod shellOperations researchSystem programmingNumbering schemeVolumeTable (information)Graphical user interfaceMereologyText editorHill differential equationMaizePoint (geometry)CASE <Informatik>Medical imagingRoutingRootInstallation artDifferent (Kate Ryan album)Computer animation
05:04
Virtual realityComputer fileView (database)WindowVirtual machinePhysical systemData managementPasswordInterface (computing)Configuration managementComputer networkPoint (geometry)2 (number)Medical imagingMultiplication signRight angleTouchscreenJSONComputer animation
05:50
WindowElectronic visual displayMenu (computing)Computer fileView (database)Virtual machineVirtual realityElectric currentGastropod shellFormal languageBootingWeightCASE <Informatik>Scripting languageCoprocessorMedical imagingIntegrated development environmentCuboidRight angleCartesian coordinate systemPoint (geometry)Raw image formatPoint cloudForm (programming)Menu (computing)Computer programmingGoogolParameter (computer programming)Program flowchartComputer animationSource codeJSON
07:30
View (database)WindowComputer fileVirtual machineVirtual realityInclusion mapGastropod shellPlanningOffice suiteGodData managementGroup actionAdditionMedical imagingConfiguration management2 (number)Installation artScripting languagePoint (geometry)Coefficient of determinationForm (programming)BitUtility softwareComputer animation
08:23
Online chatSystem administratorStability theoryChemical affinitySubsetServer (computing)BitOpen sourceSystem administratorDebuggerBit rateMultiplication signStability theoryVideo gameAffine spaceCuboidInformation securityData managementRight angleCASE <Informatik>Computer animation
09:12
System administratorPhysical systemImplementationBuffer overflowWeb pageMessage passingProjective planeOrder (biology)Web pageCASE <Informatik>Mobile appPerspective (visual)Staff (military)Interface (computing)Computer animation
09:48
DivisorBlogStaff (military)Interface (computing)Server (computing)Connected spacePoint (geometry)Stability theoryMultiplication signPresentation of a groupVideoconferencingComputer animationXMLUML
10:32
Kernel (computing)Read-only memoryLimit (category theory)Default (computer science)Computer networkService (economics)Firewall (computing)Binary filePhysical systemStability theoryRight angleCuboidLevel (video gaming)Semiconductor memorySteady state (chemistry)Configuration managementPerfect groupIntegrated development environmentUtility softwareMedical imagingGoodness of fitChainSource codeSelf-organizationComputer animation
11:40
Kernel (computing)Product (business)SoftwareData dictionaryStructural loadOperational amplifierSocial classInterrupt <Informatik>Entire functionSoftwareServer (computing)Computer animation
12:25
Binary filePhysical systemImplementationBuffer overflowWeb pageSystem administratorFocus (optics)Set (mathematics)Instance (computer science)CuboidIntegrated development environmentExistential quantificationMultilaterationComputer animationXMLUML
13:01
Data dictionaryNumberSystem administratorServer (computing)Information securityAverageSystem administratorNumberData managementServer (computing)Right angleAttribute grammarUsabilitySet (mathematics)Perspective (visual)Different (Kate Ryan album)BitComputer animation
13:41
Maxima and minimaServer (computing)Random numberControl flowCloud computingSet (mathematics)UsabilityDifferent (Kate Ryan album)Demo (music)Installation artServer (computing)AlgebraFrictionOrder (biology)Core dumpInstance (computer science)Mathematical optimizationReduction of orderComputer animationSource codeMeeting/Interview
14:17
Cloud computingFrictionNumberBlogBuffer overflowInstance (computer science)RandomizationSearch engine (computing)BlogType theoryElectronic program guideMathematical optimizationOrder (biology)Axiom of choicePoint cloudTerm (mathematics)Computer animation
14:55
NumberProcess (computing)Maxima and minimaFrictionCloud computingComputer-generated imageryDistanceData managementRandomizationMedical imagingPoint cloudDifferent (Kate Ryan album)Order (biology)Server (computing)Product (business)Integrated development environment1 (number)Software developerData centerPhysical lawQueue (abstract data type)Term (mathematics)Computer animation
15:42
DistanceSpacetimeFrictionCloud computingComputer-generated imageryScale (map)Software testingSpacetimeData managementDistancePoint cloudFrictionIntegrated development environmentMedical imagingServer (computing)Software testingProduct (business)Configuration managementRepresentation (politics)EmailSoftware developerException handlingNumberOperator (mathematics)Order (biology)IP addressMultiplication signComputer animation
17:01
Revision controlGastropod shellInternet service providerNumberMultiplication signGoodness of fitOrder (biology)Ocean currentComputer fileRevision controlLocal ringInternet service providerFreewareDirected graphPoint (geometry)Software developerLaptopCASE <Informatik>Directory serviceUsabilityCoefficient of determinationBit rateSet (mathematics)Computer hardwareVirtualizationInteractive televisionComputer animation
18:32
Internet service providerRead-only memorySemiconductor memoryBefehlsprozessorFunction (mathematics)NumberUsabilityInteractive televisionInternetworkingConfiguration managementSoftware developerCASE <Informatik>BitType theoryCuboidComputer fileInstance (computer science)Context awareness
19:07
CloningDefault (computer science)Internet service providerLatent heatVirtual machineError messageRepeating decimalType theoryDifferent (Kate Ryan album)Internet service providerInstance (computer science)BitMultilaterationComputer fileVirtualizationCuboidConfiguration managementModal logicSource codeAddress spaceIntegrated development environmentMultiplication signComputer animation
20:10
Default (computer science)Electric currentConstraint (mathematics)Structural loadInternet service providerAdditionInstallation artGeneric programmingSoftwareRevision controlFamilyCuboidHydraulic jumpArmElectronic meeting systemSource codeComputer animation
21:02
Slide ruleWindowData managementData modelNumberIdentical particlesError messagePasswordStaff (military)Total S.A.Commitment schemeComputer fileSimulationBitIntegrated development environmentLink (knot theory)CuboidLoginLaptopLocal ringMathematicsSemiconductor memoryContent (media)Configuration managementOcean currentDirectory serviceVirtual machineSoftware testingCASE <Informatik>Arithmetic meanText editorLevel (video gaming)Computer animation
22:56
LoginLaptopVirtual machineState of matterDefault (computer science)BootingComputer fileInheritance (object-oriented programming)Right anglePhysical lawIntegrated development environmentContent (media)Resolvent formalism
24:05
Revision controlGastropod shellLaptopComputer fileMultiplication signPoint (geometry)Cartesian coordinate systemProcess (computing)Product (business)Medical imagingIntegrated development environmentCASE <Informatik>Right angleGroup actionScripting languageGastropod shellComputer animation
24:54
Scripting languageScalable Coherent InterfaceGastropod shellVirtual machineProduct (business)Scripting languageGastropod shellFile formatInstallation artVirtualizationCuboidStudent's t-testComputer animation
25:42
Cluster samplingServer (computing)Internet service providerIntegrated development environmentComputer fileServer (computing)Power (physics)Software developerBitConfiguration managementSingle-precision floating-point formatClient (computing)Type theoryFilm editingCASE <Informatik>Proxy serverDirectory serviceComputer animation
27:01
Cluster samplingServer (computing)Client (computing)Virtual machineLaptopRegulärer Ausdruck <Textverarbeitung>CASE <Informatik>Client (computing)Normal (geometry)Point (geometry)DataflowBitProduct (business)Form (programming)Multiplication signMereologyComputer animation
27:48
MereologyCloningTemplate (C++)MereologyProduct (business)Channel capacityBitComputer configurationRoutingCASE <Informatik>CuboidOrder (biology)Greatest elementScripting languageGoodness of fit1 (number)Integrated development environmentDirectory serviceDifferent (Kate Ryan album)Template (C++)Gastropod shellOcean currentComputer fileWrapper (data mining)Figurate numberComputer animationSource codeJSON
29:41
View (database)Slide ruleComputer fileTable (information)Zoom lensFile formatHard disk driveJava appletVirtual machineWindowGastropod shellFreewareComputer music2 (number)Level (video gaming)BootingCuboidRule of inferenceDatabaseTheoryCartesian coordinate systemBasis <Mathematik>MereologyInstallation artView (database)Ideal (ethics)Scripting languageVirtualizationPerspective (visual)WeightInternet service providerMedical imagingMultiplication signBootstrap aggregatingConnected spaceSelf-organizationComputer fileGoogolPoint (geometry)Utility softwareCASE <Informatik>Integrated development environmentFigurate numberMetropolitan area networkComputer clusterSoftware repositoryDifferent (Kate Ryan album)LaptopTemplate (C++)Variable (mathematics)Sampling (statistics)Message passingType theoryProcess (computing)Office suiteRight angleCoprocessorConnectivity (graph theory)Computer programmingOcean currentPhysical lawPlastikkarteData compressionKeyboard shortcutNeuroinformatikRootBuildingSystem callDefault (computer science)Cloud computingOrder (biology)EmailContext awarenessCase moddingLoginWrapper (data mining)Virtual realityComputer animationSource codeJSON
37:13
WindowView (database)Gastropod shellSelf-organizationMathematical optimizationMaxima and minimaEntire functionHookingCuboidArc (geometry)Point (geometry)Point cloudAbsolute valueMedical imagingSpacetimeOperator (mathematics)Touch typingInteractive televisionDeterminismMereologySemiconductor memoryNumberBootstrap aggregatingInternet service providerInformation securityRevision controlComputer fileData centerGastropod shellScripting languageUniverse (mathematics)Kernel (computing)Process (computing)Connected spaceFunction (mathematics)Parameter (computer programming)Configuration managementIntegrated development environmentData conversionBuildingTerm (mathematics)Server (computing)Order (biology)Uniform resource locatorDatabase normalizationRight angleError messageMultiplication signGoodness of fitDefault (computer science)Latent heatInstance (computer science)CASE <Informatik>Euler anglesFigurate numberMathematicsContext awarenessTransport Layer SecurityTheoryGroup actionProduct (business)Proxy serverInstallation artCloud computingStudent's t-test
44:35
Gastropod shellWindowView (database)Complete metric space
Transcript: English(auto-generated)
00:05
So modern tooling with FreeBSD, or doing modern development on FreeBSD using modern tooling, or something to that effect. So who am I? I'm Sean Chittenden. I work right now at HashiCorp, been a FreeBSD user for a number of years.
00:23
And I'm starting to be more involved in the community again. So I want to do a quick demo real quick to kind of explain what I mean by talking about modern tooling and explain some of the benefits of modern tooling, whatever that may be in your guys' minds. And then go through what some of the values
00:43
are and some of the problems that have led to us as a community getting into the situation where we are right now, where in the cloud world we are, I will call, somewhat laggard, lagging behind where the rest of the industry is in terms of adoption.
01:01
So what we're going to do in this demo here is we're going to take a Packer JSON file. We're going to run it through Packer. We're going to take an ISO image, run it through Packer. We're going to use Packer, which will spit out a golden image, FreeBSD image, that we can then use. And then we'll customize this using some background SSH.
01:26
And please feel free to stop me along the way. So I have a Packer file here. I have a, yes I can, is this better?
01:45
Great. So I have a repository, GitHub, that has a series of scripts here. There's a shell script in here that is going to be used to build a FreeBSD image using current and ZFS.
02:02
There's a handful of arguments. None of this is too terribly fancy. And we're going to build this headless, or with headless false so you can actually see what this does.
03:01
Oh, I do need wireless offline mode before I lose.
03:43
Firing up VMware. And this is a scripted interface. So what this is doing in the background is it has launched a VM in a scripted way.
04:00
It's booting FreeBSD. This establishes a VNC connection between my machine and VMware. We're waiting for this to go through. Waiting for 45 seconds for this to finish booting. Hopefully it's VNC.
04:21
There's no screen scraping going on here. We are doing a time delayed keyboard entry using the VMware provider. There we go, starting to type commands. OK. So this is useful up to a certain point. What this is doing right now is it's going through a scripted install of FreeBSD.
04:42
In this case, I'm using a ZFS image, which is something that we don't typically see is VMs. They're always typically UFS based on the root. So this one I created as a ZFS root. And this is interesting in the FreeBSD community because the installer means that the difference between a UFS
05:05
image and a ZFS image is you push left at one point instead of pushing enter, which makes it a little problematic. You have to know exactly what you're getting into. And we will get into what that means here in a second. Configure, this is where things were breaking earlier.
05:23
Please don't time out when acquiring a lease. Yes, it's going to break. OK. So you can see down here that it's continuing to fire off commands because it does not acquire the lease.
05:41
This is what I was trying to debug earlier. And I don't know why it's not acquiring a lease in time. All right, so I'll show you what this is doing behind the screens.
06:06
So this is the language of Packer in the boot commands. And this is what we were seeing right here, is a handful of wait commands. And either I didn't wait long enough or it's just never going to finish its lease, which is actually what the case is. So you can see here that it's firing off
06:21
all of these kind of commands. Stepping through these commands right now is the only way that we can programmatically, short of building a new release image, create a FreeBSD image for use in some form of a cloud environment. So at the end here, it'll eventually bounce the box and you can SSH it. Great.
06:40
But that's not what's useful about Packer. What's really useful is the post processors and the script provisioners down here. So this allows you to, after you bring up a new VM, run arbitrary scripts, create a VM that is specific for your environment, that includes all the application parameters, and then upload that to Amazon, DigitalOcean, Google,
07:03
whoever else. You can't create an ISO today. At some point in the future, it should be very possible to programmatically create either a raw image or an ISO. It is possible to create a virtual box or a VM image
07:20
and convert them over. But that's manually not something that's handled by the tool right now. So here you can see that things are still basically hung. It worked fine on the plane. It worked fine in my office. I swear to God. I think it actually has something to do with IPv6.
07:44
So anyway, so that's what this is supposed to do in that you have a FreeBSD image that you've customized with some form of install script. You can run configuration management in that install script so that you have something that is customized. But the point is that you have a golden image coming out
08:00
of Packer. That golden image you can then add to, let's say, an auto scaling group inside of Amazon and the ability to add it to an auto scaling group so that if necessary, you can spin up additional copies in seconds as opposed to minutes after waiting for some kind of install or config management to kick off after you have kind of a base image.
08:24
So I want to have a little bit of what I'm going to call an awkward chat about FreeBSD because it's a little bit of a problem child in the open source world in the sense that its adoption has really lagged behind. And there's some interesting problems that have kind of led to this.
08:41
Most notably, it's stable. It's debuggable. There are lots of knowledgeable administrators. It performs very well, secure, manageable, embeddable to Pride. And the last one is the interesting one, Pride, actually. So what do I mean by and why are these problems? So in the case of stability, as a community, we largely have an affinity for uptime.
09:01
We like maximizing the amount of life out of a box. And we frequently brag about uptimes that exceed 1,000 days. And that's a problem. We know how to debug our stuff. OK. Thank you to Brendan Gregg. We have knowledgeable administrators.
09:22
And by this, the commit comments are fantastic. To this day, I still read SVN Digest because in order for me to stay current with the community, both because it's moving at a fast pace, but also just to understand the motivation, there's really detailed commit messages. That's great. The man pages are fantastic.
09:41
Thank you to the doc project. And we also have this fantastic reference book that everybody seems to have read. So very knowledgeable administrative staff. From a performance perspective, we are also very performant. There are large CDNs that run this and max out 100 gig interfaces.
10:02
In this case, this is from WhatsApp where they're talking about stability. And they were talking about how they were doing 2 million connections per server and what that looks like. Very, very steady performance. And at one point in time, they actually bragged about a million. And they came back a year later.
10:20
And they were like, you know what? We're doing two. And in the MeetBSD presentation from two years ago, I think they were talking about 5.6 or something like that. I have to go back and watch the video again. But that's an interesting kind of thing, right? And these are problems, though. So we have a fixed user land kernel.
10:41
Like when a box gets busy, it swaps. You kill whatever consumed all the memory. And the system returns to some level of stability. Again, these are problems, right? We also ship it reasonably secure in the sense that when there is an issue, we can lock things down. So we have this steady state stable configuration.
11:03
We have lots of good utilities for doing in-place upgrades, right? Merge Master, Etsy Update. We can create an image that is kind of perfect for our environment. It's packageable. It's useful for embedded systems. And I'm going to come back to that in a second.
11:21
Because, and one of the things that's interesting is we have this tool chain here that, if implemented correctly inside of your organization, is a codified way of describing what it is that you want, right? Make and SourceConf give you kind of a very finely tuned system, Podre as well.
11:41
Because it's embeddable, we have this notion that we expect the OS to be stable and long-lived. And we want it to kind of work forever, but because we have these assumptions, we assume that we're going to be able to front-load a lot of work by adopting a technology. And that's great, but it also means
12:02
that we take a lot of pride in what it is that we do. And these are, again, problems. So in developing this close relationship with a piece of software where we really don't want to see any kind of interruption to our service,
12:21
we have foregone an entire class of problems. And it's because these problems that we don't kind of generally have also mean that we've oriented our focus around basically the wrong set of KPIs for users, right? If you go back and you look at all of these problems,
12:47
if you have an unstable environment and you're something that's not debuggable, or you don't have performance, you're going to spin up lots of instances. You're just going to shoot the box and ask questions later. If something's secure, you don't really care. Just leave it open, it doesn't matter,
13:01
because you're just going to spin up another one. You treat them like commodities. And we generally, as a community, don't. So as a follow on, we want to actually change the way that some of these things look stable. We care about maximizing meantime between incidents, debuggable, the number of unknowns.
13:20
We're really good at keeping that number low. And we also are very interested in keeping the number of servers under management per administrator very high. On the performance side of things, we care about bits chucked per server or bytes written secure. These are all great attributes, but they're
13:41
kind of problematic from a usability perspective. If we were less stable or less all of these things, we would actually be solving for a different set of problems. And as you saw earlier in my demo that actually broke because of the installer, that's kind of exactly what I mean. We can't create disposable infrastructure, and that's what I'm getting at.
14:02
Right? We have largely lived in a world where server died, and it's a big deal. And we want to get to a world where a new set of servers are able to automatically be spun up. So how do you do that? So yeah, so how do you do that? So what are the kind of the KPIs?
14:20
Like, why do we put together the infrastructure or the tooling to kind of go and optimize that? We want to go and reduce friction. So what's the effort required in order to spin up a new instance? How do we reduce that? How do we go in and increase kind of our, like, quote unquote, street cred in that we are able to have somebody go and hit random search
14:41
engine of choice, type in a handful of terms, and get a bunch of blog entries back? Right now, I challenge you to go and look for FreeBSD Cloud, whatever else, and like an exact step-by-step guide. It's not something that's easy to go and run across. So right now, in order for us to go and create a new image, thank you to Microsoft Azure.
15:02
What they did and announced at this conference was a very big, useful step in that they provide this kind of stem cell image that actually lets you bypass this kind of VNC firing off random UDP packets, kind of step that we have to go through. But as a community, we don't kind of think in terms of that.
15:22
We think about getting an ISO and pixie-booting something. And that's an issue. And one of the other ones is we frequently have a development server under our desk or something like that. And that leads to differences between your development environment and in production.
15:43
So in the meet space world, one of the things that you would optimize for is the distance between a person and the data center. And you would do that through physical distance. We've obviously got that figured out through out-of-band management. But what we really want to be able to do is just very easily push something straight to the cloud
16:02
and have the ability to go and spin it up with low friction. So what's a modern workflow kind of look like? Well, you want to be able to spin up a development environment. You want to be able to test and beat the snot out of some simulated synthetic thing that's actually going to be representative of your production environment. You want to be able to go and create a golden image,
16:21
something that is reproducible and is able to be integrated with your target production environment. So you want to have a golden image for a database. You want to have a golden image for your application, a golden image for your mail server, for your bastion host, for whatever. What you don't want to do is have kind of like a base OS
16:41
image, and then you give it a personality after it's already in production, because there you're going to see configuration drift between my one-off and my development test environment and whatever's in production. They should be like for like, with the exception of maybe an IP address, DHCP server notwithstanding.
17:02
So in order to get there, there's a tool that has been available for a number of years now. Some people are aware. I've actually spent a fair amount of time talking to people at this conference so far. And who here has heard of Vagrant before? More than I expected.
17:21
Good. You guys are not representative of most of the people I have talked to so far. So I'm going to go quickly through Vagrant before getting to some Packer stuff, because this is an important thing, in that Vagrant allows you to, from a command line, interact with a hardware virtualization
17:43
layer on your laptop. So in this case, we've got a Vagrant file for FreeBSD. How many people use Vagrant for FreeBSD? Much less. Three, four people. So let me go through this real quick. So we've got a Vagrant file. In this, we've specified the API version.
18:01
That's an interesting artifact of history. But we're specifying a FreeBSD guest, FreeBSD 11, that's dog food, what's current, right? Set up a synced folder so that you share the Vagrant with Vagrant file mount point inside of the guest with your local current directory for wherever this Vagrant file is.
18:22
Use NFS. That works. And then we're going to put some provider snippets. These Vagrant files are generally pretty small. But they allow for some really nice usability, usable interactive development workflows. In the case of VMware, specify your memory size. Number of CPUs, you're pretty much off to the races.
18:43
For VirtualBox, there's a little bit more. I'm actually not entirely sure. I use VMware for most of my stuff. So VirtualBox, I've seen this on the internets as a common config for FreeBSD. But similar, you set your memory CPU.
19:02
In this case, GUI false. If you need to see what the output is, you can go and spin that up. But what that file gives you, what the Vagrant file does, is it lets you just type Vagrant up. And this will do a couple of things. It'll either download the ISO and then bring up the instance. Or it will, if you've already got all the local artifacts,
19:21
it will bring up the VM and allow you to SSH in. So there's a handful of different providers. I'm going to come back to just this a little bit here, a little bit later. OK. Because you want to be able to spin up a VMware or VirtualBox provider. So great.
19:40
Spin up your environment. Give it a key. Configure it as necessary. Watch things break. This is an interesting one. FreeBSD does not work out of the box right now with current. So OK, what's going on here? Oh, drat. libpam. So we've got drift between ports and source.
20:01
Do you have a handful of chuckles here? I'm wondering if this has something to do with the reason that base wasn't packaged in time for 11. But it certainly was an interesting thing. So I was like, OK. Well, it turns out I'm actually running an old version of Vagrant. So I'm going to go and pull down the current version. So I go and do that and sit and wait. And about five minutes later, which
20:21
is pretty good for campus network here, I pulled down the updated version from 4.30. And I tried it again. And it's still, it tells me that it was actually already running. And I was like, well, that's interesting. So I've already got this VM that's running, as you can see through the Vagrant status.
20:41
Jump onto the box. Great. OK, so I'm now using the updated version of Vagrant. I'm using the updated version of, or the old version, I'm sorry. The box download, I was still using the version that I originally had on there from March, not from April.
21:02
So I was like, OK, so in the interest of being able to randomly running around and commit violence towards VMs and just shut them down. So destroy. Try it again. Blew that up. It's disposable now. This is kind of like where we want to get to. So then, great, spun it up.
21:23
And it blew up again. So the box update didn't actually fix things. So what did I have to do? OK, so in this case, jumped onto the box. sudo, that doesn't work. Go and log in manually. Delete sudo. Update, that actually did provide an update. Very happy about this.
21:41
Hopefully this works. No, it doesn't. OK, libpam, not there. Damn. All right, so that's pretty easy. I could use libmap. Instead, I just went and created a sim link because it's a little bit easier and actually what I have muscle memory remember.
22:00
Great, so I'm going to go and reload, which goes and logs back in. I know you probably can't read that. Logs back into the VM, updates the configuration. And in this case, it succeeds now. And you can see way down here at the bottom, there's a little bit of thing like etsy exports, whatever, mounted NFS shared folders. We're off to the races now. So in my current directory, I've got a vagrant file.
22:23
I can SSH into the box now. I can see that in the slash vagrant, I've got synchronized file contents because NFS, great. And then I can also see that I mounted it. And that's how this happened. I did an NFS mount from the host, the laptop itself,
22:42
into the guest at slash vagrant. And this is really useful because now I can work on my local machine, test, do whatever I want with my local editors, environments, but actually affect change or files inside of the FreeBSD environment. Go ahead.
23:05
There was a drift between sudo, which had received a shared library update and was in the ports cluster, was compiled against libpam.so.6. And in the VM that I downloaded of 11 current,
23:22
it only shipped with libpam.so.5. So when the loader went to go and execute sudo, it failed because it couldn't resolve to find libso.6. So suspend then.
23:40
So I've got this VM, which is fantastic. I can show you here, I'm going to goosfraba and dump that off to vagrant foo, log out. I can look at the file foo. It's got the right contents. And I want to shut down the VM. I actually don't shut it down. I just suspend it. And then you can see here then that vagrant state,
24:02
it says that it's suspended. So this is great. And now I really want to kind of move to the next step where I've got this development environment now that I can spin up, spin down. I can test. I can have potentially dozens of vagrant files
24:21
and VMs running either simultaneously or not. And that's really flexible. Now what I want to be able to do at this point in time though is I actually want to be able to have some kind of like automated provisioning. It's nice that I can spin up a VM. But what I really want to do is I want to make this scriptable and I want to make this a repeatable process so that I've got either a repeatable install or image or something that
24:41
is specific to the application that I'm trying to build so that it's the same on my laptop as it is in production. So in this case, you have this provisioner script, script, whatever it is. OK, great. And what does that look like? Well, in this case, it's a Ruby script that's executed as a shell inside of the Ruby variable executed as a shell script inside of the VM.
25:02
So in this case, if this worked, I would have just done sudo package install go and run it. And that's nice. That gets me a VM. But I can't take a virtual box machine or VM, and I can't run that in production because they're different formats.
25:21
So there's something also that's kind of neat coming down the pipe that I want to just call out. And I'm actually really excited about this. So you can see here we've got Vagrant Beehive. This is a GSoC student that is actively working on this. And very active. So I'm extremely optimistic and hopeful
25:43
that we're going to have a Beehive provider for Vagrant in the near future. Because then you can actually work native inside of your environment, but using these exact same tools. So there you go. Hopefully to be determined. But I'm optimistic.
26:00
So having a Vagrant file per server is nice. But what you probably want to do is be able to spin up a cluster of servers. And you want to share configuration. There's one little bit here about Vagrant that I wanted to highlight that I'm not sure how many people actually know about unless you're
26:20
like a Vagrant kind of power user. But this is really useful. This is actually how I do cluster development. I go and spin up. In this case, this is me doing nomad development. I've got three nomad servers here. One up to three nomad server percento2d. And then I've also got up to six nomad clients down here.
26:40
And what you do by specifying autostart1 and primary1 up there is it allows you to type Vagrant up, and it will only act on the first server. In this case, nomadserver01. But I've got a config file defined for up to three servers and three clients. And this means that inside of a single Vagrant file,
27:01
single directory, I can go and spin up as many VMs as my laptop will reasonably support. So in this case, you can see I can just do Vagrant up, Vagrant suspend. Great. That works as normal on just one VM. But then I can also do Vagrant up on a regular expression. That's what the slashes actually are interpreted as.
27:21
And in this case, my nomad client. So I've got a cluster of four machines, and that's useful. So what I want to be able to do is create a dev to prod kind of workflow. Because I've got the dev side of things largely squared away, what I don't have is the ability to go and take that and push things
27:43
into production. Are there questions, by the way, at this point in time? So Packer. So what I'm really interested in doing here is filling in something that's in the case, and it's part of kind of a larger ecosystem. So we talked a bit there about Packer or Vagrant,
28:01
which we use to feed kind of to Packer. And ultimately, you want to be able to take that into production either through Terraform or through Nomad, which, as of a couple of weeks ago, does support FreeBSD in some very limited capacity. So we have some Packer templates. Thanks to Beardy for doing the heavy lifting and initial lifting.
28:20
I was showing you some of this earlier. What this lets you do is very easily go and spin up a VM environment that is customized for your environment. And then you can just Vagrant up Vagrant SSH into the environment. So what does this normally look like or should look like? Is you should be able to, in this case,
28:43
and the only reason that shell script exists is just as a wrapper to handle some of the options. But we'll get into what those options are in a sec. So what Packer does is it has the ability to download ISOs, in this case from FreeBSD, checks on them, compare it against current, spins up
29:05
the VM in the background. And then you can script through things. And this is, as you can see at the bottom here, where things started to go south. And I didn't actually figure out how or why. Every now and then this happens, and I didn't want to reboot my box in order
29:20
to debug that. But yeah, so there's a couple of different, in this case, script files. One of them is for UFS, one of them is for ZFS. There's also for 10 and 11. And the script goes and looks at the directory to go and figure out what's current versus what's not.
29:43
So I want to jump over real quick to walking through some of this stuff here, because there's
30:04
some important components. So like I said originally, so you pass the variables and values into packer. So in this case, we've got a packer. We're running the program packer build. We're passing in some variables here. These can be loaded from a file
30:21
or passed in through the environment variable or through the script. In this case, we're using the script. And we're calling this template ZFS packer file. So we're going to look at ZFS here.
30:40
So there's some defaults here that you can see up top. We've got checksum, checksum-type. These are things that I manually provided, though there's the script that will actually go and do an LS and figure out what's kind of current. There's different builders. So on my laptop, I've got a VMware and VirtualBox builder
31:02
support, because I have those virtualization environments installed. But there's other builders available, Azure, Google, et cetera. So these boot commands here are useful for creating that initial SSH-able connection. So you can see this is all of the things
31:27
that you would normally step through if you were at the keyboard. What these are doing is it's sending off UDP packets to VNC to go in and perform these mechanical operations over and over and over again.
31:40
In most organizations that have been dealing with 3BSD for any amount of time, they have figured out a way of building their own images and scripted away some of this with their own custom environment. But it should be easy to do this stuff out of the box. One of the things that's very frustrating about working with VNC and why we try and get out of it as fast as
32:01
possible is in this case, we're looking at the 11 current ZFS install. And you can just wait 10, wait 10, wait 10. That's waiting 10 seconds times whatever. I think it's 110 seconds or something like that. And you hope that it finishes that step in time. And somewhere in here, there is the step where,
32:23
I think it was wait 10 here. That's the one that DHCP bombed on. So not terribly useful to be able to interact with that at this level. But you can see like mod 7 0 0, I'm creating user, whatever else. But really what we're trying to get down to is the permit root login and exit, wait, wait, and bomb out.
32:45
So because I have VirtualBox and VMware provided in this, what this lets me do is I have the boot commands up top for VMware. Ordering is not too specific. And down here I have the boot commands. They should be identical for VirtualBox.
33:06
Now we get to the provisioners. So these scripts are executed over SSH. So you can have an array of scripts here. And this lets you customize the golden image for your BSD
33:22
image on a per application basis. So this packer file is what you would use. You would have copies of this packer file, one for Postgres, let's say, if you wanted to create a database, one for your application server, one for your bastion, SSH, mail server, whatever. And in theory, the only difference there is these scripts. And you can share scripts, obviously, between your packer images.
33:40
But it allows for a repeatable process. And that's the important part that I want to get back to, keep coming back to. The provisioner step down here, these are all executed over SSH. All of this work up here is just to create an SSHable image inside of FreeBSD so that you can then come down here and do the highly
34:01
repeatable part of interacting with things over a character device via SSH. And then after you have completed this, there's a post processor step that lets you do something. This is where you can potentially script and add the ability to upload your image to a cloud provider or compress it or do something, whatever
34:22
it is that you want to do. So in this case, one of the things that I like doing is I have a tendency to set log bias to throughput on my VMs because I don't really want fsync to ever cause a flush.
34:42
I want it to know op and then post install, come through and create everything that I kind of want from my local environment.
35:02
Now, the packer repo that I referenced earlier, one of the things that it lets you do is it lets you, and this is, again, thanks to Brad, he did this, not me. What it does is it does an LS off of the ISO images, figures out what's the current, the latest release,
35:20
and then sets up the environment variables, which is why we have this all in a wrapper script. And then there's your template sample vagrant file that you can then use to push up. Are there questions?
35:41
I'm moved through that a lot faster than I expected. And I unfortunately have a, yeah.
36:01
There is not that I'm aware of. Yes. OK, so we're quick, just for the benefit of folks. Can you modify the ISO before you go through the installer provision?
36:20
No. Follow up is why would you want to do that? So the thing that I'm interested in advocating for is that there should be an ISO image like MFS BSD. That's a part that allows us to just download an image
36:42
and immediately SSH into it and bypass the installer and call into the installer in a completely scriptable way. Pushing left, wait one, enter, wait 10, why, enter, down, enter, enter, wait 100 and something seconds, hope that it's done. That's pretty random and less than ideal
37:03
from a reproducibility perspective. But my point is you have to add that to the ISO. And that means that you've already invested in making this happen. In an ideal world, you should be able to go download
37:20
a couple utilities, point at a few URLs, and then go and begin doing something without any bootstrap infrastructure. The bootstrap infrastructure required is basically zero touch and already provided. Go ahead.
37:56
So that's exactly it. And I forgot to mention, Packer, you can create Docker images if you want.
38:01
The whole point is to have all of the steps necessary to have a completely reproducible, verifiable, run it anywhere kind of like build process. Like release engineering in a JSON file kind of a thing.
38:25
So the point of Packer, yes, I understand what you're saying. Yes. So that's the reason that the config files here are specified in terms of URLs. Is that way, potentially, if there is a new ISO, let's say, because there's a security release or something like that, it will slip intentionally
38:45
the version of FreeBSD that you are installing in order to stay current. What you don't want to end up in is in, and this is, I think, the problem that you were addressing or getting to earlier, which is you will end up in a world where you have a fossilized golden image that you can't reproduce. And what we want to do instead is be in a codified
39:02
universe where you can say what your intention is and then have the output of that be completely reproducible. Go ahead.
39:26
Yes. And so you can do that. In prior lives, what I've done is had to go and repack a new MFSBSD to sidestep that. And it'll just sit there with an SSH connection waiting
39:41
to do things after it fetches and does a little setup. You can hit and step through there. You're still going through VNC, though. It does bypass some of the trickery. I wanted to leave it in the default, in this case, specifically make the point of navigating a UI with timing-based interactions is problematic versus something that's completely deterministic,
40:03
like 100% deterministic. Because really what you want to be able to do is like, if I have an error, I'm just going to try it again. No attachment, just blow it away. It'd be nice to figure out why, don't get me wrong. At the end of the day, optimization to figure out what goes on inside of the kernel is really important, and I do not want to diminish that at all.
40:21
But at the same time, how do you go in the KPI of what's the cost of going and deploying 100 and going from an organization that does nothing using FreeBSD to an organization that goes and deploys 1,000 boxes? Because what you, again, it all comes back to what's that initial setup cost? And a lot of times, most companies' people just
40:41
don't care. They just need something up. So how do you bypass a lot of that? If I'm lucky here. Were there other questions, by the way? Go ahead.
41:05
Instead of shell scripts? Oh, you totally could. You would use a shell script to go and do a PKG install, Ansible, go fetch your playbook, and then go do something. Yeah, absolutely. This is headless, because then you would be able to see it potentially stepping through things.
41:21
But in theory, you shouldn't have to do that, and you should be able to just watch it work. And normally, the entire process from start to finish, when it's not timing out, I don't really understand why, to be honest. I haven't had a chance to look into it, because it was working. It takes about four minutes. ZFS takes about six minutes. There'd be an interesting optimization of FreeBSD Golf,
41:42
is what I'm going to call it, to try and get the memory size for ZFS down to the minimum. I want to go and spin up 128, 256 meg instance, because I want to use FreeBSD. I want snapshots. But I don't want a large arc. I want nothing. This is a disposable box.
42:01
I deliberately know that, but I want to be able to take a snapshot. Or I want my checksums. But how do I get that down so that at the end of the day, I'm able to spin up a bunch of 128 meg instances, because that's kind of what I want. I can do that on UFS, but it's not the same. I want to go and spin up a BSD image,
42:21
go and hook up and set up two EBS volumes, let's say, have them a part of a zpool, and then have it be a reasonably small memory footprint box. But I've got some redundancy there on the cloud service provider aspect of things. Here we go.
42:42
So other questions? Yeah. The comment was, we need to talk inside the community
43:01
to figure out how to do some of this stuff. And he's right. Or he's aware of some of this stuff. And so the active conversation is going to happen there. Being able to get to the, yeah. OK, I will make sure to, I will, yeah, well you did the work on a lot of this. So I want to make sure that you're a part of it.
43:22
Because yeah, one of the things that's been interesting in going from megawatt deployment data centers to cloud environments and working with large numbers of customers that work in cloud environments versus data centers is the tolerance for the setup and the bootstrap is vanishingly small inside of the cloud universe.
43:43
So being able to take something off of your laptop and push it straight to production, like it should take me 15, 20 minutes. I should be able to download it, and it should just go. Gone are basically the days of building a custom build world and before you can just go and provision something. You can do that, and lots of us do that, and I do that
44:01
still. But the attitude, there's an entire chunk of the user space that's out there that has kind of evolved their operating parameters. So it's like, I should just be able to spin it up in minutes. And it should be specific to my application, and I need to be able to add it to an ASG, Auto Scaling Group.
44:25
So we need to participate in that. Is this actually gonna work? All right, well, I'll let everybody go.
44:42
We don't need to watch this. Feel free to come up and ask questions, and I'm gonna let this run to completion. So thank you.