Devly, a multi-service development environment
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 88 | |
| Autor | ||
| Lizenz | CC-Namensnennung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben. | |
| Identifikatoren | 10.5446/37290 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produzent | ||
| Produktionsjahr | 2018 | |
| Produktionsort | Pittsburgh |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
RailsConf 201867 / 88
9
14
16
19
20
22
23
26
27
28
34
35
36
37
38
39
41
42
46
47
53
57
60
62
63
64
69
72
80
85
87
00:00
ProgrammierumgebungSelbst organisierendes SystemSoftwareProgrammierumgebungProjektive EbeneInternetworkingService providerPunktwolkeOpen SourceWeb SiteFreewareDienst <Informatik>CDN-NetzwerkSoftwareentwicklerDiagrammUML
01:07
ServerInternetworkingBestimmtheitsmaßKartesische KoordinatenComputeranimation
01:40
ZeitabhängigkeitSelbst organisierendes SystemEvoluteVorlesung/KonferenzUML
02:32
GatewayStatistikNotebook-ComputerLoopSystemzusammenbruchApproximationAttributierte GrammatikCodeDatenstrukturFormale SpracheMathematikComputerarchitekturRückkopplungSoftwareFrequenzProdukt <Mathematik>SoftwaretestProgrammierumgebungUniformer RaumEntscheidungstheorieAggregatzustandBildschirmmaskeFunktionalGatewayGruppenoperationKomplex <Algebra>MereologieMultiplikationPhysikalisches SystemStellenringVirtuelle MaschineZahlenbereichFlächeninhaltSoftwarewartungVersionsverwaltungGüte der AnpassungÄhnlichkeitsgeometrieNichtlinearer OperatorProzess <Informatik>Strategisches SpielZusammenhängender GraphEnergiedichteATMLuenberger-BeobachterNotebook-ComputerKernel <Informatik>Peer-to-Peer-NetzSystemzusammenbruchGerade ZahlSoundverarbeitungQuellcodeMinimalgradEinfache GenauigkeitMultiplikationsoperatorDienst <Informatik>Mechanismus-Design-TheorieSoftwareentwicklerComputeranimation
08:37
Komponente <Software>SoftwareentwicklerSpezialrechnerTelekommunikationDistributionenraumStatistikDienst <Informatik>BinärdatenQuellcodeProgrammbibliothekZusammenhängender GraphBildgebendes VerfahrenCodeDatenbankMathematikSoftwareStatistikTelekommunikationProgrammbibliothekParserVariableProgrammierumgebungGrenzschichtablösungEinfach zusammenhängender RaumGruppenoperationMereologiePhysikalisches SystemRechenschieberSoftwarewartungKonfigurationsraumVersionsverwaltungServerTexteditorProzess <Informatik>ProgrammfehlerVollständigkeitSchnittmengeAuthentifikationKartesische KoordinatenService providerBrowserInelastischer StoßFramework <Informatik>QuellcodeElektronische PublikationEreignishorizontEndliche ModelltheorieLaufzeitfehlerDifferenteLoginMultiplikationsoperatorDatenreplikationDienst <Informatik>Interface <Schaltung>Web-ApplikationDokumentenserverSoftwareentwicklerXMLUMLComputeranimation
14:06
MathematikPerspektiveTaskZusammenhängender GraphGemeinsamer SpeicherDemo <Programm>SoftwareentwicklerProgrammbibliothekVersionsverwaltungMessage-PassingDienst <Informatik>DokumentenserverXMLUML
15:06
ProgrammbibliothekSchreib-Lese-KopfDienst <Informatik>DokumentenserverKontextbezogenes SystemBildgebendes VerfahrenInformationMathematikProgrammbibliothekTaskMetadatenMailing-ListeKonfigurationsdatenbankElektronische PublikationDienst <Informatik>DokumentenserverSoftwareentwicklerComputeranimation
15:47
ExistenzsatzRechnernetzAggregatzustandMinkowski-MetrikBildgebendes VerfahrenDatenbankSoftwareParallele SchnittstelleLoginDienst <Informatik>Computeranimation
16:33
VersionsverwaltungSocketInformationThreadFolge <Mathematik>ServerAdressraumATMSchedulingEreignishorizontEinsPufferspeicherDateiformatFehlermeldungBildschirmsymbolSkriptspracheRechnernetzMathematikBefehlsprozessorBitInhalt <Mathematik>KonfigurationsraumExogene VariableWeb-SeiteBrowserQuellcodeElektronische PublikationSystem FLoginDienst <Informatik>Lesezeichen <Internet>Computeranimation
18:06
VersionsverwaltungE-MailBildgebendes VerfahrenMathematikDatenfeldSchnittmengeQuellcodeLoginDienst <Informatik>GruppenoperationBrowserXMLUMLComputeranimation
19:23
Migration <Informatik>Kartesische KoordinatenQuellcodeMultiplikationsoperatorDienst <Informatik>XMLUML
19:52
Web-SeiteQuellcodeMigration <Informatik>SoftwaretestVersionsverwaltungGruppenoperationE-MailBitrateBildgebendes VerfahrenTaskMigration <Informatik>CASE <Informatik>VerzeichnisdienstDatenfeldVollständigkeitNabel <Mathematik>Web-SeiteBrowserQuellcodeLoginMultiplikationsoperatorComputeranimation
21:09
TaskMinkowski-MetrikExistenzsatzRechnernetzMigration <Informatik>ProgrammbibliothekQuellcodeGebäude <Mathematik>TaskMigration <Informatik>Dienst <Informatik>Funktion <Mathematik>ProgrammbibliothekParametersystemNamensraumMinimumMultiplikationsoperatorXMLUML
22:28
FehlermeldungDatenbankTypentheorieIntegralSoftwaretestTaskMereologieMigration <Informatik>Elektronische PublikationKomponententestLoginDienst <Informatik>Suite <Programmpaket>DokumentenserverSoftwareentwicklerComputeranimation
24:21
Kollaboration <Informatik>GruppenoperationStabTotal <Mathematik>ComputersicherheitCodeRückkopplungVerzweigendes ProgrammComputersicherheitArithmetische FolgeDienst <Informatik>Bildgebendes VerfahrenProgrammbibliothekIntegralSoftwaretestVerschlingungDatenfeldQuellcodeLoginDokumentenserverXMLUMLComputeranimation
26:10
VerschlingungBildgebendes VerfahrenCodeProgrammbibliothekGebäude <Mathematik>SoftwaretestProgrammierumgebungAnalytische FortsetzungStellenringZentralisatorVerschlingungKonfigurationsraumRegulärer GraphVerzweigendes ProgrammATMComputersicherheitKonfigurationsdatenbankQuellcodeElektronische PublikationFahne <Mathematik>DokumentenserverKontinuierliche IntegrationComputeranimation
27:28
MittelwertSchnittmengeDienst <Informatik>Bildgebendes VerfahrenFormale SpracheSoftwaretestProzess <Informatik>QuellcodeSoftwareentwicklerXMLUML
28:02
AnalysisBildgebendes VerfahrenProgrammbibliothekGebäude <Mathematik>HydrostatikMAPIntegralSoftwaretestProgrammierumgebungGrenzschichtablösungGruppenoperationInhalt <Mathematik>UnordnungComputersicherheitEin-AusgabeKartesische KoordinatenSoftwareschwachstelleDienst <Informatik>PortscannerXML
29:06
Einfacher RingGebäude <Mathematik>PerspektiveRückkopplungProdukt <Mathematik>GruppenoperationKonstruktor <Informatik>MinimalgradUMLXML
30:09
COMGebäude <Mathematik>RückkopplungProgrammbibliothekGebäude <Mathematik>Produkt <Mathematik>GruppenoperationQuick-SortProzess <Informatik>Gemeinsamer SpeicherDokumentenserverSoftwareentwicklerUML
31:02
ProgrammschleifeRückkopplungProzessautomationRauschenRückkopplungSchaltnetzTelekommunikationCodierungSoftware EngineeringGebäude <Mathematik>EchtzeitsystemSoftwaretestProgrammierumgebungTaskFunktionalMereologieMultiplikationPortabilitätReibungswärmeQuick-SortVersionsverwaltungGüte der AnpassungCASE <Informatik>Prozess <Informatik>ProgrammfehlerSystemverwaltungRichtungVerkehrsinformationMultiplikationsoperatorDienst <Informatik>NetzbetriebssystemVorlesung/Konferenz
34:35
Open SourceWeb logMultiplikationsoperatorTwitter <Softwareplattform>SoftwareentwicklerUMLComputeranimation
35:27
Bildgebendes VerfahrenCodeInformationTelekommunikationFunktion <Mathematik>ProgrammbibliothekProdukt <Mathematik>VariableProgrammierumgebungMereologieProjektive EbeneStellenringQuick-SortKonfigurationsraumGüte der AnpassungÜberlagerung <Mathematik>Verzweigendes ProgrammFehlermeldungKartesische KoordinatenInteraktives FernsehenFokalpunktSpiegelung <Mathematik>Elektronische PublikationDifferenteDienst <Informatik>GamecontrollerDokumentenserverVorlesung/Konferenz
38:51
COMp-BlockDatentypXMLComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:12
Hello, welcome to Devly, a multi-service development environment. I am Eric Hodel. I work at Fastly in the developer engineering department on our development environment, which I'll be describing for you today.
00:24
I've written a lot of Ruby code, some of which you use every day. I'm at EZKL on most places on the internet. I've been writing software of one kind or another for more than 20 years now. We currently work within Fastly's site reliability engineering organization focusing on improving the internal engineering experience.
00:46
Fastly, for those who don't know, is a content delivery network and edge cloud provider. We serve traffic for GitHub, New Relic, Spotify, and many other popular websites and services. We also provide service for all Ruby and RubyGMs downloads, and do the same for many other open source projects free of charge.
01:03
Ask us after the talk if you're interested in using Fastly for your open source project. We have servers all over the world which serve more than 14 trillion with a T requests each month. This constitutes more than 10% of all internet traffic, which still kind of blows my mind because it was not always that large.
01:22
We also employ the owners of 100% of the world's best dogs. Very dog from the company. We've got dogs all over the world. We are currently hiring Ruby application engineers. If you're interested, please find Eric or I after the talk. We can provide you with details.
01:41
A quick note before I continue. I currently live in Portland, Oregon, but I was born and raised in a small town called Meadville about two hours north of here. This is my first public talk, so I'm excited to be giving it so close to home. Yeah, thank you for the opportunity.
02:03
So today we'd like to discuss a problem that we believe impacts organizations of all sizes. Can I raise this up? Is that cool? Awesome. So today I'd like to discuss a problem we believe impacts organizations of all sizes. To help us illustrate this problem,
02:29
I'd like to tell you a story about the evolution of Fastly's API. This is a rough approximation of the service architecture that backed the Fastly API circa 2012. The original Fastly development environment consisted of a copy of each component of the Fastly API running on each engineer's laptop.
02:46
Soon after, the early team decided the virtual machine should be employed to provide a degree of operational uniformity and parity between development and production. Another attribute of Fastly in the early days is that all the engineering work was being done by a very small group of people.
03:01
Changes to the systems were easily introduced and distributed through source control, which allowed the teams to rapidly develop and deploy changes. Another side effect of the small size of the company was that focused discussions were possible, and this made decisions easy to communicate. Let's zoom back in and step forward in time a few years.
03:24
Fortunately for us, the company was successful during this period of time, and that success opened doors for new opportunities to expand the business by adding additional functionality to our API. In some cases, when we added new functionality, we added new supporting systems. And when we, as software engineers, like everyone in this room, add new functionality and dependencies to our systems, we introduce complexity.
03:46
I don't mean to imply that complexity is necessarily a bad thing. To the contrary, we would argue that complexity is an unavoidable side effect of growth. There is something else that I haven't mentioned yet that complicates matters even more, which is that we like to use the right tool for the job, so many of our services were written in entirely different languages
04:03
with very different workflows. So despite the increase in the number of languages and services, our development environment stayed much the same. Moreover, the gaps between each group's development workflows grew considerably. This became increasingly problematic as our engineering department doubled in size every six months for a number of years.
04:25
So as a result, our original development environment became increasingly unreliable and established processes to communicate changes broke down. Maintaining any single engineer's development environment was problematic. So we have engineers working on everything from code that runs on the Linux kernel to code that runs on the browser, and
04:46
the needs of each team in those different areas are dramatically different. Our original development environment was unable to meet the needs of one team without compromising the needs of another. This growth continued regardless of our development modes. Companies may continue to grow. That's just what it's going to do.
05:02
So writing and scaling software is complicated, and there are many moving pieces and things to keep in mind while you're doing it. As an industry, we've established and continue to improve upon strategies that help us direct our time and energy. I believe this is due in large part to our ability to observe software systems in isolation.
05:22
Organizations, on the other hand, are far more complex and much harder to observe in systematic ways. By introspecting on our own experiences and listening to our co-workers, we were able to find some themes and common frustrations, of which these are some examples. So, you know, here's your laptop. We'll see you in two weeks when your development environment is running.
05:42
Does anyone know why the API gateway crashes in a loop? I updated the rest of my development environment, and now nothing works. What happened? I can't do my work today because I need to rebuild my development environment. That's clearly untenable and becoming worse over time, increasingly problematic.
06:01
How many people just out there have actually had a development environment like this or experienced these things? Sweet, okay. Well, I'm sorry, but yes, I'm glad it wasn't just us. So, you know, but moreover than the development environment, our friends and co-workers becoming increasingly frustrated with the situation.
06:22
So what to do? During the same period of time, a lot of new tools arrived on the scene, none that met all of our needs. So, through observation, research, and a lot of discussion with our friends, co-workers, and peers in other companies, we arrived at a few important themes. And we believe these themes embody the traits of desirable developer-focused productivity tools.
06:44
The development environment must be reliable. I should be able to run a small number of commands to get what I need running. I should not have to know how every system works to do my job. And I should be able to easily see the local health of systems I rely upon. I should never, ever have to spend a day rebuilding my environment.
07:05
Development environment must be accessible. Maintainers of systems must be allowed and encouraged to maintain their development environments collectively. I should be able to build and test new changes across systems owned by different teams easily. A development environment that spans multiple teams and workflows must be maintainable by the community of folks that are using it.
07:24
Managing changes in source control illuminates past and present ownership, even with many components. So structure and form should be encouraged through convention, documentation, good tooling, and feedback loops, rather than enforced by gatekeepers. We want a development environment to be able to run discrete services together in composable units.
07:43
So it should be really easy to try new supporting systems and swap things in and out without having to worry about writing like a bunch of chef code or doing a bunch of other things like that. A development environment must be reproducible. We need the ability to determine and apply the last known good state of all systems. Source control with similar mechanisms should allow us to determine how we arrived at this known good state.
08:05
And we should be able to leverage existing tools like Git, RubyGems, Perl's CPAN, Python's PIP, and Golang's DEP to arrive here. So through the rest of this talk, we hope to show you how we started to meet the needs of our co-workers at Fastly
08:21
by applying these themes to a tool we've been building together for the last year. We call the tool Devly. To tell you more about Devly, I'd like to hand things off to my friend, close collaborator, and the lead engineer on the Devly project, Eric Hodel. Thank you, Zeke. I will talk about Devly and some of its components and features.
08:43
As Zeke covered, Devly is designed for developers. Devly builds images from your repositories, it uses those images to manage containers, and it enables communication both within and across teams of developers. Devly is...
09:01
Hmm? My slides vote. Okay. Devly is distributed for macOS and Linux. We provide a standalone executable built by Ruby Packer and provide packages for macOS and Debian. Devly lets you configure all of your services. It helps you build images from your repositories using Docker files.
09:22
It allows you to configure those images to run as services, and runs groups of services together as part of a rack. An image contains the files necessary to run a service. The audit log image uses Ruby, so it has a copy of our application code. This code requires some libraries like Rails, Sidekiq, and a JSON parser.
09:43
So the image contains those installed gems, and the JSON parser requires a C library, so we install that along with the OS package system. In our repository, there is a Docker file that contains the instructions for building this image. Images can contain applications for any language. Our stats service is written in Go.
10:03
Its code has a Go binary compiled from the stats application code. The web app our customers use is written in Ember. This image runs a copy of the application code ready to run. We share all these images across all the teams by uploading and downloading them from the Google Container Registry, and this allows us to be sure we're always using the latest images and the latest source code.
10:26
A devly service is a runtime configuration for an image. Here we've created the audit log service using the audit log image. A service runs a command. Since the audit log service provides an API for managing event data, it runs a Rails server to provide HTTP interface for events.
10:45
Our audit log service needs to be able to needs to be accessible to other services so they can read and write events. To allow other services to communicate with us, we expose port 8888. And if you use a development framework like Rails that supports live development, you can mount your repository on top of the files in the image.
11:04
This allows you to work in your favorite editor from your favorite OS. You can change a file on your host OS and see the changes in your browser. This service runs the audit log API, but we also have some sidekick background jobs to run. To make it easier to read our logs, let's use a separate service to run those background jobs.
11:24
Since the background jobs use all the same models and databases as our application, we can use the same image. We create the audit workers service, but we run the sidekick command instead of the Rails server command and the audit worker service. Then we can start up the audit log service, audit log API service.
11:43
It only runs the the Rails server, and when we start the audit worker service, it only runs the background jobs. This separation helps make development a little more accessible because the logs are separate. We can also test our audit workers in complete isolation from the API.
12:01
We'll create a few more services for our applications, including the authentication API, the configuration API, and some databases they use. If we're going to work on the configuration API, we don't want to start up the services that we don't need, and the same if we're working on the authentication API. We create a rack for developing the configuration API that only contains the services it needs.
12:23
We need a MySQL database, the audit log service, and the config API services to do our work. A rack can customize a service. Since we want to access the services running in the rack for development, we expose ports for a few services to the host OS. This allows us to connect to those ports of our browser.
12:43
You can also set environment variables or mount different files to change the behavior of the service. Devly allows you to configure multiple racks. The authentication team needs to work on its services, which include the Postgres database, the authentication API. The authentication development rack also uses the audit log service, just like the configuration team.
13:05
When we start these racks, they use independent containers to run their services. This allows the teams to have different configurations and software versions for the audit log service that won't collide with each other. For example, you can start up both racks at the same time and isolate bugs that span multiple services.
13:23
Using common configuration to replicate services across teams makes sharing your work easier. The configuration for the images, the services, and the racks are in the shared Devly library repository. Fastly, we allow any developer to make changes to the Devly library and have them discuss the proposed changes with people that develop that service.
13:43
The authentication, configuration, and audit API teams all have racks, but using the audit service. When the audit dev team proposes changes to the audit log service, all of those teams need to be able to discuss them. By tracking the connections between teams and services through the Devly library repository, they become more visible, which improves the maintainability of your services and the communication across your teams.
14:07
Now that we've had an overview of the components of Devly and how they combine, I'll show demos of some common development tasks using Devly from the perspective of developers on the various teams you've just seen. We'll run through some workflows like getting started with development,
14:21
sharing changes within and across teams, and set up some convenience tools that will make development easier for ourselves and our co-workers. We'll start at the beginning by setting up Devly as a first-time user. We run Devly setup and give Devly a git repository to pull a Devly library from.
14:40
This downloads the Devly library repository and the other repositories for our services. Along with checking out the repositories, setup performs some additional checks, including the Docker version and your Google SDK version. The setup command will try to fix things it can or give you a message to help you fix it if it can't do that by itself.
15:01
This step takes no more than a few minutes to fetch your repositories and perform the necessary checks. Once setup completes, we can run Devly info to see what racks and services are available to us. Devly will give us a list of the racks and services in our Devly library. We can retrieve information for a rack, which includes the services it starts.
15:23
And we can retrieve information for a service, which includes the image, the repository, and metadata for ensuring the image is compatible with the files in the repository that we've mounted and it's up to date with the image in the registry. Now that we have completed setting up Devly, let's start a rack and perform some basic development tasks like viewing logs, using our service, and making a small change.
15:48
The Devly up command starts a rack. Since we don't have all the necessary images downloaded from the registry, first we see Devly pulling one of those images. Once all those images are downloaded, Devly creates a network to isolate this rack and starts all the containers.
16:02
When containers aren't dependent upon each other, Devly can start them in parallel to speed up startup. Now, let's check to see if everything is running okay. We run Devly status to see which racks and services are currently running. We can see that the two API services in the database are running,
16:21
and we can see that the two API services are accessible to the host OS on ports 8888 and 9999. We can view the logs for the rack by running Devly logs. This command will continue to follow any new logs until we exit with Ctrl-C. Since everything seems to have started for real, let's try out the configuration service by switching to the browser.
16:45
The configuration service was running on port 8888, so when we load it, we see the main page for the configuration API. Now we're triple sure that the rack is working. Let's switch back to the terminal and view the logs from this request. Devly logs shows our HTTP requests from viewing the config API main page.
17:05
Everything is definitely working, so let's do some work by opening up the main page in our favorite editor, Vim. We open the source for the main page from the host OS and add some text. This is an example service, and because no one has ever figured out how to exit Vim, we only save the file.
17:24
So let's switch back to our browser and see if our change worked. Reloading the configuration API shows the text, this is an example service, has appeared. Our change was successful. Let's check the log to see if it wasn't a fake. So of course, the request from the refreshed page appears in the logs, and
17:44
since this has a 200 response with a different page size, we definitely loaded new content. Now that we are done with our work, let's shut down the rack by running Devly down. This stops all the containers we had running. This might save me a little bit of CPU power, but normally our services at Fastly are lightweight, even when a rack has a dozen services running.
18:08
When we work within teams, we'll be pushing and pulling changes to our repositories, and when we work across teams with Devly, the other teams will push images for their services when they have a new set of features ready.
18:21
For this workflow, the audit log team has updated the audit log image to add a source field for events, and we need our services to use this new feature. First, let's see if we already have the source field by loading the audit log service in the browser. I see the user ID, the timestamp, and the action fields, but no source field, so we're using an old audit log image.
18:42
Let's switch to the terminal and update our image. Since we verified that we don't have the source field, we'll pull the latest image, and sorry about the lack of output, it's a bug. Our running audit log service is still using the old image, so we need to shut it down and start a new image. We can do this with Devly restart, which will replace our audit log service with a new one running the updated image.
19:05
Let's switch back to our browser to see the updates. We refresh the audit log service in the browser, and see that the source field has appeared as we expected it. Now that our audit log service is running the latest image, we can continue updating our service to use the source field and the audit log.
19:24
So far, we've worked outside the container, but sometimes we need to run commands from inside the container where all our dependencies are loaded. So let's pretend now that we're on the audit log team, and we'll go back in time a bit. We're now working on adding that source column to our database, and to do this we need to run this migration
19:41
we've just finished writing. We can't do this from the host OS because none of our applications' gems are available. They're only installed inside the container, so we need to run the migration from inside the audit log service. Let's start with the browser again. We view the audit log home page, and of course we don't have the source field because we haven't run the migration yet.
20:03
Devly exec lets us run commands inside the container. We don't exactly remember the image layout, so we start a bash shell so we can explore. After the shell is open, we remember to change the audit log source directory. Then to check to make sure we're in the right place, we run rake capital T. Then we can run the migrations.
20:22
We see that the migration said that it added the source column, so let's go back to the browser and check it. Reloading the page in the browser shows the source column migration is complete. Now that we've remembered where the rake tasks live, let's run the command directly so we can use our shell history in case we need to roll back and retry the migration if there was an error.
20:43
So we switch back to the terminal and run our migration using Devly exec with a complete rake command line. Now this is a little better, but it's really only okay for this one task this one time. When we share this work with our other teams or team members, how will they remember how to run the migrations?
21:02
What we've done is not very usable, and it would be nicer if the migrations ran automatically when we started a rack so we can get to work right away. We can automate running the migrations at rack startup using a post build task. The post build tasks run for a service after the rack starts to perform any extra tasks and the extra setup tasks
21:21
you might need, such as the migrations like we saw or seeding data. This lets users who are unfamiliar with a service get to work right away. Post build tasks live in Devly library and are built as rake tasks that Devly upruns. The tasks live in the post build namespace. The task is named the same as the service it will run for.
21:45
The rack argument allows Devly to run migrations on the correct service if you have multiple copies running in multiple racks at the same time. Here we'll run Devly exec on the audit log service, just like we saw running the migrations earlier,
22:01
and we'll use the same rake command line to run the migrations. After shutting down the rack, we can start it up again with Devly up. We go through all the steps we saw before from the starter rack section, then at the bottom we see Devly exec run our migrations, including the migration output.
22:21
Now, whenever someone starts our service, the migrations will run automatically, so they won't have to look up or ask what to do. Of course, we still need to run migrations during development. For the next time, we want to change the database schema. Shutting down and starting the rack takes several seconds, and we don't want to want to take all this time.
22:43
To make this easier, we can save the long Devly exec migration command as an easy to remember command. We don't want to have to remember or look up or type this long command to run migrations. Let's give this command a friendly name that's easy to type and remember.
23:02
The saved commands live in a Devly YAML file Excuse me. Saved commands live in the Devly YAML file for the repository we are working from, here the audit log repository. Each repository can have its only Devly YAML with custom commands. Let's zoom in and look closer.
23:24
The run commands are a collection of the friendly command names that we want to run. I chose audit migrate as the name of the command which will run the migrations. The command runs on the audit log service. And the command line is the one we've seen earlier that runs the database migration task.
23:46
We can also define a test command that runs the tests inside the service. This way anyone can run the tests where all the dependencies are up to date. So now we can Devly run migrate from the audit log directory, and we see the migrations run.
24:02
Or we can run the tests. Since these tests are inside a container, which is running as part of a rack, they may communicate with other services in the rack. You can have separate racks where one is configured to run unit tests that don't talk to other services. And a larger rack with more services that runs integration tests. Either test suite could be started from a saved command.
24:23
Sometimes we have to work on a service together with another team and Devly has a workflow for cross team development. The audit log team is working on some new high security features. Their work isn't complete yet, but they want our feedback before they continue and make something that's too difficult to use or integrate. To give them feedback, we need to work with their work in progress branch.
24:42
We were told that if we went to the audit log page, we would be running the correct code if a high security logo appeared. We go to the audit log page and see the same one as usual. No high security logos anywhere. So we'll need to switch to their branch. The high security branch may have new dependencies that our image doesn't have. So we can't mount the copy of the updated branch on top of our existing image because updated gems and code won't be there.
25:06
We'll need to build a new image to be sure everything will work. To build the new image from the high security branch, first we need to tell Devly to use our repository we control. We use Devly link to tell Devly about our copy of the repository.
25:21
This will let us build an image from the correct branch. We see that the audit log repository is now linked inside of the Devly library. Next we change to our repository copy and we check out the high security branch. Then we use Devly build to create a new image for the audit log service.
25:46
Now that our new image is built, we can restart the audit log service. We use Devly restart again like we did when we pulled the audit log image that had the source field. Now when we reload the browser, we can see we're using the high security branch because the high security logo is present.
26:04
We can now do some test integration with the new code to give feedback to the audit log team on the high security features. As adoption increases, we'll want to centralize image building through continuous integration, so you always have an up-to-date images in your registry.
26:20
By running your tests through Devly, you have a more consistent environment because the image, service, and rack are all built and configured the same way both continuous integration and local development environments. Regular Devly setup may take too long in a CI environment as it performs more checks and retrieves all the Devly library history. The CI mode for Devly setup reduces the history and repository fetch to save time.
26:45
The CI environment has a repository checked out to the correct commit already, so we can use Devly link to use the correct source files. Overwrite flag makes sure we replace any existing files. The new code we're testing may have new dependencies, so we need to build a new image, just like we did with the high security branch.
27:06
Finally, we start the correct rack for testing this service, then everything will be ready to run tests, same as our local environment. The next thing to do is run our tests. This uses the saved command we saw earlier.
27:20
If the tests successfully pass and we're on the master branch, we can then push our new config API image to the registry to share this image with all the teams. Adopting Devly has given us a common way to start more and more of our services. Once you have a sufficient set of teams using Devly, you can build on top of this capability beyond the workflows I've demonstrated.
27:41
All the workflow demos I showed were for Ruby applications, but they are no different for developing a Go application, which runs a compiled binary inside of an image. With a Go app, you edit the source code, create a new image with Devly build, and test it with a saved test command. This makes the development process more accessible, as you don't need to learn as many new things when working on different languages.
28:03
I've already shown the basics of CI and Devly, but with a common way to start services and run their tests, you can go beyond running tests for a single service. Services you build are composable into larger racks. The more services in a rack you have, the closer to a real deployment you come, so the easier it is to run integration tests or end-to-end tests across your services.
28:23
By ensuring your images, services, and racks are reliable at every level, you can more easily move your containers toward deployment. The image as the base of all your services makes the contents of any application accessible to various security scanners. This allows you to run internal compliance processes,
28:42
run vulnerability scans of the libraries you're using in your images, or find issues through static analysis. You can perform enhanced testing, such as a rack for fuzzing, where bizarre inputs are sent to your service that try to break it. You can get started with chaos engineering from an isolated stable environment.
29:01
You can build separate staging environments for groups of services, or to run integration tests. And Zeke will share with us a few things we've learned while building and collaborating on Devly with our co-workers. One of the first things we learned is that finding early adopters is key.
29:26
We were fortunate enough to have a really diverse group of early adopters with varying degrees of experience who were willing to provide us with constructive criticism early on. We had a few early adopters of previous container experience who were able to provide us with feedback on containerization and orchestration strategies,
29:40
which was really, really helpful. We also had a few early adopters who were relatively new to the company and who had little or no previous container experience. All of our early adopters made Devly an early product, a better product early on. And especially with the folks who were new to the company and had little to no container experience,
30:04
their perspective was key because accessibility is one of our desired traits. So on top of the feedback of our early adopters, their advocacy helped us increase our internal adoption from 5% of product engineering groups to over 50% in a little over six months,
30:22
which was pretty fast. So with that kind of adoption rate, we learned the importance of building and sustaining a know-good and supportive community within the company very, very early on in the process. So we talked very openly about our plans, successes, and especially our failures, acknowledging the fact that we're making mistakes and we're learning with everybody as we're going along.
30:42
As of today, our Devly library repository has 30 contributors and includes people from almost every team in fast engineering, which is kind of pretty rad. So all this fosters a sense of sort of shared ownership and togetherness that has been really important in the development and adoption.
31:05
Establishing feedback loops with the people, with this community developing is very key. The more heard people feel, the more likely they will be to talk and ask questions and provide feedback. And that's ultimately what we're trying to do is to get people to talk more. It's kind of interesting that as software engineers we're building tools and we're
31:23
kind of thinking about everything as being codes and requests going between things, but really the communication is what a lot of this comes down to and where a lot of the problems and noise gets introduced. One of the ways that we're really working to establish these feedback loops is to sort of acknowledge, discuss, and ticket bugs that our users are finding very, very quickly.
31:41
And when we fix them, we let the people know that we reported them and we try to get them involved in sort of the review process to make sure that we've actually solved the problem that they're seeing. It's been really important. Documentation is really good. So good getting started docs can reduce friction and it absolutely needs to be maintained.
32:02
One of the things that we did early on that I think was really impactful was to have sort of like separate operating, per operating system, getting started directions. So that people would just say, just go to the docs, so that they're actually accurate. That in combination with the packaging that we're doing lets people say like, here's your laptop, install this package,
32:24
go through these directions and you actually have something that's functional within a pretty relatively short period of time. Our desired target was sort of to go from multiple days for the time to sort of a new development environment to 15 minutes, and we've been achieving that for three or four months now, I think, which is pretty kind of phenomenal.
32:47
So we also, because we're building this community, we want our users to know that they're free to update the documentation if they want to. We don't tell people like, go update the docs yourself, but we'll do it as well, but we want it to sort of become a shared resource that everybody's using to learn from and teach their peers.
33:06
Another thing to document, and this is something I think was a major learning that he helped enforce, that I was not doing a very good job at, was documenting the administrative tasks and release processes that we were using to get these packaged things out. You'll be very glad you did that.
33:22
Because the last learning was to automate everything that you can as part of your internal pooling. It turns out that QA automation for cross-platform CLIs is really, really hard. I think QA in general is really hard, I find it really hard, but doing it when you're dealing with multiple operating systems and a lot of moving pieces makes it incredibly hard.
33:43
So despite the deadly CLI having 97% test coverage, we find bugs in weird state-specific edge cases all the time, or I guess our users do. And so the more automation that we have, especially around our release process, is proving to be a real time
34:01
saver if you're going to be releasing a lot of new versions pretty rapidly and delivering them to end users. We've also started to extend both the tooling and test harnesses so that we're able to check for and report common issues around things like the rack schemas. We're beginning to make it really, really easy for people to run a rack, and then we do some sort of
34:20
automated testing against it to see if all of its services are exposing the right ports or if they're doing anything that might be untoward for the environment so that we have a way to inject new conventions into our CI process. We regret to say that Devly is not yet open source. Supporting our users at Fastly and preparing for
34:42
this conference did not allow us the time to prepare Devly for open source release, but we are very close. Watch us on Twitter and probably add Fastly as well. We'll post something on the blog almost certainly when it does go open source. Once again, we are hiring. If you're interested, talk with us. We're both very nice.
35:04
We'd like to thank our reviewers, and there are a lot of them, for helping us bring this talk. We'd also like to thank the Devly users who helped us make it a great tool for development. It legitimately took a village, and we have one at Fastly with a lot of really supportive folks.
35:23
Here are the credits for logos and things that we used. Thank you. Thank you for your time. Are we managing configuration for services? The Devly library contains how the application code gets shared within Devly and is a reflection of how it will run in production.
35:46
Inside of the service, if there's any, whatever ports it needs to connect to for other services, those are handled internal to the image. Those live in the repository. Anything that connects two services together, like exposing the ports
36:02
or setting environment variables to run a certain way, that lives in the Devly library. How do we manage two projects having the same service?
36:21
Many of our racks reuse services. I think the most used service is used in 15 racks or so. And that one, the configuration inside of Devly is very small, because there's been a convention about talking within all the teams that use it.
36:42
We're going to run on this port, we provide this API, and there's the regular communication that Devly is facilitating between the teams that use it. So Devly doesn't necessarily need to manage the service. It more is a focus on managing the communication that needs to happen to share that information.
37:01
And the service configurations can also be overridden if they want to do something different. And we provide some sort of local environment override as well that can allow you to point it at different branches and those sorts of things. So the question was, this looks a lot like Docker Compose. What's the relationship? So we started by using Docker Compose.
37:23
We found that Docker Compose's features seemed more production focused. And we wanted to have more control over what commands our user had to type, what error messages we would give, because if we used Docker Compose as another command line tool,
37:42
so we'd have to parse its errors, convert those into something that we could tell our users, this is what you do to fix it. So because of that, we are borrowing large parts of the Docker Compose schemas, but we provide all of our own Docker interactions. We have our own client for that.
38:01
We did a hackathon project to kind of kick this whole thing off that I sat in a room for three days and wired up Docker Compose, Docker, and a bunch of Rake files and made a monorepo.
38:23
And then sort of we did it that way, and pretty quickly it was like, yeah, this is going to become unmanageable. It was also a really good opportunity, I think, for both of us to really learn a lot more about the Docker APIs and sort of what's sitting under the covers. And how it's been good as we're looking at using containers in production as well, to actually have a pretty good idea of what's actually happening.
38:45
Thank you for your time. And you can speak with us afterwards, outside.