You you're ready perfect right on time. So run out of glass for Lucy and she's gonna be talking about DevOps and Distributed DevOps before

Hello, thanks all for coming My name is Lucy and one of their well with my team I worked as a bunch of things We maintain a bunch of services and platforms and tools that the rest of the developers use And one of those is a thing called PS CLI or piscally which is a CLI We've written in go that basically. I like to think of it as a user experience layer for launching a bunch of docker containers containing

tools that the developers want to want to use So I'm Before I talk to you about this tool I need to give you a little bit of brief background about where it came from

Didn't just bring out of nowhere Gonna talk to you about How it started from really really useful when we made this CLI and go and Then talk a bit about how we've started open sourcing it because this is positive after all So in the beginning the problem we had that we need to solve was new starters at the company

we needed a way of Giving them a single thing that they could install and run all the tools they needed We didn't like the idea of giving them a document saying install this configure this here's some settings no, we wanted them to have a single thing that would for the most part just work and

Our initial solution to that was a vagrant image that contains a bunch of stuff And that worked for a while, but as the company grew bigger that wasn't really a sustainable solution For start teams across the company are sort of semi autonomous and what that means for us Is that while they might use the same tools they didn't necessarily use the same versions of those tools

Which for us meant potentially different versions of VMS for different teams, which is not a sustainable solution These things are fairly big and which annoys developers when they have to download a new one and they take a long time to build which is not a problem in and of itself because you automate that but

It means that the testing cycles when we create new versions this are really awkward but the biggest problem with this was That it took ages to start up and for developers to update to new versions of these which basically meant that they didn't and

A lot of the issues that they saw is to due to having Old versions of stuff and not having the same version on their laptops as on CI So about this point in the timeline when we were getting bored of this Docker for Mac came out of beta and

One of our team came up with a brilliant idea of why don't we just replace this VM with Docker And that looked like this which Yeah, it works But that's not a thing that we want to give to our developers and I mean getting developers to update a VM every now and then is

awkward enough making sure people are running exactly this and Making sure hundreds of people across the company are running exactly this is not going to happen So while this worked in our team and it does everything that the VM did We needed a better way of distributing it

Which is where go comes in So we created a tool called PS CLI, which is written in go it interacts with Docker SDK Same code that the Docker client uses It uses Cobra and viper to wrap stuff up into a nice CLI developers can just run

And there's a lot of nice nice library code that we've got in there as well That we can run prior to launching a container So some of the nice features we get out of this It's able to update itself so developers Previously updating that VM was a pain in this case. It's just one command and updates

Tools that are run with this Run against whatever is in your current directory, so you don't have to clone your code twice once for outside your VM once for inside it But another feature we've been able to add to this is that it's able to run without

Needing your code to be in your current directory. It's able to run against a git repo which stores in a sidekick data container Which in CI is? helpful Another thing that's useful for that is that you can there's a parameter that we've added to it that lets you use a remote

Docker server But all of that's not particularly interesting on its own because there's lots of things to do that and the reason I think that this became incredibly popular at my company is that We are able to add arbitrary code prior to launching a container Some of the things that we do fairly frequently in this tool

include authentic hitting a user with vault Which lets us get access to secrets that the container might need An example of that is anything that needs access to AWS we Use vault to generate short-lived credentials

Which our security team loves us for because now there's no excuse for the most part for long-lived. I am users But it also lets us do nice little things little User experience things stuff that in workshops. I've noticed people make mistakes or find frustrating

Like if they've not configured git or if they've not got an SSH key, so they can't clone anything from bitbucket yet This will detect whether they've got that and prompt them tell them what they need to do so they don't have to ask why is this broken because it'll tell you and Then once we have this tool and basically all the developers had it

We realized that by accident we managed to distribute a thing that lets us run any code that doesn't require containers at all So there's a bunch of stuff in here that doesn't even use containers at all But one of the most interesting things that we've found by using this is

We've got anonymous usage statistics in this so we know which versions of this tool people are using we know What people are running with this? So we found out recently that 25% of people in the past month are using the latest version of this and a further 35% are within a few patch versions

Which was unheard of back when we had a VM that took 45 minutes to update So I'm gonna show off a couple of things that we have in this can I get our friendly neighborhood mic stand I

Feel like I should make people pay for this I'll pay you in stickers So some of the stuff in here and so most of the stuff in here requires access to the work network so Disclaimer I'm faking some of that

So it's built in Cobra so some of the stuff we get for free includes tab completion So some of the commands that include in this there's a lot of them most of them running containers Bunch of global flags that are common to everything I

Mentioned this thing has the ability to update itself So there's two commands that are part of this first one PS CLI version tells you which version you're running And checks artifact II where we store all the go binaries for this check if there's a new version And PS CLI update updates you in place of the new version the inconceivable go updates library is pretty good for that

So oh no, I'm on 7. Oh, that's not the latest version. Let's fix that and now I'm in the latest version. That's like significantly quicker So what else I've gone with a very

Complicated example of one of the containers that we run in here, but it's a fairly common thing we do in this tool In this case. This is the AWS shell which is something that Amazon provides as a wrapper around their CLI, but The reason I'm using this is because it is an example of us Authenticating the user with vault getting secrets getting a list of all the accounts they have access to

generating credentials and Then at that point it knows what it needs to do to launch a container so it launches the container Binds your AWS directory which contains the credentials and also contains the history that this tool needs So I'll show you what that looks like

So if I say basically a diverse shell First it prompts me Friday help that credentials and that's definitely my real username password hunter to How are you typing? Magic, I'm psychic. So there's all the list of accounts that I have access to I picked dev account. I say I want admin access to the account

So it's can downloads docker image This was recorded at a slightly higher resolution. So that normally looks better than that but then at this point, I'm in the container and I just press up and That's the last command. I ran the last time I had this container on my laptop

In this case, it's just checking the content of a particular s3 bucket But yeah, so that's an example of what launching container using this thing looks like sure Yes, those come from vault as well. So the question was when it shows the list of roles

Where did those come from? And yes, those are One of the features of hash called vault as well. So That's where that comes from as well as the list of accounts So another example, it's a terraform. This is something we use to manage our infrastructures code

and This is one of the things that teams across the company like using but they are not necessarily on the latest version so HashiCorp has a fairly frequent release cycle, which means that When they release a new version, it's not necessarily backwards compatible So we so if I do If I'm gonna need my max stand because this one is one that requires me to actually type

So if I ask it which version I'm currently running I'm on one o11 one Two hands typing is easier. I

Did over time and if I can if I specify that I want a specific version It'll use that So yeah, and that applies to basically anything in there But this particular command this is one I use a lot in the workshops and I've seen the frustrations people have so I've added

Just some nice usability stuff to this When you saw the list of accounts for example Anything in this tool that uses AWS you can specify the account as a flag You can also use a config file and by default it will check for a config file in your current directory Or it'll check for one new home directory if you can't find it and also because we manage all the AWS accounts

We know which roles exist. So I'll show you what this looks like Here we go So have a look in the config file. It says that the code that exists in this directory Corresponds to that particular account. So if I run this in debug mode to get some more logging. So what we see

We see that it uses this config file from a current directory. It's chosen to use that account Because we know how this tool works and we know which roles are available in each account We know that plan only requires read-only access so it drops me down from admin which I chose earlier down to read-only and

The rest is basically the same sort of thing runs this in a container in this case It's just some test DNS records. So that's not particularly interesting. So that's that part I said this is open source and Basically this tool has sort become the victim of his own success at work people have been asking us

Can you add this other tool to it? And we've been thinking yeah, that's a useful thing to do in something like this But that's more stuff that we have to maintain and we don't want to maintain lots of things We want to maintain a small number of things So there's some refactoring work we needed to do with this anyway, so we instead of just refactoring that in place

we Split out some of it. So while there's a lot of cool cool stuff in PCI at its heart It's a thing which runs Docker containers and functions in a nice CLI So what we've done we've abstracted away the Cobra and Viper and the Docker

SDK stuff that is part of Piscally and Put that into the open source Kelly and this lets us do stuff like setting sensible defaults Like we basically know that anything that will be written with this Would want the opportunity to run from your current directory or from a git repo possibly in a remote docker

Stuff like that, but their philosophy behind this is that while we are doing stuff by default We want to let people access native docker stuff and use that preference if they have special stuff that they want to do so demo time again

so I've got a version that uses this called loosely named after myself because I'm vain and So I'll show you what one of these things looks like So mains not that interesting. So I'll show you the root package first thing it does

I'm defining CLI variable which is available to all the other commands in this New CLI I'm calling loosely Saying an init function which just sets short and long description sets does the help text stuff I could do all sorts of stuff here. I could define a bunch of global flags, but for this demo. That's all I need

So if I run this it'll show me all the tools available into it and A bunch of global flags so docker host again a bunch of stuff for interacting with git repos if I want to and

Yeah, so I've got so showing it before I show you how this thing launches containers I'm going to show you how it does simple functions and so My version function I've got so from my CLI. I'm defining a new sub command version, which just prints out which version we're running I

Define a function that just prints out a bunch of variables from my from my version library, which I set at build time when I compile the binaries and then the task Which is the run function of this command just runs this function

so if I do Tells me it's running. Oh three one That global debug flag and where is it? If I specify that we can see a bit more information when it was built which get to commit it was built from

Terraform again I'm gonna use this as an example mostly because I really like Terraform and I use this myself to manage my live DNS records So what does this one look like? This one is slightly more complicated. This one actually does use a doc container So again command is a new command from this CLI. I'm saying it's free. Okay

Can you hear me? Okay fantastic You may sit down

So yes in this case defining a new command off my CLI short description shows up in the main Help per text long description shows up when I ask for help text for this tool specifically And I'm defining a flag and in Cali which is the open source version of this a

Flag corresponds to a Cobra flag and it also corresponds to a viper config so I can specify this profile field both when I'm running my tool and in my config file and You saw what that looked like when I did that with Terraform earlier So I'm defining my Task for this which in this case is a string which Cali interprets to be a docker image

So by default it will download this image and just run it I'm also defining an init function which happens first in this case It just takes the value of that profile flag and passes that in as an environment variable So I'm going to show you a bit about what that looks like

So I'm gonna use a git repo for this This is a private repo that only I have access to so it's going to use my SSH key to clone this and I'll use the same commands that I did before plan in this case. I've previously cloned this so this

Sidekick container still exists. It's just checked whether it's up-to-date, which it is and It's going to check if my DNS records are up-to-date Which I think last time I did this they weren't but I'm not changing the live DNS records in the middle of a demo So let's leave that And

Another one, this is the simplest possible example I could come up with for running a container in this I'm using vim just because I like vim So the command for this looks like this So All I'm doing, defining a new command off my CLI, setting the task to this image and I could have just stopped there

I could have used an Alpine image with vim installed, but of course this is not normal vim This is vim plus all my configs, all my plugins. It's even got go format in it Which means when I save it the vim go plugin will automatically format that for me because there's no way I will write go code that doesn't automatically do that for me. So yeah

There's a lot of stuff that I could do with this I could rewrite the entirety of what we've got in PS CLI using this if I wanted to and why will at some point I've got a dedicated 10% of my time at work that will be doing that

But for now, that's probably all I have time for So I've explained where we come from with these massive VMs that weren't particularly scalable I've explained where we are now with PS CLI being this CLI that runs docker images And I've shown you a bit about what the future holds for us with

Kali as this Abstraction that this library that we can just use anywhere we want to make a tool like this Links to source code on screen now, the Kali library itself, my loosely example There's another one that I don't have time to demo today that Someone in our company has made in the past two weeks. Thank you, Alice

This is called Statically and it's what a couple of us are using to manage our Technology blog at the moment and of course Ashley McNamara's fantastic go for images without whom This would be a significantly more boring set of slides So yeah, that's all for me unless anyone has any questions

Sure, so I say that again

See that right so the question was how do I make sure that the user inside the container is the same? user User ID as outside the container and this is something that I deliberately simplified for this talk on Mac OS

It's simple. It doesn't matter because docker for Mac makes that simple on Linux We do actually have some extra stuff that we need to do so I'm figuring out a way of standardizing this in Kali We it's a solved problem that we already have solved at work

Which is you just pipe in the user ID and and you use a group ID as environment variables And then you have something in your entry point which makes Make sure that the user IDs match. It's not particularly nice, but it works Is it

So the Statement was that that's not particularly secure because you can just pass in any environment variable you want and yeah, that's true. It's not Great. It's not it's the solution we have at the moment

We need a better solution in future, but that's what we're working with for now anyone else sure

so the question was How do we? Decide where to mount the volumes from right? So Stuff like the dot AWS directory with the credentials and your other stuff in there We work on the assumption that those are stored relative to the users home directory at the moment

We haven't found anyone at work that stores them anywhere else. If we do then we'll add something to make that configurable Everything else stuff that runs from your current directory. We just say use the current directory and Anything these is home directory use that

Yeah, I hope that Answers your question. So the question was when we mount the current working directory How do we know where to mount that in the image and the answer to that is usually that?

We mount it into a directory usually referred to as usually slash temp slash workspace and then that is the workspace that we Look I use when we launch in the container that's usually how we do that If we need to do anything special for a particular image, we will configure it by image

But that's the default that we usually go with. Thank you. Oh And I have stickers for the logo if you want some because stickers are important and the stickers will not be on this desk

There will be outside just so you know If you want to give a bunch of stickers and we can pass them around we can throw it over you