all right everybody so as we have here

our talk is going to be about what's going on with storage and new things that are coming up in kubernetes as they relate to storage so how many of you use kubernetes today any of you okay cool so most of you are probably already familiar with the basic concepts and stuff like that so that'll make a lot easier so my name is John Griffith I'm an engineer at Red Hat most of my work these days is around kubernetes Cubert project and storage so those are the kind of things that I do I've been doing a lot of container type stuff for a number of years so it's kind of a thing

a quick recap on some of the basics on storage just in case some of you don't know it's kind of there's some interesting building blocks in principles that we use inside of storage so the base building block is a storage class storage class is basically a definition that allows you to kind of define a type and an administrator can kind of use this in a fungible way to set it up to be a back-end or QoS service or SSDs or something like that there's all kinds of interesting options the next thing is the PV which is the physical volume and that's the actual allocation of storage on a back-end device it's non namespaced it's a global thing it's also something that isn't used directly usually by an end user the end user actually gets into what's called a claim so it's PVC and that's a actual request from a user to use a piece of storage from the PV so just quick breakdown on that so the big thing

about everything that I'm going to talk about is it's all related to CSI which is the container storage interface CSI is the direction for communities going forward in terms of storage up until recently there have been a number of options there have been a number of different entry providers there have been in Flex volume provider and then a whole bunch of different auditory providers that kind of did their own thing the whole intent of CSI was to put together a standard interface and move all of the storage plug-ins to kind of a common behavior or set of behaviors and expectations and things like that and sort of tighten that up we went GA with CSI as of Kade's 1.13 so it's it's a real thing now and now it's kind of in the stage of continuing to grow and getting more and more folks to write plug-ins for it and then also we're working on converters to convert you from old architectures that were non CSI into CSI one thing that I'd like to point out about CSI that kind of gets lost a lot of times in my opinion is that the intent initially was for it to not just be a kubernetes thing so it truly we wanted to do you know docker swarm maze O's all the different things so that's kind of the idea so it's it would be a pretty cool effort all right so now onto

features so everything that I'm going to discuss is going to be centered around CSI there's no discussions about back porting any of these features to flex or anything like that anymore we did that for resize but that's going to be it just because I talk about these things doesn't mean they're actually ever gonna happen so these are things that we have put on the roadmap and talked about most of them have somebody signed up but you know how it goes things could get squashed they could get changed drastically so you never know and then finally if any of you have any use cases or feedback on these types of things I'm gonna ask some questions later but also reach out to me I'd love to get input from folks that are using kubernetes and see what we can do alright so first up volume resizing so

in 1.10 we added support for resizing the volume that was not in use currently getting ready what we were targeting for 114 is the ability to actually resize the volume that while it is an so there's a lot of extra things that we need to do there right so you do things like quiets the filesystem and deal with file systems and all that sort of thing so it's it's it's a pretty massive undertaking I'm not sure when that may or may not hit but that's kind of the plan right now as far as use cases the number one use case that comes up is I have a database MySQL database it's getting full but I don't want to take it down to actually increase my volume now personally I would shut down my pod and increase the volume and then come back if some people want to live on the edge topology awareness this is another one it's kind of interesting so with the CSI plug-in model the thing that's interesting is the plugins are actually external so just like any other pod that you deploy or anything else inside of kubernetes you're deploying a plugin what's interesting about that for some storage devices is they may have requirements in terms of where their software runs inside of the cluster so what's being worked on today is the ability for a external controller or CRD to be running in the controller that actually publishes information about the topology and the different nodes and what their capabilities and characteristics are that way when you deploy a driver or plugin it has the ability to go and figure out which nodes it can or cannot run on efficiently all right that makes sense it's kind of straightforward okay local PBS there's another one that's it's currently available it's targeted to GA and 114 you know typically a lot of people in cluster scenarios are using shared storage of some sort whether it's I scuzzy or NFS or even Fiber Channel it's still still popular but apparently there are use cases where people actually want to just use the local disk on a node alright so that's pretty cool for performance it's great for things like that it's great for doing things like allocating an entire disk just to yourself so you don't have any contention issues you get all kinds of performance and you get some security out of it if you need it that sort of thing of course that means that you can't do fancy things like failover to another node and things like that and take your data which is kind of a problem but anyway that's continuing to advance and be flushed out so I think there'll be some really good stuff in the next release to kind of make that better snapshots snapshots is another really big thing just to kind of recap a snapshot is a point in time copy or record of a volume or data set right your standard storage concept they're made available back in 1.12 as alpha again that continues to grow it runs as an external CRD as well so there's an external controller a snapshot controller that you load up and you run it you're editing it figures it and everything so so some of the things that are currently going on there are adding additional features like group snapshots so actually having a group of volumes defined and snapshotting them all simultaneously and then with that Longo's consistency handling so again we're back to the whole deal of being able to freeze applications quiet file systems and things like that and take take point in time snapshots so I wanted

to give a kind of on some of these an example of what the manifest looks like so you can see it's your standard API type your kind is a volume snapshot content which is our external CR D and then you can define the source which is gonna be your PV C that you want to snapshot ok the next one is inline

volumes and I'll be completely honest with you I've been in a lot of discussions about inline volumes and some of it's still really foggy to me so so currently today include he's with non CSI you have the ability inside of your pod spec to define a volume and it will just allocate storage on the node and and pass that into the container right so the proposal here is with CSI you can't do that today CSI assumes everything is external so the proposal is to actually bring that behavior forward into CSI and let you do that so there are two types of in-line volumes that are being worked on for CSI one is persistent and one is ephemeral so persistent obviously it means it's durable and you can delete the pod and you can get the volume back ephemeral of course it will when the pod is deleted that volume will be deleted as well but in both cases what's interesting is this is not actually a standard PvP VC model this is actually something that's created on the back end outside of that PvP PC context so here's

an example of what the proposed manifest would look like for an inline persistent PVC so you'll notice the thing we have down here is a volume handle so that would be some sort of identifier that the plug-in could pass to the back end of ice to know what piece of storage we're using right and it must exist already so an admin is typically going to have to create that or something like that in the ephemeral case we don't pass that in and what happens is if the backend supports it it will receive an instruction from the from the CSI controller and it will tell it hey I need a piece of ephemeral storage to attach to this node and we'll do that in either case what these do is they bypass the provisioning aspect of CSI so all right now these last two are ones that

I'm working on so I can actually talk in more detail about them so the first one is the ability to do a namespace transfer of a PDC so currently PVCs are named spaced and PVS are not so what that means is once you - PVC and you put data on it that's great but it is relegated to that island right you you can't take it out of that namespace so what we'd like to do is add the ability to transfer an existing PVC into another namespace this lets you do interesting things in my opinion like having a trusted user create a template of some dataset or an image for example if you're doing VMs inside of kubernetes and then provide those things to users in the cluster or even a set of secrets or whatever it might be but it's also really useful for things like volume clones snapshots as well because you can take a clone or a snapshot of these things but what if you want to put them into another namespace inside of your cluster right today you can't do that well you can but it's really really hard and it's very manual and you have to be an admin right so so the idea is to give the ability to do those things the next one that we have is a thing that we call populate errs and the idea of populate errs is to make PVCs inside of kubernetes declarative just like everything else so what I want to do with the populate ER is I want to be able to say hey give me a PVC with this populate ER type that puts you know some data on it right so this could be a repo from github it could be a tgz file on an HTTP server it could be a blob inside of an s3 object store wherever your might whatever it might be so the idea is again you can have back to this model of template type things or known-good data sets and things like that you can actually just have your spec specify exactly what you want and get it in a declarative manner instead of allocating a volume and then populating it with the data and then passing it into your ear pods or your workloads so and we're going to look at these to the proposed

manifest for the populate er I wanted to show what's what's kind of interesting about that is I'd like to have multiple types so this is where you could specify get or s3 or HTTP and I'd also actually like to make this extensible so that an admin could create additional types and implementations on this if they want and this might make a little more sense in a minute when we when we walk through it here's an example of what it would look like if we just swapped it out swapped out get with an HTTP location all right so with that I'm going to

all right is that a decent font size for everybody okay or you want it bigger up a little bit all right I couldn't tell if you were flipping me off or ah I can I can certainly do that

oops so this is not my my laptop so bear

with me all right let's try that's worse

man that's terrible all right there we

go all right let's do that of course my the

the screencast scanner plane it's probably gonna be the wrong color oh that shouldn't should work okay so what I'm gonna do is I got burned really bad doing live demos yesterday so so last night I did screencasts of these I'm learning alright alright so this is gonna be the example of the name space transfer so you can see here what we have is I just did a get name spaces and you can see I have the default name spaces and demo NS or demons those are the two name spaces on the system right now I'm also gonna go and just kind of show there's no PVCs on the system I'll run a cube CTL get PVC using both namespaces just to show and then the short round I'll just show that there's no Peavey's so you can have any PVCs if there are no PVCs but they're peeves so now let's take a look at a spec for a PVC so we just want to create a PVC we're gonna call it PVC one we didn't specify a namespace which means that will go into the default namespace then do a create there if you look now you'll see so we now have a PVC PVC one is bound and it's ready to go the next thing that we'll do is we'll go ahead and check and just verify that in the demo namespace we can't see it so we get no resources found there as we would expect there's our PV so we know the PV is there everything's cool so the next thing we want to do is we want to create this thing called the transfer right so

we have this external controller this CRD running that is a or we have a change running inside of the controller for the persistent volume that is actually going to look for annotations on the PVC that says hey I want to do a transfer okay so right now I'm using annotations for this long-term it will either be a new API type or it will be a label or something else but what this actual manifest is is this is the exact same PVC one manifest I just added this transfer destination annotation right so what's going to happen is the PV controller is going to see that annotation and act upon it so we'll do an apply on that and then now if we look

at the PVC again you're going to notice some things were added so we're a little wide on our output but you can kind of see so here's the transfer destination annotation that I added but then also the where's I guess I can look at this screen the other thing you'll notice is the controller added this other annotation here so that indicates and lets me know that the controller did pick up my annotation and it is pending a transfer so the way this works right now what wanted to do is make sure that I wasn't like introducing a whole bunch of new security problems or quota problems or anything like that so the way this works is this transfer is pending and staged until the volume the PVC is deleted from the user that currently owns it right so what's going to happen is they're gonna delete that volume that PVC and when that gets deleted it's going to trigger the controller to say okay change the reclaim policy on the PV to keep it and not delete it but only allow it to be claimed again by the new namespace with the name that was specified so there's two pieces it has to be in that namespace and it has to be the name that we put in the transferred okay okay so I delete the PVC see you can't see it anymore and I'll show that again it's not transferred yet and then finally you can see now the PV is still there now normally what would have happened in the default scenario without any changes is that PV would have been deleted so the reclaim policy is by default is delete so so now oh sorry so what I did now is

I had I changed in the in the new namespace I went ahead and created the new PVC right so I just did a manifest that had that and so now in the default you see there's no resources found now if I use if I specify the demo namespace and get a PVC there's my PVC right so that and it's the same data its everything's exactly the same it's just issued off of that PV does that make sense anybody okay I kind of forgot to explain some things in the middle there so I want to make sure but so now if I delete it and I didn't put any sort of transfer or anything else it's just a back to the normal behavior the PVC is deleted the pv is deleted everything's gone neither namespace has anything cool alright alright so the next one actually

yeah so we'll run through this next one really quick and then I want to leave some time for any questions unless you guys tell me now you have no questions alright so this one is the popular piece Oh yep yep sorry at least least the colors are there right okay so what we do here is again it's a CR D it's an external CRD we've already deployed it what I did there just now is I went ahead and I just did a scribe on the CR D so that you can kind of see what sort of fields are in there and stuff like that so the next thing we want to do is get populate so popular laters are the object that we create right now we don't have any so I've got this populate or llamo file here this manifest this is going to create that get populate earth at I talked about you can see what we're doing is we're saying hey I want a populate or of type get so it's a github action I want to mount this data inside of my container at slash get so that's where I'm doing a git clone to basically this is the name of the repo that I want to use and this is the branch that I want out of that repo you can also specify a tag and things like that if you want so now we'll just create that populate ur object real quick we can do a describe and you'll see we now have that so that's good and ready to go now here's my PVC specs so what I've done now is I just changed my feet expect to have this thing called a data source and the data source in this case is going to be a populated storage cage down aisle right so before the only data source that was valid was a snapshot API so what I am trying to do or what I would like to see is adding populate errs and P existing PVCs to that data source field so that you can do cloning and you can do external population type things and what I'd like to do long term is build this out almost like an SDK and make it extensible enough that then you could do plugins into this populate er so it would almost be like CR DS plugging into the CR D so to speak which might be a little goofy but anyway so that's kind of the run that's how that works what kind of happens in the background with the populate ER is it sees that data source and when it gets that what it's going to do is it's going to go and launch a kubernetes job that is going to then use the image that you specified and then in the arguments that you specified to actually populate data on to that PVC one of the things that's missing right now is something to keep your actual work pod or application pod from grabbing that before it's populated what we're trying to work on right now is taints and toleration for PVCs that will solve that so what would happen in this particular case is the provisioner would see that there's a data source associated with it that has to do some work and so it will put a taint on the PVC so that nobody else except for that populate ER can use it okay so the question was can I not use the unique container stuff that already does that so there are there is no concept of tanks and toleration for volumes at all so it today so that's that's what we want to change there is paints and toleration for containers and pods the proposal is to carry those over and allow those to be used on volumes as well how do you handle not to have like zombie balloons coming around I mean if you didn't claim the volume in the second namespace so if the question was is how do you how do you deal with orphans if you created a transfer and the end user never actually picked it up right now I don't so long term what we could probably do is figure out if we want to have a controller run that puts a timer on that or make it admin configurable or user configure role that says time to live on this transfer request is one hour one day whatever it is and then automatically delete it if it's never claimed so I think that's a really good point yeah sorry you speak about resize like this done feature but in fact it is mostly useless I'm sorry I'm having trouble hearing with okay so you're speaking about resides feature is like one feature but in fact it is have no sense I mean in general we use PVCs in a stateful sets but resizing stateful set is not supported and which means that hey okay that is actually so stateful sets are one of the things that are targeted to to in the enhancements of the existing resize so you're correct today right now resizes only just on a bare PVC and it's when it's not in use and it's pretty it's not all that useful in your case right so one of the things that is being worked on with the class and everything in the online is the ability to actually carry that through into stateful sets and do resizes on stateful sets as well thank you all right yeah so the transfer that is triggered on the delete sounds a little bit scary for me so why if I don't have everything in place right I delete the data so so the transfer actually right now the way it is is is about as safe as I can come up with to do it because what it's actually doing is running on the existing PV controller and it's just using the reclaim policy Smith semantics but the beauty is it's also a gated feature so you can just turn it off and not do it if it's if it's troublesome if it's something that you're interested in though in terms of the feature but you don't like the idea of how it works or how it looked here I would love for you to take a look at it and give me feedback on how to make it better thank you all right I need a question all right you [Applause]