Hello, everyone. First of all, thank you for waking up on a very cold and early Sunday morning. Thanks for coming here. I appreciate it.

My name is Thiago da Silva. I'm going to be talking about an object storage system called OpenStack Swift. I work for Red Hat. I'm a Swift core reviewer, meaning a group of people that review and maintain code and merge code.

And I've been working on Swift for about four years or so, since 2004. I thought I'd start with a brief introduction into object storage in general, like what drove the need for object storage, talk about some of the use

cases and what requirements those use cases lead to, and then dive into Swift, look at the functionality, do some overview, talk about the architecture, and then just answer some questions. So it's going to be pretty intro and brief.

So we've all heard a lot about the explosive growth of data that is just being generated. There's been a ton of statistics about the exponential amount of data that is being generated.

And what's interesting about that is that data is being generated a lot, but the rate of that generation is also growing. So it's really an exponential growth. And the second point that's interesting is that what's driving that generation of data growth is mobile applications and web applications.

When you talk about data growth, you need to also talk about what types of data is being created. And I try to break that down between two types of data, what I call structured data, and that is data that is organized in a relational form.

Think of databases. It's data that is easy to analyze, easy to query, typically in a column and rows and in a very tabular form.

Very easy to consume and analyze. And we have also what is called unstructured data. So that's data without any structure or any order or schema. Your typical example of that is all the media files that are out there being shared across the web,

document files, research files. Think of from x-rays to even down to JSON files being shared across the web. What is interesting about unstructured data

is that it makes up about 90% of the generated data that is being generated today. When you talk about types of data, you also have different types of storage systems for storing different types of data.

We all know very well block and file storage. Block, data that is divided into chunks of blocks. We know how it's accessed. It's really well understood. File systems, again, very well understood.

And you have object storage. That is just a logical architecture that is used to manage data as objects or blobs of data. What's interesting about object storage is that when you talk about an object, that object contains the data and the metadata

about that data. And it usually contains a unique ID. How you access that data is through a unique ID. So it has a very flat address structure, for example, compared to file storage where you have directories and you have files.

Object storage, typically, is just a very flat, maybe you have buckets or containers to put your objects into. And that's pretty much it. The way you access it is through its unique ID. And the other point that is interesting about object storage is that, differently from block and file,

is how you access it. Typically, you access the object as a whole. So you're either writing a whole blob of data or you're getting a whole block of data. Again, compare, for example, to file where you might be able to open and append and write parts of it and read just parts of it.

Typically, with an object storage, you put and you get the whole thing, the whole blob of data. The other distinction that I wanted to make here that I think is very interesting is that with block and file, you have local access. So you can think of your SATA drive, your local hard drive,

and you're accessing it locally. Through, for example, your file system, you have XFS file system that you're accessing locally. Or you have remote connections, too, like your Sunbus share and whatnot. But with object storage, what's distinct about it is it's always remote.

Typically, you don't access an object storage through a local connection. So what are the use cases that are driving the need for object storage?

The first and foremost, the typical one that you see are the private clouds. So the object storage market was pretty much created by Amazon S3. But today, there's many public and private clouds out there that are running OpenStack Swift. I tried to list just some example of them. These are all companies.

OVH is based here in Europe. They've spoken at many OpenStack conferences talking about a very large cluster that they have running OpenStack Swift and storing a lot of data. What's interesting about this use case is that typically, if you are maintaining that private cloud, you don't know

what kind of workload you're dealing with. You're just providing a service and your users can be very varied and you don't know. But there are other use cases too. You have what I called web and mobile applications. For example, Wikipedia is an example

that we always talk about where all the media content in Wikipedia is actually stored on a Swift cluster. Which is interesting to think about that is that I don't think the size of that cluster is too big, but the number of users is just tremendous, right?

Terxel is a Red Hat customer. It's a public reference that we talk about. They are a telecom company in Turkey and they've talked extensively about some mobile applications that they built. Something similar to like WhatsApp that allows their users to send messages

and share photos and share videos. And they also built a pretty massive Swift cluster and supporting, again, millions of users. Not only a very large cluster in capacity, but also supporting millions and millions of users.

Another use case is data archival. And what I mean by data archival here, it can be either what I like to call active or passive data archival or cold storage if you think about it. Video companies that are producing or filming on site need to upload their footage

so that it can be whatever they do, processed production in their production site. So Digital Film Tree, for example, is a company that talked a while back about how they were filming on site in Vancouver,

for example, and they have their offices in Los Angeles and they're able to use a Swift cluster to upload their data to the cloud and download that data from their Los Angeles offices and speed up how that production of the video happened. Next Cloud also has been talking here this weekend.

They offer an application similar to Dropbox where you can just share files across many devices or with other people. So again, all these different applications allows you to put data on a Swift cluster. You can put it into cold storage and just back up

or you can actively be sharing content. So based on these use cases, what kinds of requirements does that entail? What can we learn from these use cases? The first one is that you need to provide durability.

So from an end user perspective who is uploading that data, again, going back to that distinction that I made between block and file, with object storage, you're really abstracting the data from where the data is being stored.

I don't want to know, I don't care how that data is being stored, but I need the guarantee that it's durable. It needs to, I can't afford to lose this data. So the object storage system needs to take care of that. From a user's perspective, I don't know where the data is being stored,

what kind of drive, how it's being updated, how refreshed that system is, what kind of backup it does, or if it's using Flash or hard drive, I don't know. I'm just putting my data on a cloud and it needs to be durable.

It also needs to be available, meaning I need to be able to access this data 24-7 if I have to. It cannot be unavailable. That's a huge difference, for example, again, from a lot of times even looking at the SLA for EBS on Amazon or what have you

in terms of block storage on public clouds. It's actually, they provide very low SLA, like 99.9, compared to object storage that has many nines. It's always available. It's always being able to connect.

Accessibility is really how easy it is to use your API and how accessible it is. That's another thing that drove the need, or another requirement that drove the need for object storage in terms of all your web apps and your mobile apps and how they're accessing your data.

They need to provide a very simple-to-use API to access that data. And of course, scalability. Nobody starts with 100 petabytes of data. You start small and you need to be able to grow. You need to be able to have that flexibility for your storage system to grow as you grow.

You need to be able to grow extensively, but you also need to keep it all at low cost. Companies are trying to get away from the storage arrays of old time, and they need systems.

And that's why we have this room. That's why we have the SDS. That's why we talk about software-defined storage. It needs to be able to be running on low-cost hardware. So based on these use cases and requirements, let's dive into OpenStack Swift

and just talk a little bit about that. So Swift is a highly-distributed, eventual-consistent durable object storage system. It allows you to store your data safely and cheaply.

Talking a little bit about the OpenStack Swift community. It's an open-source project. It was one of the founding projects of OpenStack. It was produced by or founded by RecSpace. And what drove RecSpace is that they were a public cloud.

They had the need for an object storage system, so they implemented Swift and then donated that to OpenStack to found OpenStack. What that means, though, is that Swift has been in production running a public cloud for about eight years.

It's very stable. The API is very stable. The system is very stable. And it has a very active community. We have over 700 total contributors. We have about 20 or so active contributors on a monthly basis. And one of the things, like I said,

I've been working in Swift for about four years. And one of the things that I always like to highlight about the community is its focus on automated testing. There's a really huge focus on CI and then making sure that the code that we write gets properly tested. So we have really a huge focus

on how we test every single code that we put in there. As an example, I always kind of thought it was interesting how we merge a recent feature, sim links. And I looked at the piece of middleware

that we're actually providing the functionality. It was about 300 lines of code. And I looked at the tests, and it's about 2,000 lines of tests. That's a big difference. But I think it speaks to the focus of the community in terms of making sure that you're writing good code. It doesn't mean that we don't have bugs or anything, but we focus a lot on that.

And it's a very friendly community. It's been a lot of fun to work on those folks. Quick overview. So you access Swift through a RESTful API. It's all through HTTP verbs. So your GET, your PUT, your DELETE, POST.

The object key, the way you access a given object is through a URL. So differently from other object storage systems where you typically put your data and the system gives you back a unique key, the user itself is who defines

what that key for that object is. So the key is just a URL. You have, like I talked about, the flat address space before. You can group objects, a collection of objects, into buckets or containers. And you can group containers inside a account.

And think of account more like a bank account where you put stuff in as opposed to a user. It's not really, an account is not really a user. It's more like just a group of containers. So it's just a way to organize your cluster

across multiple accounts. I mentioned already before, it's a highly durable and distributed storage system. You're able to load balance your cluster and you replicate the data many times to maintain data durability or you use erasure coding.

But the point is that there's no single point of failure. Your system can be, depending on how you deploy, you can deploy in such a way that you don't have a single point of failures. A interesting distinction about Swift is that it's an eventual consistent system.

So it's designed for high availability and partition tolerance, sacrificing a little bit of consistency. The myth that I like to dispel about eventual consistency, that's the number one thing that I hear a lot is that a lot of people equate eventual consistency

to eventual replication and that's not true. So eventual consistency does not mean eventual replication. It does not mean that Swift, when you put data into Swift, you store just one replica and Swift will eventually replicate your data. Because if that was the case, then you would have a window of time there

that you might lose your data because you might have just one copy and that disk might fail. That's not the case. Swift will always store your data durability with multiple replicas so that when a system goes down or something, your data will not be lost.

Shameless plug. There's gonna be another talk later today where it's actually gonna dive a little bit more into how eventual consistency can actually play a really nice role into the Swift architecture. And that's when we talk about, for example, global replication.

So when you have multiple data centers and you need to have a cluster that goes across or spans across multiple data centers across the globe. That's where eventual consistency really shines.

Swift has a very rich API exposing a lot of functionality. I list some of them there. I'd like to highlight just a couple of them. Object versioning is when you're able to store or if you have a workload that overwrites objects,

you're able to maintain old copies of that object. And object exploration is when you're able to store an object and set an expiration time on it and it will be automatically deleted for you. So the system takes care of that automatic deletion. SLO is talking about,

or is a feature that provides large object support. So similar to other object storage systems, Swift applies a artificial limitation on how big your object can be. And that limit is typically five gigabytes.

And that's really artificial just so that you don't have people trying to upload using HTTP a terabyte object. It wouldn't be a good idea to do that. So we put the artificial limitation of saying five gigabytes is the maximum size by using a feature like SLO or static large objects. You can segment your one terabyte object

into smaller chunks, upload them separately, and then provide a manifest file and Swift will concatenate your objects for you. Temporary URL is another feature where, for example, you upload an object and you want to provide temporary access to somebody else

and you don't want to provide them your authentication secrets. You can produce a signed temporary URL that you can hand it to them, they can use to get that data and that URL will expire after some time. Swift also provides support

for different types of authentication systems. Temporal is one of them, Keystone and OpenStack, FormPost. There's also going to be another talk by Christian today talking a little bit about how to develop your own apps and he'll cover a little bit more about authentication.

Let's talk a little bit about Swift architecture. Swift has a very modular architecture that allows for the flexibility when you're scaling it out. Covered here, four main components. The proxy is your main gateway to the cluster.

So that's how users will send their requests and the requests are received at the proxy. And the proxy will then proxy out those requests to the backend storage system.

The backend storage system is made up of the account server, the container server, and the object server. The account server and the container server, the account is really holding just a list of all the containers for that given account and the container is just holding a list of all the objects for that container.

And some metadata. The account and container server are typically, data is typically stored on a database on disk and it's replicated properly and all that. And the object server itself really becomes kind of the most important part here where that's the service

that is storing your data on disk. What's interesting about this modular architecture is that you could have those four services running in one machine or you can have them run in separate machines independently of each other. They are just web servers running

or implementing the WSGI interface and the communication between them, so not only the communication between the user and the proxy, but the communication between them is also through HTTP. Just giving a little bit more of the flow of the data here.

So imagine, for example, that a user is putting an object, is writing an object. It sends that request to the proxy and what happens is that the proxy will then replicate that data. So we'll write three copies of that data in a load balance and distribute it across the cluster in such a way that the algorithm makes sure

that you're writing the data across different failure domains. So either a failure domain can be a rack, a node, or even a whole data center. So if you have multiple data centers, you would make sure that it distributes your data across two different data centers or if you have multiple racks in a data center,

you would put in different racks. The other distinction that I like about this slide is that you can also break up your cluster between what we call the access tier and the storage tier. And it allows you to scale them separately from each other. So the access tier is gonna be the proxy

and it's gonna be very CPU intensive. So if you need to scale out bandwidth, you might just be able to add more and more proxies. But if you need to grow capacity, you might be able just to grow your storage tier and add more disks and add more storage capacity as opposed to having to add more proxy.

So that's a really neat thing about this architecture. But you can't talk about Swift architecture without talking about the ring. The ring is really about how Swift is able to determine where the data is stored.

So the ring is just a binary file that is using a hashing ring to be able to determine where the data or where an object is stored on your cluster. So that's something that you produce,

the ring as an operator when you're deploying your cluster. And the way it's used is when a object request comes in or a request comes in, the proxy is going to look up, is going to hash the name of the object,

that unique ID, the URL. And from that hash, it can look up where in your cluster the data would be stored. So the location is deterministic by using the ring, and that guarantees that no single point of failure.

Every node in your cluster is going to have that ring, and you can look up that information there. Another interesting point about the Swift architecture is what we call storage policies. So we have the support to store data

across different data storage roles or replicas, we say. So I talked a lot about three replicas to give an example, but you could define really pools of data, kind of break up your cluster into pools of data, and you might have data that needs to be stored with three replicas or two replicas

or original coding. You can also use this to say, let's say, I have a very, for legal reasons or whatever, this data needs to stay in Europe. You cannot go to the US, for example. You cannot be replicated to the US. You could also define a pool that maintains that data just in your cluster or your data center that is in Europe.

So you're not using storage policies, you can do that. And that's an operator-defined role that the operator would define. From a user's perspective, the way you would use storage policies is when you're creating a container, you define what storage policy

that container is going to have, and then every object that you're going to put inside that container is going to use that storage policy. So you could have a policy of flash drives, for example, very high performance or what have you that can be defined by the operator.

I mentioned the four main components of Swift, but there's also what we like to call the consistency engine. That's really background demons that are running in the object storage system themselves and just maintaining that eventual consistency. It's making sure that data healing is happening properly,

bit rot detection, the updates when you put an object, make sure that the container database or the account database is getting updated, expiring the objects. So I gave an example there with my little diagram when the proxy is trying to write to those three replicas and it's not able to write one of them.

It will actually write to another place to guarantee that durability. So it always will try to write through replicas even though it wasn't able to write to its primary location, and eventually it will heal and it will move that data back into place. So that's all done by that consistency engine.

That's it. Maybe at this point you're thinking, yeah, Swift is cool, right? I'd like to contribute. We have a lot of work ahead of us. The work continues. Right now we're focusing a lot on the scalability of Swift.

So container sharding is an issue that we have where there are users out there that are trying to put billions of objects inside the same container, and that you get what we call deep containers, and you need to be able to shard that container. Lots of small files. Again, putting 16 byte files or objects

inside the cluster can become a problem. Data tiering, just being able to move, when I talk about policies, being able to move from a three replica policy, move your data automatically from a three replica policy to a rigid coded policy, for example.

That's another feature that we're working on. We also have a project called Hummingbird that is implementing or porting parts of the system to Golang, and that's for performance reasons. So there's really a ton of work to do there.

And that's it. You can chat with us if you wanna, or you can contribute. You can find the code on GitHub, or you can find us on Freenode if you have questions, if you wanna talk about. And that's it.

Any questions? Any questions? Yes. What is it?

The priority. The priority, so what is the priority of the project Hummingbird? So again, just to restate the purpose. The purpose is to rewrite parts of it in Golang. And the priority, I would say, is very high,

because again, it affects the scalability of the system. But right now, we're kind of prioritizing deep containers a little bit higher than that. So it's something that we're working on. We have actually patches out there that builds the foundation work for being able to port.

And we're really just talking about the object server that we're going to port, and maybe some of the data consistency background demons. So it's important, it's very important to us. But there's more important work, like the deep containers that we need to tackle first. But it's ongoing parallel work that is already ongoing.

I thought I saw, here and there, yep. Being a core project of OpenStack, I understand it means that the release schedule, the release dates are tied with the releases or OpenStack, or does Swift have its own case?

I don't think, so does Swift have a separate schedule from the OpenStack cadence? We, I guess you can say that we do. So actually, a lot of OpenStack projects are following sort of their own cadence,

where they're able to just release their own component. And when there is an OpenStack release, it kind of gets bundled together. And we just tag a specific release that we did, the last release. And we say, okay, this is going to be the Queen's release for OpenStack.

I'm going to, you actually bring up an issue, which is, Swift typically actually gets called out in a nice way nowadays, that you're actually able to run OpenStack Swift outside of OpenStack. You don't need a whole OpenStack cluster to run Swift.

Swift is its own storage system that you can run without any, actually, Swift component, OpenStack component. If you have your own authentication system, you don't even need Keystone. You can just spun up a Swift cluster, and it does not depend on OpenStack,

other OpenStack components. Without...

Yeah, so, I'm going to repeat, make sure that I understood your question. So if you have a three-replica system or a policy, and you want to reduce that policy to two replicas, for it to lower the cost, yes, you can, without downtime, without downtime.

Yes, any more questions? All right, cool, thank you.