We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

[nextcloud] Cloud Federation

00:00
subtitles
off
playback rate
1
subtitles
off
English (auto-generated)
playback rate
0.25
0.5
0.75
1
1.25
1.5
1.75
2
quality
576p
20
 Cite
 Share
 Download Purchase
Logo of FOSDEM VZWFOSDEM VZW
 Schießle, Björn

Formal Metadata

Title
[nextcloud] Cloud Federation
Subtitle
sync, share & collaborate in a decentralized cloud
Title of Series
Number of Parts
611
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2017

Content Metadata

Subject Area
Genre
Abstract
The internet started as a decentralized network based on Open Standards. Backthen everybody could set up his own web server, mail server or news server andcommunicate with the rest of the world. The last years we saw, that for manyusers the internet was reduced to a few centralized services used to connectwith their friends, to communicate and to store and share all kind of data.Nextcloud wants counteract this development for people who want to store, syncand share their data. It provides a cloud platform based on Open Standards andFree Software. To connect the nodes of self-hosted cloud platforms, Nextcloudis the driving force behind the concept of Cloud Federation. The aim is toprovide a Open Standard to share data across multiple cloud servers. At themoment this protocol is not only implemented by Nextcloud but also by Pydioand ownCloud. Together with many partners, Nextcloud works on formalizing theprotocols to create a real standard. This talk want to discuss how to restorea free, decentralized and open internet in the area of data sync and share. The internet started as a decentralized network based on Open Standards. Backthen everybody could set up his own web server, mail server or news server andcommunicate with the rest of the world. The last years we saw, that for manyusers the internet was reduced to a few centralized services used to connectwith their friends, to communicate and to store and share all kind of data.This lecture looks at this development from the perspective of a cloudplatform which enables people to store, share and sync all kind of data. Inthis area we saw already the last years a strong push to more decentralizedsolution, developed as Free Software. This was a important step in the rightdirection, but at the same time it created a new problem. Everybody who set uphis own cloud platform can only collaborate with users on the same server. Itbecome obvious that we need a way to connect all this clouds again. Like withemail, for example, it shouldn't make a difference if you communicate withsomeone on the same server or not. In order to build this bridge, a conceptcalled Cloud Federation was introduced. The aim is to provide a Open Standardto share data across multiple cloud servers. At the moment this protocolconnects all Nextcloud, Pydio and ownCloud users. Today Nextcloud is thedriving force behind this development and work actively with many partners onformalizing the protocol in order to create a real Open Standard. This talkwants to discuss the development in the world of cloud platforms and howNextcloud can help to restore a free, decentralized and open internet in thisarea.
00:00
InternetworkingCloud computingComputer animationLecture/Conference
00:34
Cloud computingServer (computing)Uniform resource locatorSoftwareAddress spaceCollaborationismEmailBridging (networking)Computer fileBuildingDifferent (Kate Ryan album)MassSynchronizationNormal (geometry)GoogolProjective planeInformation privacyInterface (computing)Proxy serverSoftware developerPlanningMultiplication signNumberImplementationCuboidPoint (geometry)Link (knot theory)Web browserInternetworkingDigital photographyDirection (geometry)BitLine (geometry)Representational state transferTrailWeb 2.0FreewareInsertion lossVideoconferencingKeilförmige AnordnungGame controllerDenial-of-service attackQuality of serviceDevice driverINTEGRALFood energyOrder (biology)SubsetOracleBoss CorporationData storage deviceTwitterVirtual machine1 (number)TelecommunicationWindowTouch typingDisk read-and-write headRevision controlForestPower (physics)Service PackState of matterArrow of timeSpacetimeDirected graphOperator (mathematics)CASE <Informatik>Musical ensembleComputing platformMathematicsComputer animation
07:52
Type theoryServer (computing)Electronic mailing listComputer fileProxy serverGroup actionCloud computingMoment (mathematics)Link (knot theory)ArmGame controllerCASE <Informatik>Maxima and minimaComplete metric spaceInternetworkingComputer animation
08:43
Server (computing)Workstation <Musikinstrument>SoftwareEnterprise resource planningStatement (computer science)TwitterMultiplication signSolid geometryHome pageGroup actionNuclear spaceMathematicsCloud computingNumberPublic-key cryptographyAddress spaceGame theoryWeb pageMoment (mathematics)Different (Kate Ryan album)WebsiteComplete metric spaceSelf-organizationDenial-of-service attackCuboidDefault (computer science)AreaWhiteboardLevel (video gaming)EmailForestPay televisionComputer programmingIdentity managementMatrix (mathematics)Installation artBitMetropolitan area networkParticle systemSubject indexingUser profileMusical ensembleCanonical ensemblePoint (geometry)MereologyElectronic mailing listFormal verificationToken ringSystem administratorProxy serverSet (mathematics)SynchronizationInformationXML
14:16
Line (geometry)BitDependent and independent variablesCloud computingServer (computing)Data storage device.NET FrameworkClient (computing)ForestWeb pageEmailMathematicsAcoustic shadowAddress spaceWhiteboardCall centreWeb 2.0Public domainGame controllerSet (mathematics)Moment (mathematics)Different (Kate Ryan album)Public-key cryptographyMultiplication signTelecommunicationAreaMaxima and minimaData structureDigital rights managementVideoconferencingPlanningReflexive spaceDirection (geometry)Musical ensembleSystem callBijectionReal numberFront and back endsLink (knot theory)Group actionEvent horizonProxy serverRandomizationConnected spaceLoginPoint (geometry)Information securityElectronic signatureSystem administratorKey (cryptography)CollisionPattern languageRight angleComplete metric spaceMereologyIntrusion detection systemCASE <Informatik>Computer fileUser profileIdentifiabilityXML
19:48
Cloud computingFreewareCommunications protocolOpen sourceBusiness modelProxy serverMultiplication signINTEGRALRevision controlComputer fileServer (computing)Direction (geometry)Open setSoftwareMoment (mathematics)Latent heatSoftware developerInternetworkingPower (physics)Address spaceBitContent (media)PrototypeIntrusion detection systemParameter (computer programming)Enterprise architectureOffice suiteMagnetic-core memoryGroup actionFunctional (mathematics)Error messageUniverse (mathematics)Matching (graph theory)Row (database)Right angleComplete metric spaceReal numberCrash (computing)Field (computer science)Web pageVirtualizationAliasingWater vaporView (database)Classical physicsCASE <Informatik>Workstation <Musikinstrument>Discounts and allowancesEmailSummierbarkeitMotherboardLecture/Conference
25:02
Computer animation
Transcript: English(auto-generated)
00:00
Okay, hello everybody. Great to see so many people here in the room. I'm really surprised how many people are interested in the whole topic about decentralizing the internet and also how many people stay still in front of the door and want to come in. That's really amazing that so many people are interested in this whole topic and all the stuff we just cast here the whole day. That's really amazing.
00:24
My name is Bjorn. I work at Nextcloud at the moment, but I started at Owncloud already three years ago to think about how Cloud Federation could work, how we could set up clouds which works together in a decentralized way. And then we developed this with Owncloud and then I went over to Nextcloud and I will develop this further and I just want to
00:46
present you how all this works, what we already have today and especially also what's our plan for the future, what we want to move to. Did some already use Owncloud or Nextcloud here in the room? And you already tried to use the federated sharing feature? Oh, okay, quite some people. That's really nice.
01:07
So just some introduction. As you know, the internet as it started at the beginning, it was all decentralized. Everybody could set up his own web server or his email server and they could all communicate with each other. But then at some point in time, we entered the dark age of the internet, how I would call it, where we moved to more and more centralized servers.
01:25
And if you look a few years back, or honestly even today, if you look at the mass market, at 90 % of the users, if they want to store their files and to sync them with their devices and share them, they normally go to two or three vendors which are out there, which is mainly Apple D-Stay, Google's or Dropbox.
01:44
Well, they put their files in there and they can do all this stuff. And then they have, of course, all this in one centralized server, which on the one side is nice because they all, of course, know how to develop software. So there is a nice shiny interface. This works all great. And all peoples are there so we can collaborate with all of them. So it looks quite nice.
02:01
But of course, it raises a lot of questions about privacy and who controls the data and who has access to it. So a few years ago, some projects started to develop free software cloud solutions which everybody can host by their own and install their own cloud server to do basically the same.
02:21
And I think this was great work from all the projects out there which did this because there appeared a lot of great software. And then people start to set up their own server, be it their own cloud server, next cloud server, Pydio and C file, whatever you name it. And people just start to upload their files, sync their contacts and calendar with their own devices and everything works nice as long as they stayed on their own server or started to collaborate with the few peoples on the server.
02:46
But especially if you think this a few steps further, for example, like projects like the Freedom Box, where we think about maybe a point in time where everybody has a small box at home where he has all his cloud services, then the number of users on one of the services goes down to nearly one, maybe two or three, depending on how many people live in your building.
03:07
And then, of course, it was really fast to get in the situation that you want to share with someone a file or want to collaborate with someone, but he's not on your server at home, but he has his own server. And so the question is how can you collaborate with these people in a way that is as
03:23
good integrated and as seamless as if the user would be on the same server, so that's the challenge. And that's where we came up with the idea of Cloud Federation, where we said, okay, we need a protocol, a way to create a bridge between the servers so that the user or one server can share with the user on the other server and work with them together on some files or collaborate.
03:43
And for the user, it should be completely transparent. For the user, it shouldn't matter if the other user is on the same server or a different server. To achieve this, we have invented the idea of a federal cloud ID, which looks quite similar to an email address because we thought, okay, most people are familiar with it. They know how to handle it, how it looks like.
04:06
So it's basically the user on your cloud server and then add the URL of your next cloud server. And then we developed some REST API, which is quite small. It's just four to five calls, basically where you can send a share invitation to a different cloud server and then you can accept or decline it.
04:25
And then you have a request to unshare it or to change permissions or send a notification, but it's a really small API and easy to implement. And once this invitation is accepted, once a user on the other server accepted your share, then we create a normal WebDAF mount.
04:40
For us, it was really important at Next Cloud all the time that we don't want to reinvent the wheel. So whenever there is a standard already out there, which is also widely used, we try to use it because I think that's the best way to make it available to many people on different projects. So we choose WebDAF even if we know, especially for syncing and stuff like this, it has some limitations. But we thought that's the widely used standard and most easy to implement for most of the people.
05:06
And as I said, we don't want to create the next island, right? If this would be only implemented with Next Cloud, then of course all the Next Clouds could work with each other or all the own clouds, but we would still generate an island.
05:21
The island would be just bigger. It would be the island of all the Next Clouds or all the own clouds, and you could interoperate across these different cloud solutions. So for us, it was always really important to also implement this in as many solutions as possible. So at the moment, as of today, no matter if you have installed a Next Cloud server or a
05:40
Pydio server or an own cloud server, you can do this federated sharing across all the servers between the users. It doesn't make a difference. All these three solutions already implemented it. And I just talked last week with some C5 developers, which are also really interested in this. So the chance existed maybe in one year or something. This will also work together with C5.
06:04
So let me show some screenshots and explain how this works. Also a little bit historical because we moved from a simple implementation and then advanced it over time. This all started with the idea Next Cloud always had these public links.
06:21
If you want to share with someone outside of your Next Cloud, you could create a public link and then put it in an email or something and send it to someone. And then the person could open the link in the browser and access the file or the photo you shared with them. And then our first step was to say, okay, if you send someone such a link, then add an add to the Next Cloud button at the top of it. If the user also has a Next Cloud, he can click the add to the Next Cloud button, enter his cloud ID,
06:46
and then in the first version, it was then just directly created in web.mount from this file to the other user. So then it was seamlessly integrated in his Next Cloud file listing, his sync line started to sync it, and it was already a great step forward to what we want to achieve.
07:01
Then in the next iteration, we change this a little bit to not create this web.mount directly. But nowadays, if you do this at least on Next Cloud, it's not implemented in own cloud, in Pydio yet, but on Next Cloud. If you go on this open a public link on Pydio cloud ID, then we don't create this public web.mount directly,
07:21
but then we ask the server from the file, please make a federal share with the other user on the other server. This has many advantages. One is that the owner of the file can keep track who has mounted his public link to his Next Cloud, because in his share list, he will then see all the people who click this add to my Next Cloud button and mount it. So then the owner of the file can control the permissions, for example, which permissions
07:44
the different people have on the file, and can remove one of these mounts again. So this makes it already way nicer to integrate. And of course, the next step, we didn't want that people first have to send out this public link, and then people can add to Next Cloud. So the next step was to integrate this in the share dialogue we have in Next Cloud.
08:02
If you are logged in, you have a dialogue where you can share with internal users and group, just type in the name, and then you also have nice auto-completion, and then you share with the user on the same server. And there we allowed now to also type in the federal cloud ID, like you would type in a local user, and then the file gets shared to the user on the other server.
08:22
And again, you have then your listing with the people who have access to the file, all the people, also the internal people, and the external people, and can control all the permissions and revoke the access again, and stuff like this. This works quite nicely, except for one thing, group shares at the moment are not possible. But we are also thinking about how we can solve this problem, that you can also address a group on the other server, so that if you
08:44
want to share with a working group which is on a different server, you don't have to share it one by one by every person in the group, but you can also address the group directly on this other server and share with the group. Another problem, at this point it already worked quite nicely and was quite integrated, but another big problem was always how to find people.
09:06
Because to share with someone, I need to know the federal cloud ID. And then we thought about many different things, also partly some stuff which was just discussed before me in the matrix talk. I think we made a thing about quite similar problems, and maybe it also makes sense
09:22
to discuss this and see if we can maybe even have a common solution for this. One first step we did, and which is already implemented for quite some time in Nextcloud and owncloud, and Pydio now also implements it at the moment, is the concept of trusted servers. So this was a request from mainly research institutes, which has many Nextcloud installations on different institutes.
09:42
And if people want to work together, they want to somehow have them in the shared dialogue, autocompletion of all the people, no matter on which cloud they are. So there the administrator of the Nextcloud server can add trusted servers, which they know, for example, they are in the same research institute or in the same department or the same university. These are the servers I trust, and if the other admin adds the same server, then
10:04
they start to exchange an authentication token and then use this to synchronize the user lists. So then all trusted servers know the users from the other trusted servers, and then you can find them in autocompletion and share seamlessly with them. And if you want to go one step further, you can also enable it that whenever you create a federated share
10:24
successfully with a user on a different server, then your server automatically adds the server to your list of trusted servers. That's basically the idea that as soon as you have created one valid share with a user on the other server, then you can somehow assume that the other server is not a completely spam server
10:40
or a completely stupid server, but then there are users which are really valid users you want to work with. So if you want, you can enable it, and then your network of trusted servers basically grows over time and you will know more and more people you can share with. Then we wanted to take this one step further. That's what we implemented now, just recently a few
11:05
weeks ago with Nextcloud 11, and we want to move this forward with Nextcloud 12 in a few months. And this is some kind of a global address book, so we extended the profile page of the user you have in your Nextcloud where you can not only add your full name and your email address, but you can also use all the other stuff you know from context,
11:22
like your phone number, your address, your Twitter handle, your website, and stuff like this. And then we allow you to define the visibility of all of this information. By default, of course, everything is private, so nothing happens. It's just there for you. But you can decide to set it and add it to context. This means that it
11:40
will sync with the trusted servers of your Nextcloud, or you can set it to public. And if you set it to public, it will sync to a global address book server, which is the same as at Matrix, at the moment centralized, and we are looking at ways how to solve this problem. But then the data you want to make public are published on the server together with your cloud ID, and then people, again, can find you and share with you.
12:05
And we want to use the Twitter handle and the website. We want to do some verification, a little bit similar, like Keybase, that you can sign your presence there, and then you can verify that the person is really the person you want to share with. But, of course, we also need to think further, like, or also verify your email address. Your phone number, as I said in the talk
12:28
before, would be really interesting to verify, but that's not that easy problem to solve, as you may already guessed when you heard the last talk. So our next step is that we want to move from the central server over to a federated server, so that
12:42
you can, like, you can set up your own Nextcloud. Everybody can set up his own address book server, and then configure in his Nextcloud to which server you want to contact, which roles you want to contact if you have a request. And, therefore, basically, we have two ideas. One, we think if you look at customers, big customers
13:02
at Nextcloud, big companies or research institutes, they might want to have just one look-up server or address book in their institute, which all their Nextclouds connect to, but which doesn't federate to the outside. Or it's completely valid that you just find the people in your own organization, or you can decide that you are part of the global network and federate with all the other servers out there and exchange the data.
13:28
So how we thought in the first step, how we solved the problem that we authenticate user and make sure that you share with the right person. At Nextcloud, already, every person, every account has a private-public key attached to it.
13:41
So our idea at the moment is that before we send the data to this address book, we could sign it with the private key. And then an invalid address book server would only accept updates from the user if he changed his email address or if he changed something, only an update would be accepted if it's still signed again with the same private key.
14:04
And we could, at the same time, use the Twitter and homepage verification to also make sure that the data set is signed with the same key, like the homepage and the Twitter account and stuff like this. And to exchange these public keys, we just thought about to do some kind of trust on first contact because we thought
14:20
that the time where you share the first time with the user on the next cloud is quite a random timestamp, right? If someone wants to really get into this communication, it's really hard to guess when this happens, this first share. So we assumed that this is a relatively secure point in time where this happens. And for your first share, you also are sure that you know to whom you share your files, basically.
14:44
And so the first share, when the share connection gets established, you also get a public key from this user, from the server back. So then every time you share again with this user or every time you get an update from the address book from some data of the user, you can use the public key to verify a signature.
15:00
And as long as the signature is the same, you assume, okay, it's still the same user you are in contact with. And if it changed, then you know that, basically, probably something happens. Either the admin of the lookup server changed something in the data or maybe someone managed to upload some false data. So that's an idea we want to try out if it works and just want to push this forward and see how this works at the end.
15:27
So why are we doing this with this global address book? I think there are a few benefits. One, of course, as I said at the beginning, it makes it way easier to find people to share with. And I think that's a big problem we all in the decentralized world have to solve somehow.
15:41
How do we find people if they are not on the same server? And this is one approach we want to try to investigate if this is a feasible way to do it. And one thing which is also especially important for us is if this works, this would allow people to migrate from one server to another. So if you have your own next cloud server and you just change your domain, for example, or if you choose a next cloud provider
16:03
and at some point in time you want to go to a different provider, you could just migrate all your data to the next provider, including your keys. And if you are on the next provider, you push an update of your data set with your new federal cloud ID. And then all the existing shares could adjust to the new cloud ID. And also the new shares you do after that could directly end up at the right user account.
16:24
Another problem we need to solve at the moment at next cloud is that especially in research institutes, you often have an add-up back end to authenticate the users. And it's quite typical that users use their email address to authenticate at the next cloud server. And then the cloud ID becomes quite ugly because then you have two add signs in it because at the front you have the email address with the add
16:45
and then you have another add which refers to the server. So that's a pain we heard already for years from users that they want to solve somehow and we thought about it and we came up with two solutions we want to implement now in next cloud 12. One is that we want to allow the administrator to define a pattern, how the first part of the cloud should look like.
17:05
So in this use case an admin should just define that for every user ID we strip the add from the email address and the email server and just use the first user identifier and the cloud server. Or the next approach will be that we allow the user really to freely in his profile settings choose a complete random cloud ID,
17:24
however they want it to look like. As I said that the admin approach would be for a whole server but he would define a pattern how to do this because the admin would need to take care that he doesn't create any name collisions for the cloud IDs and the server could relatively easily resolve this.
17:43
For the user part we thought that it would be quite nice because many users already have the email address and often use the same address also for example the XMPP address. So it would be really nice if you could use again the same address to also use it to share with people. So they have one unique address where you can email the people, you can chat with them and you can share files with them.
18:01
So that was the motivation of this. And there we thought okay if you put your own cloud ID into your profile settings then you could put on your server if you control the domain either a well-known file which just redirects to the real cloud server or maybe even DNS entry which does the redirection. And then if you do a federated share with this email address basically we would just look there if there is the well-known file
18:25
if there is the well-known file we then would make the share request to the real cloud server instead of to the local server. And the cloud server would provide an API to then resolve just this user-defined cloud ID to the real cloud ID which matches also the server which is out there.
18:43
And of course now I talked a lot about federated sharing with files but I saw that many people already use Nextcloud and owncloud. With Nextcloud and owncloud you can do much more than files. You can also have your contacts there, your calendar. At Nextcloud we work really hard to bring web conferencing to Nextcloud
19:00
so that you can have conferences with people and one-to-one calls and stuff like this. And we want to move federated sharing also in this area. So our goal is that in the future you will also be able to share a calendar just to a different user on a different server or an event or you can share an address book to a different server. And of course with the whole video conferencing at the moment if you try it out you can just call basically a user on the same server
19:24
or a group on the same server or you can create again some public links you know from WebRTC which you send someone and then you go on the web page and then you are in the call. But there we also want to develop something where you can just say okay I want to call this user at this server and then his mobile phone or his desktop client or whatever
19:40
rings automatically and he will get pulled into the call and you can have a call with him. So this was my really fast overview over federated sharing and I'm happy to take some questions. Is there some formal specification for the federation protocol
20:06
or is it just de facto whatever Nextcloud, owncloud does? The question was if there is a formal specification of the protocol for the federated sharing and yes we started out with basically this was a trial and error approach right at the beginning
20:21
where we just implemented an owncloud and see how it works and then it was automatically of course implemented Nextcloud because Nextcloud is derived from owncloud and the patio people contacted us and asked how to develop this stuff and then we helped them to implement this. As I said it's just three of our records. But we are also involved in an initiative which is called Open Cloud Match
20:40
which is an initiative from all the universities in Europe which use this federated sharing quite heavily and with them we are at the moment working on a real formal specification of this protocol. So because we really want that this becomes an open standard which everybody can easily implement and set it up. So that's ongoing at the moment we just finished the first draft
21:00
which is now we have now a discussion around on this draft and we are heading in this direction. In the final share between two servers is it possible to simultaneously edit What was that? Is it possible to edit simultaneously if you have two users?
21:21
At the moment it's not possible because it's just an Oh sorry, the question was if you share a file from one server to another server if it's possible to simultaneously edit a document. So basically work, collaborate on this document at the same time. At the moment this is not possible because as I said it's just a web.daf mount so the file is just mounted over there
21:40
and if people edit it at the same time then basically the person will save the latest which will store this main version and the other version will go in the versions of the file as the older versions. But with the integration we also integrate Collabora Online Office for really collaborating on Office documents and there we want to make it possible in the future that then you can
22:02
which already works with Collabora Office just not in this federated sharing setup but there we want to make it possible of course that you then can really collaborate on the same documents. Have you considered ADSes for the user IDs without addresses? ADSes?
22:20
Yeah, have you considered ADSes for the cloud ID? Yes, we considered it. I also think it's not a bad idea. I even implemented some kind of prototype of ADSes but then people in the next cloud and back then it was the on guard community decided that they want to have it.
22:40
They thought it's too complicated, too disturbing for the normal user, whatever. So it was the contents at the end that now keep it simple and I think now with the new approach that you can define your own federated cloud ID completely independent from the server we are heading again a little bit into this direction. Of course you still don't have multiple aliases but at least you can specify this one address
23:03
the way you want to have it. Yeah? During your talk you spoke about C files but they recently split up. Are you talking to German or with the protocol? The question was, I said that I talked to C file about implementing this federated sharing
23:21
and the question was if I talked to because they split up recently if I talked to the German entity or to the Chinese people. And yes, we talked with the Chinese people because I only followed this on the internet but which was public like most of us I have the feeling that the redevelopment power and stuff like this is by the Chinese people
23:41
and I think that's the people who tried it forward and I talked with them about it. Yeah? Talking about split and forms, what about on cloud and next cloud? If you today have to make an argument that you should choose this solution rather than this one what would be your argument? My argument, I mean we are in first
24:02
and we are all free and open source software developer and my main argument would be next cloud is completely free software. Oh, sorry. And the question was if I would need to make an argument whether you should choose on cloud or next cloud because I mentioned both and I worked at both companies. And so yes, as I said because we are at first and we are all free and open source developers
24:22
or interested in this topic my main argument to this group would be I would choose next cloud because it's completely free software. If you look at on cloud, this has some kind of open core business model. There is this community version which is open source and free software but there is also then the enterprise version which is no longer free software which has a commercial license attached which also has functionality which is not available in the community version.
24:42
So to this group this would be my main argument. This is the complete free software solutions driven by the community and with the community together. Alright, time is up now. Thank you very much. Thanks a lot.