csync: Roaming Home Directories
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 97 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45695 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201080 / 97
1
2
4
5
6
8
14
15
16
17
23
29
38
41
42
44
46
47
48
50
53
54
62
63
64
65
66
71
74
75
77
78
79
80
82
84
85
94
00:00
Directory serviceSystem programmingDirectory serviceSynchronizationComputer virusComputer animationXMLLecture/Conference
00:29
Software developerDirectory serviceLaptopFlow separationIntegrated development environmentShared memoryBackupClient (computing)WindowWorkstation <Musikinstrument>Server (computing)Directory serviceComputer animationLecture/Conference
01:19
Directory serviceComputer fileContent (media)Condition numberIdentical particlesMetadataEquals signAlgorithmCommunications protocolSynchronizationLocal ringSoftware developerRead-only memoryNetwork topologyInformationDatabasePlug-in (computing)Physical systemVirtual realitySoftware testingSoftware frameworkExecution unitClient (computing)PlanningFlow separationAlgorithmDifferent (Kate Ryan album)MetadataPhase transitionNormal (geometry)SynchronizationDatabaseLocal ringMultiplication signMoment (mathematics)Communications protocolSoftware developerServer (computing)Network topologyComputer fileWeb 2.0Revision controlInteractive televisionWindowDirectory serviceBitMereologyExtension (kinesiology)Library (computing)Physical systemPlug-in (computing)PropagatorVirtualizationFile systemDomain nameCausalityProfil (magazine)FamilySystem callAerodynamicsWordMetreDirected graphDirection (geometry)Nominal numberWeb pageBackupSemiconductor memoryIntegrated development environmentContent (media)InformationComputer animationLecture/Conference
08:42
InformationRead-only memoryNetwork topologyMetadataDatabaseVirtual realityPhysical systemPlug-in (computing)Communications protocolExecution unitSoftware testingSoftware frameworkSynchronizationDirectory serviceIntrusion detection systemAuthenticationData managementPasswordKerberos <Kryptologie>Software developerComputer filePhysical systemPlug-in (computing)PlanningModule (mathematics)Data managementProcess (computing)Multiplication signDirected graphInformationMusical ensembleFeedbackSynchronizationSource codeUnit testingPasswordFamilySystem callUniform resource locatorSoftware testingSoftware frameworkElectronic mailing listBlock (periodic table)Directory serviceAuthenticationAsynchronous Transfer ModeStack (abstract data type)Kerberos <Kryptologie>LoginComputer animationLecture/Conference
11:36
DisintegrationVariable (mathematics)Revision controlAlgorithmCommunications protocolSynchronizationWebDAVDemonContent (media)FreewareGraphical user interfaceCartesian coordinate systemCommunications protocolAlgorithmData conversionPlanningINTEGRALSynchronizationIdentity managementDemonDirectory serviceContent (media)Moment (mathematics)Forcing (mathematics)Flow separationRevision controlDifferenz <Mathematik>Stack (abstract data type)Computer animationLecture/Conference
14:17
Web pageCommunications protocolPlanningConnectivity (graph theory)File systemRevision controlServer (computing)Computer fileBenutzerhandbuchComputer animationLecture/Conference
15:35
XML
Transcript: English(auto-generated)
00:08
Hello. I will talk today about a file synchronizer, which is called C-Sync. It's a bi-directional file synchronizer, and the main purpose of SYS is to provide roaming home directories for Linux systems.
00:29
So, let's get an overview of what this is about. Normally, what you have or what you want, here is just an example with Windows.
00:43
Assume that you have several Linux clients which are, for example, in an active directory environment, and you are working on a Linux workstation, and you have a server with a share where you can put data on, and what you want is, on login, that your data is synchronized from the server to your client,
01:04
then you work, and when you log out, the data is synchronized back, so that you have a backup and have actual data if you, for example, log into another client.
01:21
I want to make sure that this is, that we don't here talk about to synchronize profiles, this is just about synchronizing files, so there is no merge of content or anything like that going on, maybe in the future.
01:42
I want to make sure that you understand what the conflict is, which I will talk about it. Simply assume you have two directories, you have the same file in these two directories, you edit both files, and they have different content, and if you want to synchronize these files,
02:02
you have a conflict, because both files have changed, and I will tell you how we will resolve this later. So, let's look a little bit at the concepts. What we have now is C-Sync, which is a library with a simple command line client,
02:28
and what C-Sync does is, it does the synchronization in several steps, or several phases. The first is that you collect metadata from your file system, so this means that you walk over a file system tree,
02:47
a directory tree, and collect the metadata, and then you check the metadata normally with a database, so this is called update detection. After that you do this for, for example, if you synchronize with a Linux client and the server,
03:07
then you do the update detection for, on your client, and then on the server, and the next step is that you compare the metadata, so you know what you have on the client,
03:21
and on the server, you compare them and decide which file needs to be synchronized in which direction, and the final step is then simply to copy the files, which is called propagation. So, let's look at how updates are detected. Assume that you have, that you get the metadata from a file,
03:46
that we use the modification time to detect if a file has changed, and we have a database with an entry for each file we already synchronized, and then we simply compare the file or the modification time of the file with the entry in the database.
04:07
If there is no entry in the database of that file, then we have a new file. If some modification time is new, then the file has changed and needs to be synchronized. If it is equal, then there are no changes, and if both files on both replicas or on the client and on the server have changed,
04:28
then, yeah, you have conflict. So what do we do if we detect a conflict? This is important. Currently, there is only one algorithm to handle these conflicts.
04:44
It is called the merge algorithm, and this algorithm doesn't need any user interaction. It is used by Microsoft for roaming profiles. The intention is not to be better than Microsoft. The reason is, I've developed this especially for Windows environments,
05:08
where you use Samba or a part of Samba to join a Linux client to an Active Directory domain, and if it detects a conflict, then it will use the most recent file and will overwrite the file on the other replica.
05:24
This means that it is possible that you will lose any data, but it turned out that this is working really great on Microsoft, and so that's why I implemented it in the same way, so that most users who are used to it don't get confused.
05:46
The plan is to support other conflict algorithms in the future, but not at the moment. So, C-Sync doesn't have any special protocol. It uses what is already available.
06:07
So, protocols which are supported are, for example, SMB, with a Web SMB client from Samba to synchronize with a Windows server, or you can synchronize with a Samba server,
06:25
which should have CIFS Unix extensions enabled, or you can synchronize using SFTP, which is done with libssh.
06:40
This is currently supported. It's wrong that this will be supported in this version. 0.43 is already out. I will release the next version next week, which has been updated to use a new version of libssh, and you can use this file synchronizer to synchronize local directories, too.
07:04
The difference between, for example, I think most of you know rsync. rsync is a unidirectional file synchronizer, so it just synchronizes in one direction, so the purpose of it is for backups, and here you have a synchronizer
07:22
which synchronizes in both directions, and you can use it for local directories, too, to keep them completely in sync. Just for some developers who are interested how this is done,
07:41
C-Sync uses a red-black tree to store all the information and memory, so it is really fast, especially on the update detection. And if you do reconciliation, then this is completely done on the red-black tree, and it is really fast, and if you have a lot of files,
08:01
then it can do it in less than one second, normally. SQL database is used to store all the metadata you need for the synchronization. It has a virtual IO plugin system to support all the protocols.
08:20
Currently, these are, as you have seen, SMB and SFTP. I haven't looked at other protocols yet, but... Where is this page? I haven't looked at other protocols, but it would be possible to simply support them.
08:52
And C-Sync has been developed within a testing framework, so if you, for example, want to create a new IO plugin for the system,
09:05
then you can simply run the testing framework and make sure that it works. If you want to now have roaming home directories using C-Sync, then this is done in the PAM stack. There is a PAM module which is called PAM C-Sync,
09:24
and this PAM module does authentication management to get the password, and it passes it to the session management, and this will call the file synchronizer and synchronize all the files.
09:40
The problem here is that the PAM stack is really old, and during file synchronization you are completely blocked during the login process. The problem is that maybe some users get confused here
10:02
because they think the whole machine blocks here, but there is no possibility to give feedback on the synchronization process back to the system and to show it to the user.
10:22
PAM C-Sync has some features, especially it works really good with PAM WinBind, and if you use PAM WinBind, it has a special mode that gets the information from the Active Directory server. It has Kerberos support, so if you use the SMB plugin,
10:47
it will use Kerberos for the synchronization of the credentials. It can exclude users which you don't want that they get synchronized.
11:03
I've implemented some more features to change several locations with a list there because some user requested it, and the source code is also at the same place
11:21
as you can find the file synchronizer called C-Sync. I have some plans for the future, but I don't know if I will find time in this year to implement everything. One thing that has been requested would be free IPA integration.
11:44
Free IPA is an identity management like Active Directory, but only for Linux. Here, what I want is a way that you can define a roaming home in free IPA, and free IPA has to provide a way that it can pass this directory to the PAM stack,
12:08
and I can use it to set up synchronization for it. If you are already using free IPA, you can use PAM C-Sync and C-Sync with the SFDP protocol.
12:25
You just have to set it up on every machine, and there is no central possibility to change it. Some users have requested that it would be possible to integrate GUI support.
12:44
For this, I want to create a conflict algorithm. If you want to solve a conflict, there is one possibility, which is the merge algorithm to always use the latest file,
13:01
and the correct solution would be to ask the user what he wants to do, and for this you need a GUI application. It's really hard to support it with PAM because you don't have any conversation with PAM, and for this you need a GUI application which uses C-Sync to synchronize all the stuff.
13:27
The plan would be possible to support more protocols in the future, like maybe it is possible to talk to an rsync daemon, and to differential just to synchronize diffs.
13:44
WebDAF would be a possibility, and an own daemon which has MIME-type support there, which has MIME-type support too, to merge contents would be nice too. Some users ask if this would be possible, but not at the moment,
14:04
and I'm thinking about a bigger picture of file synchronization, but this needs a lot of work, and for this you need a daemon, but this would be a start to research the stuff. So if you have questions, we have I think two minutes left.
14:25
You can ask them now or grab me here at FOSTA and ask. There is a user guide which describes everything I've talked about in details. Please take a look at it if you have further questions or ask me.
14:56
The plan was completely here to be that you can do everything as a user,
15:06
with protocols which are available, and you don't need to install a special server component. But I'm not sure if a subversion would be suitable for that.
15:25
I think we are handling here files, and the best is normally to stay at the file system, but maybe.