Interacting with Bareos Using Bareos
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 611 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/42115 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2017 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2017536 / 611
10
14
15
16
17
21
22
24
25
27
31
36
40
42
46
50
55
56
63
70
73
78
84
94
101
102
104
107
108
109
110
111
112
113
114
115
117
119
122
123
126
127
128
130
131
132
135
136
137
138
141
142
144
145
146
150
151
157
158
159
160
162
163
164
166
170
171
173
175
176
177
179
181
184
187
189
191
193
194
199
200
205
207
208
209
211
214
218
219
222
223
224
225
226
229
230
232
234
236
237
238
239
245
248
249
250
251
253
255
257
258
259
260
261
264
265
266
267
268
271
272
275
277
279
280
282
283
284
287
288
290
292
293
297
302
304
305
306
307
309
310
311
312
313
314
316
317
318
319
321
322
327
329
330
331
333
336
338
339
340
341
346
348
349
350
352
354
356
358
362
363
364
367
371
372
373
375
380
384
385
386
387
388
389
390
391
392
393
394
395
398
400
401
402
405
407
409
411
412
413
416
417
418
420
424
425
427
428
429
431
435
438
439
440
441
443
444
446
448
454
459
460
461
462
465
466
468
471
473
477
478
480
483
487
488
489
491
495
498
499
500
501
502
503
504
507
508
510
511
512
514
518
519
520
522
524
526
528
530
531
533
535
536
549
550
554
555
558
560
563
564
573
575
578
579
582
585
586
588
589
590
591
592
593
594
595
596
600
603
604
605
609
00:00
Projective planeSoftware developerStandard deviationVirtual machineInterface (computing)Public domainFunctional (mathematics)Compilation albumCASE <Informatik>AuthenticationOpen sourceVideo game consoleFreewareDatabaseExpressionPlug-in (computing)WritingConnected spaceOverhead (computing)VideoconferencingComa BerenicesGraphical user interfaceBackupEmailInteractive televisionConfiguration spaceMultiplication signData compressionNormal (geometry)Module (mathematics)Data storage deviceElectronic visual displayPasswordClassical physicsParameter (computer programming)Library catalogLaptopSampling (statistics)Client (computing)Moment (mathematics)Process (computing)Conformal mapSystem callService-oriented architectureHand fanYouTubeDefault (computer science)Service (economics)ReliefForcing (mathematics)InternetworkingWave packetComplete metric spaceParadoxObservational studyMaxima and minimaScripting languageFunction (mathematics)Computer fileMereologyPhysical systemCodeWeb 2.0Electronic mailing listType theoryNumberResultantDirectory serviceDirection (geometry)Vapor barrierCentralizer and normalizerRootLine (geometry)Run time (program lifecycle phase)Different (Kate Ryan album)Limit (category theory)DemonMultiplicationSet (mathematics)Integrated development environmentWordLevel (video gaming)Regulärer Ausdruck <Textverarbeitung>InformationSoftware frameworkContext awarenessIntrusion detection systemXMLUML
08:46
Process (computing)Internet service providerMultiplication signClient (computing)Function (mathematics)DatabaseConfiguration spaceSoftware engineeringDirectory serviceData loggerScripting languageInformationNeuroinformatikOnline helpWindowElectronic mailing listData storage deviceType theoryAuthenticationComputer fileTape driveStructural loadMereologyPhysical systemSet (mathematics)BackupComponent-based software engineeringParameter (computer programming)CASE <Informatik>Message passingLatent heatDefault (computer science)Run time (program lifecycle phase)Operating systemVolume (thermodynamics)HypermediaEmailPasswordArithmetic meanDifferent (Kate Ryan album)Asynchronous Transfer ModeLine (geometry)Scheduling (computing)Electronic visual displayTransportation theory (mathematics)Wave packetOffice suiteHand fanState of matterProjective planeFreewareRevision controlMatching (graph theory)Particle systemDistanceBlogINTEGRALPlanningShared memoryMusical ensembleData managementSource code
17:28
Client (computing)Process (computing)Mixed realityComputer fileInformationMusical ensembleBlogDirectory serviceFactory (trading post)Scanning tunneling microscopeFile formatElectronic mailing listFile systemServer (computing)Table (information)DebuggerDatabaseVideo game consoleBackupForm (programming)Level (video gaming)Physical systemResultantMetreStructural loadElectric generatorVirtualizationYouTubeMultiplication signService (economics)Video GenieException handlingConnected spaceComputer scienceVolume (thermodynamics)Mathematical optimization1 (number)PasswordFloppy diskAddress spaceMathematicsEstimatorSystem callInheritance (object-oriented programming)CASE <Informatik>Scripting languageReal numberSoftware developerOnline helpAsynchronous Transfer ModeDrop (liquid)RootDeclarative programmingNormal (geometry)Cache (computing)Field (computer science)Arithmetic meanMetadataData structureWeb browserLatent heatWeb 2.0DemonData storage deviceVapor barrierModule (mathematics)Source code
26:09
Sampling (statistics)Client (computing)Physical systemInformationProcess (computing)AreaBackupFactory (trading post)Volume (thermodynamics)FamilyGroup actionWindowConnected spaceVector spaceFunctional (mathematics)Line (geometry)Asynchronous Transfer ModeVideo gameVariable (mathematics)Musical ensembleMultiplication signAlgebraScripting languageData storage deviceWordServer (computing)Normal (geometry)Scheduling (computing)Data centerMedical imagingLaptopDirection (geometry)CodeSystem callForestVideoconferencingService (economics)Office suiteIntegrated development environment1 (number)VirtualizationDivisorSoftwareMiniDiscPulse (signal processing)Video game consoleLocal ringConstructor (object-oriented programming)Logic gateConfiguration spaceGoodness of fitData recoveryMemory managementSystem administratorSurjective functionDrop (liquid)BootingResultantType theoryLatent heatElectronic mailing listComputer fileLoop (music)File systemDirectory serviceVapor barrierRead-only memoryOcean currentCASE <Informatik>File archiverBoolean algebraXML
34:50
InformationIncidence algebraPlug-in (computing)Musical ensembleHypermediaGoodness of fitElectronic visual displayWordPhysical systemDatei-ServerMedical imagingSystem callSoftwareNormal (geometry)INTEGRALVector spaceServer (computing)BackupFreewareFile formatGraphical user interfaceConfiguration spaceUniform resource locatorDigital photographySlide rulePasswordClient (computing)Hand fanEvent horizonProcess (computing)Functional (mathematics)Connected spaceVideo game consoleLogic gateCompilation albumWeb-DesignerData managementData storage deviceCASE <Informatik>Key (cryptography)MotherboardDigitale VideotechnikSuspension (chemistry)Virtual machineService (economics)MereologyVapor barrierModule (mathematics)Latent heatComa BerenicesData compressionPower (physics)Scripting languageParameter (computer programming)Level (video gaming)DatabaseMachine visionOpen setStorage area networkComputer virusTable (information)BootingComputer fileLocal ringDefault (computer science)VarianceLine (geometry)Tape driveWeb 2.0Ocean currentDemonIntegrated development environmentTouchscreenRepository (publishing)VirtualizationCore dumpGastropod shellOptical disc driveDistribution (mathematics)MiniDiscFile systemData recoveryDirectory serviceDifferent (Kate Ryan album)Social classMultiplication signHard disk driveSource codeXML
43:31
Group actionService (economics)Functional (mathematics)YouTubeInternetworkingBlogAuthorizationMusical ensembleSynchronizationConfiguration spaceCore dumpComponent-based software engineeringPlanningInformation securityMixed realityUnified threat managementImplementationRight angleWordDatabaseCASE <Informatik>Computer fileInformationClient (computing)Task (computing)System callResultantEncryptionBefehlsprozessor2 (number)Axiom of choiceMaxima and minimaSoftware testingData storage deviceTelecommunicationBackupSet (mathematics)Projective planeMultiplication signMechanism designDemonFile systemLecture/Conference
50:14
Computer animation
Transcript: English(auto-generated)
00:00
I'm Jorg Steffens from the Barrios project, and I'd like to show you some methods to interact with Barrios. I assume this talk makes most sense for people who are already using Barrios or Bakula. I hope you're all using it. Yeah? Okay.
00:22
We will see. Okay, then I will start with a short overview about Barrios, and then I show different interaction methods. One way to interact with the system is to configure it. Yes, okay. I will handle this shortly. But the other thing I guess you are more interested in is runtime control of Barrios,
00:45
and the other way around things that get triggered by Barrios. And a short roadmap afterwards. So, how does a Barrios infrastructure looks like? You have a central directory located, which gives all the data.
01:07
It controls all your backup environments and stores its information in a database. You have the storage daemon, which you have one or multiple storage daemons,
01:22
each with one or multiple storage devices attached. This is where your backup data is getting stored. And you have your file daemons or your lines installed on the different machines that you want to backup. You can interact with the director during runtime through the Barrios director console interface,
01:42
and this is used by command line tool or web interface or some real tools. Okay. Yeah, this is the same, just a bit newer. Yeah. Lines can also be some plugins like VMware, MS SQL, NMT, and cluster, et cetera, et cetera.
02:05
Compuration file. Yeah, configuration is handled in resource. So, the director configuration consists of a directory resource, a catalog resource, how to connect to the database, things like this.
02:20
You add lines, the file sets, so the data should get back up, your jobs, pools, schedules, storage. As an example, I put in here the job resource. Each resource got a name. In this case, backup Barrios FD from type backup, level incremental, and then it references to other resources.
02:41
So, it references to the lines Barrios FD resource. It references to the file sets, all resource, to the schedule, and so on, to this different pool. And it is enabled, so you can also disable it. Oh, I missed some lines here, but it's not... Okay.
03:01
If you're not sure what you can type in here, and I'm not building to read the documentation, you can always ask the daemon itself. So, for example, the director, Barrios director, Minus XS, like export schema, will show you all available directives that you can configure.
03:20
If you change a configuration on the director, you can reload the director, so it's immediately working for storage daemon and file daemon, you have to restart the daemon. I hope this has been everything on this slide, but I guess so. Confirmation can be quite large, but it's relatively straightforward, and there
03:44
are modules available for individual shared puppets or that I'm aware of. So far, for the documentation, so it's a confirmation, sorry. So, runtime control of Barrios. You can direct, connect to your director.
04:03
Here's a de-control command, if you would like to handle this on the command line, type in de-control. It opens up a TCP connection to your Barrios director, authenticates, after this gets you an interactive prompt that you can type in your commands. With that, you get a list of all available commands.
04:26
But you are not limited to interactively used, you can also type in some commands in, and then they will be executed and the results will be listed to standard out. So there are quite a number of scripts that just use this very simple interface.
04:45
There are some helper functions inside of de-control to generate, to write back bits. Like, for example, these are the so-called add commands, so you can redirect the output directly to another file.
05:00
You can also handle this part of this command should be outputted to this file, and if you have another output, then the rest will be outputted to another file. You can do the same with input for commands, or some other commands are available. Most commands have also the parameter yes, which says, don't ask me any questions, just do it, so confirm, immediately confirm.
05:28
And if you have missed some parameter, then the command will not start. But it is also important to wait until the job is finished, or in this case, all jobs are finished.
05:41
As said, you have, this uses a direct console interface, and this can be accessed by de-console, which we have seen right now. There is also a Python module, Python barriers, to access this. So the barriers.ui, you see a screenshot of it here, also uses the same connection.
06:05
This is written in PHP, so if you have programmed something in PHP, you can reuse part of the code. And Q3-based interface GUI, but also uses the same interface. But this is not as deprecated, because the whole thing you need is implemented in various web UI.
06:33
You can have it wherever you like, because you just use the console interface, and you can connect to it, in all ways you will connect to it via TCP, so it can be on any system.
06:47
Yes, sure. Yes. Sure. Therefore, I have to say, mostly it is easier to SSH to the machine, and then use the de-control part, and also use it remotely. This is, for example, done on some clients, if you want to give them a specific access to trigger its own job,
07:06
or something like this, then you can have it de-control locally installed, limit access rights, and use it from the client. Also, the database can also be done on other systems.
07:25
Okay, I've talked about the control interface. You should be aware that there are different types of controls, so-called default control, or root console, which gives you access to all resources available in your directory.
07:41
This is normally the console used when you just type in de-control. So, the default configuration, this is, you get the root console. The other console is the named console, or restricted console, and to use this you have to define ACLs for this console. So, in this case, we have a console, and give it the name user1, provide the password,
08:04
in this case, secret, and allow the user1 to execute all commands, except for the delete command. And only access data from the mycatalog database, which is default database, so probably all, but you can limit it.
08:23
And only to the client, client1 and client2, and all these words are regular expressions. So, here in this case, file set, all file sets starting with the name Linux, and then doing job IDs also. If you don't specify ACL for some resource, then you don't have access to it.
08:43
So, you have to specify it, and provide all or some frameworks that you want to assign to it. I walk you through some of the important commands of barrios. Help, it's always important to get the list of all commands available.
09:01
If you also want to know the parameters, you would type in height, and then type help, and then the command name, with the parameters. For one-time information of your different components, you can ask status, status director, which is your information computer director, or you can ask the status of one of your storage systems that's attached to it.
09:25
Or if this is a tape library, then you can ask what volumes are beside in what slots, or ask the status of different lines. You can also ask when the scheduler has planned to execute the specific jobs.
09:44
If you don't specify a job name here, then you will get a list of all jobs, or otherwise you can see when a specific job is planned to get started. The list command is to retrieve information from the catalog, from the database.
10:05
You have two versions, the normal list and the long list, which provides you more information, and you can ask about information about your backups, your clients, and so on. For example, if you are interested what jobs are stored on volume full one,
10:22
then you just list the jobs that are stored on volume full one. Or the other way around, if you are interested where the data from your job 123 is stored on what media, then you ask for the job medias from this job ID. Or if you want to get the information you normally see at messages,
10:42
you can also ask for those messages that have been created during the job, as the job did run. List jobs last is quite important, I think. If you want to get a quick overview about the status of your jobs, then you can type this command,
11:05
and it will show you the last one of all your jobs. So, normally if you don't do this, then you maybe just get all jobs from the last 34 hours or something like this, but with this you also get jobs that didn't run even for a longer period of time.
11:28
What sometimes confuses people on the mailing list is, for example, if you modify or edit file sets, and then you restart your director, and then make lists file set, then you will not see it.
11:47
Because this data is first synchronized to the database, and there has been a job run that uses this file set. The same with clients. This client will only show all the clients where a job did run for this client.
12:02
So, if you have changed your configuration, and want to verify that the director is aware of this, you can use the show command, which shows the configuration. You can also ask for specific resources in here. Yes. Yes.
12:31
Yeah, and then you have commands to execute jobs like backup jobs, restore jobs, or wait for a job to be finished.
12:46
A small example to put this together is, let's say you have already installed your client's file daemon on the new system, and want to configure it on the director. You can use the configure-adds-clients, give it a name, give it a address, and give it a password.
13:01
This command will complain if you have missed some required parameters, or if you have put in invalid data. Otherwise, it creates a resource file, a population file, for the new client, and immediately loads it. So, you can directly start to use it, and check if it's really working, restart this client, and then the client name.
13:26
The same you can do with jobs, so you create your jobs, and after these two commands, three commands, you can respond your job, and... No. Not here. Not if you use this command, then the director creates it automatically.
13:54
Yes. Okay. You can also specify all these specific parameters, but if it's a common default job, then you only have to spend money.
14:03
Yes. It will create configuration files for you, and immediately loads it with the partage, one of the partages. In this case, it also creates part of the configuration file that's needed on the client. So, if you look the other way around, you first add it here, then it has also a directory export directory, and you can copy it over to your client.
14:33
No. Not with this, because you have not the knowledge about this. You want to add some scripts that helps you with this, but we haven't done so.
14:43
Okay. For Windows, we have the installer, and this has silent install options that you
15:01
can use to configure it, also with the director, with the password, and things like this. We also provide Opsi packages for the Opsi Windows management, software management system Opsi, so you can integrate it in your Opsi environment, which you hopefully have, and then install it and configure it.
15:22
This is all things that work well, sometimes they don't, so you can also enable debug during runtime. You do this with set debug, specify the level, handle it with a starting volume, you specify that it should block to a log file, and adding temps, temps is also usable, and then what components should have the debug enabled, in this case the director.
15:48
You choose the storage, what this is. You can't specify to what file, in what file the log file will be generated. However, if you issue this command, it will write you the file where it's logged to.
16:03
This is not a big deal on Linux, because it's always in this directory, but on Windows you are glad, because depending on the Windows version, it uses different log files. This has been now all about the normal commands, yet also the special dot commands, and they have
16:25
been intended for non-interactive use, but if you're creating scripts, then you might want to use them. You will not see them with the help command, but if you use the dot help command, you will also see these dot commands. For example, dot SQL, normally you should be able to get all the information you require by the
16:46
list command, but if this is not the case, you can also specify arbitrary SQL commands to execute. Of course, you could also directly connect to the database, but here you have already the
17:01
authentication to the database, and you don't know practically what type of database you are using. So, now it's getting interesting. The normal output from the vcontrol is in RP mode 0, meaning RP mode disabled. So it's intended to be human readable, and yeah, it's quite okay.
17:25
I guess for that, some time ago, RP mode 1 has been introduced. Yeah, and there's a reason why we haven't used this anymore, but we introduced RP mode JSON, or RP mode 2, which outputs data in JSON format that are usable by other programs.
17:48
So, as an example, I'm going to start on top. In AP mode 0, list drop 1, you're getting this table, nicely formatted, and gets the information in there.
18:01
With AP mode 1, yeah, the declaration is stripped, the first bar is stripped, and I'm not sure what else has changed, but unfortunately this is not really consistent between the different commands. With AP mode JSON, it's the same command, you're getting a dictionary named results and insights, because we asked for jobs.
18:26
We're getting an array of jobs, this is the information that the fields are called. We already tried to mimic JSON RPC 2.0 format, because we have to use some format, and maybe then it gets easier to extend it someday to the JSON RPC server.
18:46
Yeah, how to use this? For this you could use Python barriers. Python barriers is something like the vcontrol, so it first connects to the director, but there's also a submodule for JSON, so you can handle the JSON data immediately.
19:07
You can also directly connect to the storage daemon or file daemon, but they don't have, they only provide limited commands that you can use. So, as an example, the same example as before, you first have to prepare your password, you
19:27
import the barriers-psop class, prepare your password, then use the director console JSON class, connect to this address with this user and this password, and then you call the command that we issued before.
19:45
And you immediately get returns an item structure with the data included. Also not the result text, just the real result. If there is an error, then you get an exception, so this is quite usable to create your own scripts.
20:05
If you're doing backups, you probably care about the files, and I will now show some methods to access data from files. So even before you really did run a backup job, you can call estimate listing and get in return, so then the
20:28
director will connect to the client, and this will return all the files that would get back up if this job would run. If you have really won this job, then you can list files of this job ID, you get the list of
20:44
the files and directory that are back up by this job, but without extra information, without the start information and so on. If you use the restore command, say the last backup from this system, then it will generate a virtual file tree,
21:06
giving you a new prompt, and then you can use dir or ls or cd command to walk through your backup data. If the file or directory is not really included in the backup, but only this
21:21
because it's the parent directory, then the values here are all zero, like in this case. But here you also got the real backup data from your files, you can mark the files, say done, and then these will be restored. This is fine, but if you really want to write a script to interact with your file data, then this is not enough.
21:43
For this, you have to use the bvss API. This is described in the developer guide, and it looks quite nice, as you see. You first have to update the cache for the job ID you care for, and then ask where does this backup start.
22:04
So bvss ls directories from this job ID, there's an empty path, you get root directory in return, this root directory has path ID three, and other values are set to zero. And aaa means this is all also zero, because what you see here is a serialized form of the start field normally.
22:30
This is done in this way and stored in this way to be more flexible about the database, about different lines, but it's not really handleable.
22:44
You now know that your slash directory is at path ID three, you get the directories of path ID three, then you see there's a user directory still with no information, but that's the path ID two. And then you look for the directories in path ID two, then you'll see, oh, this is really something, because here are some information, and
23:06
the user sbin directory is in path ID one, and there are no further subdirectories, but you can list the files that are in this directory. So you need bvss if you want to ask the RDS director for specific directories.
23:25
Yeah, I say that's not really easy, but this is the way that, and also the web UI restore browser did work and uses the commands, because they can't work with the list of all files, because they can be quite, quite a many.
23:41
Oh yeah, in JSON format, you can at least read the data, yeah, if you want to, but it doesn't help too much. Another way to access this is using Barrios FUSE, so a FUSE file system based on Python Barrios JSON mode, and if you mount this, this will connect to your director, and in this directory, you
24:06
mount it to BarriosFS, for example, you get directories for clients, for jobs, for pools, for volumes, and then you can list, for example, the jobs running directory, and you will see, okay, currently there's one job running, has this job ID, has the name Gonzo client Ting FD, level is full and status is running, because otherwise it wouldn't be in this directory.
24:29
You can not only ask for running jobs, but you can also ask for all the jobs for client Ting FD, and then you will see, let's call it to here, you have on the 20th January, you have a job,
24:44
with this job ID for the name Ting client Ting FD, level full, status terminated successfully, so this is good. And the day before, you have a similar job, which is incremental, I guess they both started at the same time, but you see the
25:02
full job did end quite later, two hours later, and so on, and here on the 70th January, you get a job that did fail. You not only see the job, you can also go to the directory, and then get access to job block, for example, or the info, which is the information you would get from this job ID of this job, and the data subdirectory.
25:30
You can descend to the data directory, and then you will see all your metadata, all the files that you have backup in your backup job. Of course, you can't access them there, because you haven't done a restore, you just look at the metadata that is stored
25:45
at the directory in the databank, but you get the information, you get the metadata, the file, change date, and so on, and you can use your normal UNIX tools like Find, for example, to find all the files that are larger than 100 megabytes. Will it be incremental?
26:13
Incremental will only show only the specific job. You can handle this otherwise, this is just really
26:20
shallow Python code with list jobs and list files and so on. If you want to have to have other features, just it will be a few lines of code, three or four, something like this. You not only can see the jobs of specific clients, you can also get a list of your pools and your volumes in your pools, then you can immediately see we have this always incremental full volume that is in
26:48
status full, so we have written this amount of bytes to this volume, and it's in read-only mode because it's full. And here on this other volume you see it's in append mode, so you can still write data to
27:03
it, and therefore it's in read-write. This last time data is written to it has been returned this January. Again, you can change to the directory of a volume, you get some extra information, and also jobs, so you can descend to a volume, to a specific volume, and then the jobs, and you will see all the jobs that are back up on this volume.
27:22
And in this job you can change to the directories in there and get the files that are back up with this job. So quite useful, and you can write your normal Unix tools to script to retrieve some information. It's not the most efficient way to get the data, but it's quite comfortable.
27:45
Okay, now we want to care about specific problems and writing script for this. For example, you have your barriers backup environment, and you back up all your servers, and everything works fine, but your servers are always there, so you can have a fixed scheduling for them and back up the data.
28:07
Now I think you have your co-workers with their laptops, and they are sometimes in the office, sometimes not, and maybe you have dynamic APs, and so it's complicated to get time when you can back up them.
28:21
So how to solve this, with this short script I can say, and it looks like it gets a list of all, no, I have to start otherwise, we use the client in the initiate connection, so in this case not the director connects to the client, but the client is configured to connect to the director.
28:43
The director is then aware of all the clients that did the connection to it, and have the connection, and if there's a job for the client, it will reuse this connection. You get the information with status director, and look at the field, connected
29:03
client, client connection, and you only care about the name of the clients. Then you loop through all the connected clients, you have the trigger function, you can call them by a cron for example, every hour or every minute as you like, you get all the connected clients, you loop through the client, you get a job name associated with the client, so if your client
29:26
is called client one, you look for a job backup of client one, and if this job exists, you check if is there a job for this job, the client in the last 24 hours, so if there's no job for this client, then you just execute one, the job name that you
29:47
have there, yes, to confirm it, and in result you get a job ID from the newly triggered job, and you can just print it out. So with this short script, you can solve the problem with your mobile clients that are sometimes at the office, sometimes not, just
30:02
by running this periodically, checking the clients that are connected, and starting a job if no job exists for the last 24 hours. So I have how much time left for 15 minutes?
30:21
Oh, 20 minutes. Okay, then I go, no I don't go through this because it's not all on the slide. If you have a question about pruning volumes and how to automate this in a specific case, ask me after this talk. Oh, is this a question? Yes.
30:47
No, no, this is for the time that the clients are back in your office, connect to the director, which is also located in the office, and then you know that your clients are there, and then you trigger the backup.
31:04
Okay, now the other way around, things getting triggered by barriers, for this you get one script, you have got quite flexible one script, either you can call this job a command on the console itself, so use the barriers,
31:21
direct barriers commands, you have options like they should only run if the job has succeeded, or only if the job has failed, and you can specify when they should run before a job, after the job, I guess, on the job. And one of failure only works with after the job, or on Windows after a VSS, something like this.
31:40
You can also execute arbitrary system commands with the commands of one script, and then you have the additional options, if this command should be executed on the director, or on that line where the backup runs, and if the job should fail, if the system command fails.
32:00
Some examples, for example, if you want to create a virtual full backup from your existing backups and store it somewhere else to an archive pool, but don't want that this newly generated virtual pool is again used as a base for the next virtual pool. Then what you can do is create your virtual pool, store it in the archive pool, but after you have done so,
32:27
update the job ID, because you know the current job ID while it is variable, and change the type from backup to archive. Archive drops are similar to backup drops, but they are not used internally for any other actions.
32:40
So, you can just run this, this is a control command, you can also type it in the bcontrol, but you can integrate it with your job, and it should run on the server, and it should not run if this job before has failed. No, not the same. No.
33:06
What other jobs are available are admin drops, normally you get backup drops or restore drops, and with admin drops you don't do any actual backup, you just use this to run some commands, and in this case,
33:23
after the backup of your own system, you optimize the database, which can be an advantage if you are using Postgres. So, onto backups, but just use this as a kind of con, but to get my other things in here.
33:42
RelaxantRecover, yeah, maybe some of you have heard, have used, are using this, or have heard the talks before. RelaxantRecover is a disaster recovery environment for Linux, which works quite well. It works in two steps.
34:00
Once it creates a rescue image from your current running system, it creates a rescue image, and you can boot with this rescue image and restore the system. Second step, we itself could do backup, but it can also integrate other backup solutions, and thankfully, Barrios is also part of this. A minimized way and quite comfortable way to integrate it with Barrios is, normally you get the question, okay,
34:29
when my system broke, I need my rescue system, so I should be able to have a recent rescue system. So, when to create it and where to store it, because you normally don't want to store it on your local system.
34:44
We have options to store it on NFS and things like this, but we are a backup solution, so this approach just creates this regularly on every full backup and stores them locally on our own file system. Because our full file system is back up to the backup server, and if you really require it, you can extract
35:04
it from our normal backup and then restore it to another machine where a CD burner or a USB drive is attached. We can put it on there, and then we have the rescue image, we can place it in the same server with the replace TARDIS and restore our system.
35:24
We do so with a run-before script, so before we actually do the backup, we run on the client this command, this is a shell script, this has one parameter, this is a backup level, so full, differential or incremental.
35:41
We only execute this on full, because if you would do this in incremental, then every incremental has an additional size of 50 or 60 megabytes, which you don't want, but on your full backup with whatever few gigabytes you use, 50 megabytes more or less are not really important.
36:03
It's important to create this rescue image regularly, because it integrates your current kernel, your current network configuration, your current hard disk layout, so if you don't do this regularly, you may run into problems, because if your
36:20
rescue system is one year old, you can only get back to the status from your one-year-old system. So again, something is missing on the slides, what I would show you if the screen has been larger, you then have the bconsole, you make, you type in restore of, let me think, control minus, it's also an option, hey, thanks.
36:54
Okay, that's what you have to type, restore from the current backup of client, you care about, client FD1, extracts the file
37:08
var librear-output-barriers.iso, this is the default location, and restore it to some other system which has the debonor attached.
37:21
And use the special restore drop, just, we don't make the restore and the var librear-output location, but in the temp directory or some other directory you want to use. So with this, you have your normal backup drop, and yeah, and we are directly integrated with this.
37:47
It requires you, right? But it also offers different methods, like, maybe also copying, some are for sure, and a lot of other possibilities.
38:03
I think, yeah, so. So that will just boot up the ISO, and then when you have the recovery system up, then you restore the file from various, from various. Yes, so the rear barriers integration is that rear, you can configure your rear, that should use barriers, and then rear itself, in the
38:25
rescue image, creates, uses the kernel, your network setup, your disk layout, but also copies all the back barriers tools you have installed on the system, on your rescue system, so it boots up and has the same file daemon available that you have on your running system.
38:41
And with this, it restores just your data from your normal backup. And that will automatically be already into the recovery, so that you create? Oh, okay. You just, this is one line, backup barrios, something like this, and this is all. That would be in the boot common line? No, this is in the configuration via local.conf, but it's just one line in the configuration, so it's really easy to handle and really powerful.
39:10
Yes, for sure. Okay, then this is about scripts. Now we say two slides about plugins. Barrios can be extended by plugins for the director, storage daemon, or file daemon.
39:27
With this, you give that to specific events you care about. On the file daemon plugins, normally you use them to back up specific data, like a database, virtual machines, or something like this. plugins on the storage daemon are normally made for rewrite data, like on-the-fly compressed data that's coming in.
39:50
Normally this is done on the client, but if your client is not that powerful, but your storage server is, then you can also make the compression there. Or scasi-crypto to use crypto tapes like this to read you some status information.
40:07
Or for the director, this is normally, if you write a plugin there, then you do this for status information. And the shortest possible plugin is something like this.
40:21
You have to import the Barrios class for the director, and then you have to overwrite the init function. Then you register for the events you care about. In this case, we only care for the event we dropped at end. Then you have to overwrite the handle plug-in event function.
40:40
When this is called, it's called with a specific event. We check for this event. In this case, it should always be event dropped end, because we only register for this, but we check nevertheless. Then we retrieve some other information, like what lines to drop the one, have there been errors, how many bytes, how many files, how many bytes have been transferred.
41:01
We format that in some other way, and then we send it to a singer or a graphite or whatever you use for monitoring or for visualizing your backup system or your environment. Okay, so I'm mostly done.
41:21
Just roadmap for the upcoming release for the things that's important for integration is the Python Barrios is currently available. It's a Barrios concept repository, but will be moved to Barrios core repository in the next major release. The Barrios web UI is actively developed by Frank over there, and it will continue to be enhanced.
41:48
Therefore, we also will extend the director to provide more information. The part about media handling and optimizing this for this storage I skipped, so it's not so important.
42:03
What I missed here at the roadmap, we have seen here the console connection where you specify a user and a password. We have received a pull request for adding PAM support, which has quite some implication, but we want to put work on this and get this integrated.
42:24
We plan to integrate this also for 17.2 so you can directly authenticate your ADAP directly or whatever you like. Okay, now we got time for questions. No questions? I'm done. Good.
42:44
Okay, one question. For the Python Barrios module, which opens a GSP connection to the director, this also works with Python 3 and uses Python 3.
43:09
For the plugin, yes, as I heard before, it's only Python 2 right now, but it's already prepared for Python 3 and will also be done soon, I guess.
43:22
Yeah, I guess, per distribution, because normally we compile it against what's available on the distribution, and if the distribution only offers Python 3, we will do it for Python 3.
43:43
Yeah, yeah, yeah. Okay, other question? Yes?
44:14
In principle, it connects to the director, and then you can call the call function and put in all the commands.
44:24
You could also enter the B console, and so this will be executed and the results will be given to you. Okay. How to implement a second backup server, like a failover, in the scenery down the backup server?
44:47
Yes. When it fails, you want another one, like a failover or something? Failover. If you want to have a failover, the only thing you need, you have redundant database that you do with your database tools, and then you can have a second director, but this is only important, the configuration director,
45:06
etc. Barrios, Barrios director, and you can copy them over or keep them synchronized, and then you can start the other director. Well, we tried it in one project, or we did it in one project.
45:24
Okay, but now, not simultaneously, so it's just a failover.
45:52
Okay.
46:04
Both the director and the file daemon, like the file daemon, it is possible that you need to expose those to the internet, you know, they need to be exposed, because maybe you don't have everything in your lab. You're doing remote backups, you know, so is there a security concern for you?
46:23
Of course, it's a security concern if you provide services to the internet, but yes, we are aware that people are doing so. You, of course, should have your TLS encryption configured to do so, and probably also, what would be a good choice is also to encrypt the data on the client. This is also a Barrios option, so the support is encrypted, and also through the
46:45
encrypted transport, your data is also have been encrypted before, so this should be quite reasonable. So all the communication is encrypted, and the data is over the internet? Exactly, the data is encrypted on that line, and can only be restored on that line, or if you've got the master key, also configured somewhere else.
47:07
Okay, and the second question is... The very bare minimum requirements on the client, you know, to keep it lightweight, is having the file daemon storage. The file daemon, only the file daemon.
47:21
If you want to keep it as small as possible, that's the only requirement you have on the client. How resource angry is that daemon storage on the client? Do you need a powerful machine? Do you think that the director needs to be 4 gigabytes, I think, was the best practices?
47:41
I'm not sure about this. I know that some people running the director and also the storage team are on a Raspberry Pi, so I'm sure that's not so... Of course, it's not the fastest solution, but a lot of people are using it this way, and also the core components are all written in C, C++, also they are running on AEX from 10 to 15 years old, so it's not so resource-intensive.
48:09
Even with big storage data sets, it doesn't get very, very heavy if the data set is growing? So if you have to detect the files that have changed since the last backup, then this can
48:22
consume some time, and also encrypting the data on the client will also consume time, but this is normal. So if you cannot afford this from your CPU time, you can configure it to make this lightweight and maybe unencrypted, but yeah, maybe you don't want to. Okay, this is it? Okay. Oh, sorry, yes?
48:45
Are there plans to speed, for example, testing BetterFS after they are having their own end-of-file table and want to check the files? And when I run the various backup and use Shredder, that file takes a long time to check it. Are there plans to make improvements regarding the files?
49:04
There are plans, so the question has been to use some file system mechanism to detect changed files more quickly. And this is for ZFS or BetterFS, yes, and we discussed this and we have also a task
49:22
defined for it, but currently nobody is working on this, but yeah, it could be hopefully done quite easily. But I checked this with BetterFS, and this wasn't as easy because if you want to retrieve the file information that has changed, you have these virtual mount points, and then you, yeah, it wasn't as easy.
49:43
It's easier to get changed blocks, but we're not backing up blocks, we're backing up files, so also this is not too efficient. But yeah, with a file daemon plugin, this can be implemented quite easily.
Recommendations
Series of 14 media