SELinux fun with MySQL and friends
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46896 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020159 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
Greatest elementComputer animation
00:22
Software developerRWE DeaDatabaseComputerInformation technology consultingNeuroinformatikSoftware developerDatabaseSystem administratorPhysical systemElectronic mailing listComputer animation
00:44
RWE DeaSystem programmingInformation technology consultingPhysical systemGroup actionComputer animation
00:58
Physical systemPresentation of a groupComputer animation
01:20
Kernel (computing)Module (mathematics)Information securityZugriffskontrolleMechanism designSpacetimeDistribution (mathematics)Non-standard analysisModul <Datentyp>Asynchronous Transfer ModeEnterprise architectureDefault (computer science)Asynchronous Transfer ModeDefault (computer science)Metropolitan area networkKernel (computing)Set (mathematics)Presentation of a groupModule (mathematics)Multiplication signNormal (geometry)Information securityGroup actionLink (knot theory)InternetworkingComputer animation
02:32
Information securityDefault (computer science)Installation artPhysical systemContext awarenessProcess (computing)Information securityDefault (computer science)Physical systemLevel (video gaming)MereologyAsynchronous Transfer ModeMappingType theoryContext awarenessConfiguration spaceDirectory serviceEnterprise architectureBitProcess (computing)CASE <Informatik>Object (grammar)Regular graphTrailComputer configurationComplex (psychology)Different (Kate Ryan album)Core dumpField (computer science)SequelEntropie <Informationstheorie>Rule of inferenceForcing (mathematics)TouchscreenComputer animation
05:35
Inclusion mapDirectory serviceContext awarenessServer (computing)Instance (computer science)Sheaf (mathematics)Error messageModule (mathematics)Polygon meshServer (computing)CASE <Informatik>Default (computer science)Process (computing)Context awarenessDirectory serviceType theoryUniform resource locatorMereologyNumberKeyboard shortcutRight angleDatabaseService (economics)Different (Kate Ryan album)Instance (computer science)Sheaf (mathematics)CuboidFunction (mathematics)Boolean algebraError messageInstallation artObject (grammar)Electronic mailing listMathematicsRevision controlSequelForcing (mathematics)ACIDPhysical lawExtension (kinesiology)WebsiteSet (mathematics)Matching (graph theory)Goodness of fitEqualiser (mathematics)Computer animation
10:33
Operations researchProcess (computing)Computer networkHacker (term)IterationBlogError messageProxy serverSocket-SchnittstelleDatabaseDemonDefault (computer science)Point (geometry)Proxy serverState of matterModule (mathematics)IterationDirectory serviceComputer fileOrder (biology)Projective planeService (economics)MultiplicationCASE <Informatik>TouchscreenSystem administratorGroup actionPhysical systemLoginEmailRotationKernel (computing)Process (computing)Function (mathematics)Electronic mailing listEntire functionOperator (mathematics)Revision controlDifferent (Kate Ryan album)Configuration spaceNumberError messageType theoryCuboidBit rateRoutingDesign by contractIdentity managementCodecExpert systemElectric generatorProduct (business)Scaling (geometry)Set (mathematics)Computer animation
16:22
Default (computer science)Enterprise architectureSanitary sewerComputing platformAsynchronous Transfer ModeServer (computing)Physical systemExterior algebraServer (computing)Different (Kate Ryan album)Default (computer science)Remote procedure callKey (cryptography)Computer animation
17:03
Forcing (mathematics)RoutingWhiteboardDefault (computer science)OracleComputer configurationGoodness of fitDatabaseKernel (computing)RootkitComputer animation
18:44
Point cloudSource codeOpen sourceFacebook
Transcript: English(auto-generated)
00:05
All right, let's get started. Welcome to SL Linux Fun with MySQL and Friends. As you can see, I'm missing a co-speaker. Ivan unfortunately couldn't make it today. He's in South America, and he couldn't get approval to come here. But he did help me prepare this slide, so we left his name on there.
00:22
So, quickly, who we are. My name is Matthias Grauwels. I live here in Belgium. I live in Ghent. I've been a bachelor in computer science, been a Linux user and admin for over 20 years. Then I've been a PHP developer and started my career for about 10 years. And since eight-ish years, I'm in MySQL DBA.
00:41
Currently, I'm a lead database consultant at Pittian. Ivan lives in Argentina in South America. He's a systems engineer. Unfortunately, he also has left Pittian. He's now a senior consultant at Percona. This is something about Pittian. One disclaimer. I'm not going to claim here that I'm the SL Linux guru.
01:04
I'm just a DBA having to deal with SL Linux on systems that are managed by other people. And the presentation goes about how we go about that and what we did and what we learned about dealing with those kinds of situations. So, let's get started with some introduction to what SL Linux is.
01:24
So this is the definition you can find on Wikipedia. It's something because someone thought that the normal Linux security system wasn't good enough. So on Linux you have the privileges for user groups and others. But it wasn't granular enough.
01:42
It's originally developed by the NSA in America and Red Hat. And it's distributed to your kernel as a set of kernel modules for enhanced security or for bothering DBAs. There are three modes to SL Linux.
02:01
And it's very similar to what Nick earlier said in his presentation about ProxySQL. The default mode is set to enforcing in Red Hat Enterprise Linux and CentOS. You can set it to permissive or disabled. Permissive does what it says. It will track all the settings for SL Linux, but it will allow them, but it will log them in the log file.
02:23
And disabled will just completely disable SL Linux. And a wise man once said, every time you disable SL Linux a kitten dies. Think of the kittens. The truth is, mostly the compliance or security teams will bite you if you disable it.
02:46
It's often a part of their audit trails. They want the logging at least, or they want to block everything they don't allow. Which is also the default policy. So there's a deny policy for anything you don't specifically allow.
03:01
A useful tool on Red Hat Enterprise Linux-based system or CentOS-based system is PolicyCoreUtils in Python. It provides a lot of tools to manage and define your SL Linux policies. If you want to go a little bit deeper and really start writing policies, the devhold package gives you a lot more tools also.
03:23
And how you can check it? So there's a tool, sa-status, and you can run that and it will show you the current modes. Like in this case it was enforcing, it's enabled, and these are the config directories.
03:40
You can get enforce, you can do get enforce to see what the status is of the enforcement. And remember the kittens here, because I made this screenshot, I killed the kitten. So sa-linux is defined by users, policy, context.
04:00
There's another one-on-one mapping between Linux users and SL Linux users. So the SL Linux users by default here in this system, you can get by sa-user-l and then you get all the SL Linux users, you have get, root, staff, and regular user. And you can then assign a login, so in this case I assigned user John to my SL Linux user.
04:27
And then you can see that this John user is there, and the default is that they are unconfined. So it's very straightforward, very easy to track or something. And also sa-linux adds a dash capital Z option to commands like ls or ps,
04:45
so you can see what SL Linux users and object roles and types it uses. And also for a process you can see mysqld is running as the type mysqld underscore t.
05:02
And mysqld-safe is running as the type mysqld-safe underscore t. And so the contexts are defined as user, role, type, and then a level. The most use that I have seen from it is we're using user and type, and role is usually like a system object or system R or something like that.
05:22
And level is if you want to go even more granular, you can start doing like this user has access level to these SQL Linux levels and you can go up to more and more levels. But that takes us a bit too far. So, mysqld and SQL Linux.
05:41
Out of the box experience is so that everything works. So if you just install, yum install mysqld server, which gives you MariaDB server, or if you do mysqld community server, everything will work. So there's a module, a policy predefined. You can do sa manage module L and then you can grab for mysql.
06:01
So you see that the mysql object is there by default. And it's very granular. So you have a lot of different types that are defined for different locations. So in the etc mysql, it's mysqld etc. Then you have the log directory, the data directory has its own,
06:21
the mysqld process has its own context and everything. So it's very granularly defined and you can then start playing with it. So in this case, this example, we wanted to change the data directory. So the default is far-lived mysql and somehow we were requested to put it on data mysql.
06:43
So we made the directory, we gave it the permissions. And you can see here by default it's unconfined. And we started MariaDB service and it didn't start. So what we used to do was turn sllinux off and then it worked. But remember the kittens?
07:02
So there's an audit log. And if you install that python package that I referred to, you can do audit to allow. And dash w and dash a will give you the list of things it denied for you. So in this case it says I denied write to the location, I tried to write with mysqld context
07:28
and you are writing to default d so it's not allowed. And it's true because this is by default default type and so you cannot write to that with sllinux enforced.
07:42
So we were then finding the right data directory context. So on varlib mysql this is the default context. So we were trying to just set the context to the data directory context. And then we did an ls again and hey, wait, we changed this.
08:00
But it didn't change anything. So you still have to apply that if you change that. Something we learned. If you do then restore con and dash v gives you useful output so it shows you what it's actually changing. Then it will start changing the types and now you can see that it has the right type and if you now start the server or get the status, you can see that it's active running
08:22
with the right data directory as you were expecting to. So this is one way of dealing with it. Another thing is custom ports. So we had a running mysql instance and we just wanted to change it from 3.2.06 to 3.2.07. So what we did, added ports equals 3.2.07 to the mysqld section of the config,
08:45
restarted and boom, it didn't start. Why didn't it start? So you check journalctl, there's... It says it doesn't start, it has a failure but it doesn't say why. The error log says permission denied. Do you have another process running on that port?
09:01
No, we don't. So obviously SA Linux was bothering us there. So again, if you do the audit log, you can see that it is trying to use an unreserved port and it gives you a possible solution. So it says you allow nis-enabled and nis-enabled will allow a process to bind to any port.
09:24
And so if you set that SA Linux boolean, it starts and it binds to the port. It works. But the compliance team may or may not like that level of freedom to give it to you. So another way you can go about it is SA Linux manages the port.
09:42
So if you grab the mysql port from port-l, in sa-manage you see that it has a number of ports predefined, 3.2.06 is part of it. And you can just add 3.2.07 to that list by doing sa-manage port-a for add to the type mysql-t port-t.
10:02
And then you give it TCP 3.2.07 and then it adds it to that list. And then you can just restart the database server and now it starts with port 3.2.07. So that is just examples of things we run into while dealing with it and how we try to work around it without getting the compliance teams on our necks
10:24
or getting us in trouble for disabling things or changing booleans and all those kind of things. So another friend of mysql is proxy SQL. Nick just presented about 2.0.
10:41
And we also use proxy SQL quite often. And also, by default, it works. So if you have as a Linux status enabled and forcing everything, it works, it runs, but it's running as an unconfined service. And it's running with the default privileges for varlibt,
11:03
where the default type of directory for proxy SQL is. So, why should we bother? It works out of the box. In our case, the log rotation failed. So we have log rotate rotating the proxy SQL logs. And if I run this as root, it works.
11:21
I defined my proxy SQL log rotate config and I tested it and everything worked. Until cron started running it. cron runs and it says it starts, I start log rotate proxy SQL and it exited abnormally with status 1.
11:44
So, why? If we do this default system has its mailing to varlog mail root, and so it says error renaming my .2 log file to .3. Why? It says permission denied.
12:02
If we try it as root, it works. But if we try it in cron type, it doesn't. So it has root, we can just run the log rotate, we can get all the output, it does everything, everything works. But cron doesn't let us. So if we go back to this audit log,
12:21
we see it tries to write this file, but it tries to rename this file, so it's not allowed. So in this case, you can use also the audit allow tool to generate a policy for that. So if you do audit allow on the log file,
12:42
so here is the same command we did for that audit log, this just prints the output on the screen. If you do dash m proxy SQL into a file, it generates this file for you. The problem is, in this case, it only generated for the one error it saw.
13:00
So it saw one error, the rename, but there's multiple things that log-rotate will do. It will create new files, it will rename files, it will delete files if they are expired. So it needs more privileges. So these are the commands you can do then to compile this .de file into an actual module file
13:21
that you can load and package before it gets loaded into the kernel. So at the end, you have the module loaded version 1.0, which is the version we defined here. So if we define a different version there, it gets a different version number down here. As I said, this only just allows for rename. So more iterations of the same process are required to make this work.
13:46
An easier way to get to this is to set SLAs temporarily to permissive, so you can see what it will do. It will allow, it will log everything, and then you can run the process. Obviously, it doesn't do all the operations it might do in a single run.
14:03
So in my case, I was only at my second or third iteration, and I want to keep five log files. So when it reaches the fifth log file, it will start deleting. So you have to run, make sure that you have the entire process that it will do.
14:24
No. Oh, yeah, the question was, can you set SLAs permissive only for a specific process? The answer is no. Not as far as I know, but... So if you then have the entire list of things it can do in the audit log,
14:44
then you can generate the full file that you end up with, and then it looks something like this. So it has access to those types, so the varlibt, log-rotate, unreserved-ports, TCP connect, and then all those file actions.
15:01
So it needs access to the admin port for proxy SQL in order to flush the logs, because otherwise proxy SQL will just keep writing to the old log file. And it needs all those privileges on the varlib proxy SQL directory to make sure that it can rotate the file.
15:21
In the end, it was a great success. We'd have no more errors. Our log-rotate in cron was running fine. We started rotating and everything started working. Thank you, Dave. Is this the best solution? Probably not. What would be a better solution, I think, would start to define proxy SQL's own SL Linux policy.
15:41
So the varlib proxy SQL directory gets its own policy, gets the right privileges for log-rotate. But as I said, I'm not the big SL Linux expert, so I didn't get that far yet. Maybe in some next project I will have to, and maybe next year I can come demonstrate how you generate the SL Linux policy for proxy SQL.
16:06
But in this case, this was enough for us. So we got log-rotate to rotate, and in this case now it has permissions to do so on all the files in varlib, which is probably not what you want.
16:20
But for this project, it was okay. Alternatives for SL Linux, most well-known, most probably AppArmor, which is the default in Ubuntu-based systems, I think, and SUSE Linux. And then the key difference is, you can see here,
16:44
SL Linux supports a remote policy server. If you define it centrally, then all the servers get it. Which, AppArmor doesn't. And then there's some other tools which are less known, but one looking up alternatives, I found them,
17:01
so I'll put them here. So, any questions? You said that SL Linux has some good defaults for MySQL. Does it include... Not yet, I think. Not yet, I think.
17:21
So, does the default policy include a port for MySQL X? I don't think it does yet. It might, and there might be a feature request for that, but I don't know. And I think the policy is not managed by Oracle itself, or by MariaDB, but I think it's managed by the kernel.
17:41
Yeah, it's got to be a surprise if you... Yeah, sure. Shlomi? So the question was, when you specified the port to access,
18:04
is there an option to include SSL? I don't know.
18:21
So the question was, can you export a database after you set SL Linux? I think you can, you just export it to anywhere you want. I think as root you will be able to write anywhere. Okay, thank you.