Scale Your Auditing Events
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 561 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44532 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2019346 / 561
1
9
10
15
18
19
23
24
27
29
31
33
34
35
38
39
40
43
47
49
52
53
54
55
58
59
60
63
65
67
69
70
78
80
82
87
93
95
97
102
103
104
107
110
111
114
116
118
120
122
123
126
127
131
133
136
137
139
141
142
148
153
155
157
159
163
164
168
169
170
171
172
173
174
181
183
185
187
188
193
196
197
198
199
200
201
205
207
208
209
211
213
214
218
221
223
224
226
230
232
234
235
236
244
248
250
251
252
253
255
256
257
262
263
264
268
269
271
274
275
276
278
280
281
283
284
288
289
290
293
294
296
297
300
301
304
309
311
312
313
314
315
317
318
321
322
327
332
333
334
335
336
337
338
339
340
343
345
346
352
353
355
356
357
359
360
362
369
370
373
374
375
376
377
378
383
384
387
388
389
390
391
393
394
395
396
406
408
409
412
413
414
415
419
420
425
426
431
432
433
434
435
436
438
439
440
441
445
446
447
448
453
455
457
459
466
467
471
473
474
475
476
479
480
484
485
486
489
491
492
496
499
500
502
505
507
508
512
515
517
518
529
531
533
534
535
536
539
540
546
550
551
552
553
554
555
557
558
559
560
561
00:00
Scale (map)Event horizonScaling (geometry)Slide ruleEvent horizonComputer animation
00:38
Configuration spaceNumberGroup actionAuthenticationEvent horizonProcess (computing)Intrusion detection systemComputer fileDependent and independent variablesData integrityKey (cryptography)RootVideo gameSlide ruleMultiplication signEvent horizonPhysical systemIntrusion detection systemIdentifiabilityInformation securityType theoryPoisson-KlammerLine (geometry)Different (Kate Ryan album)Cloud computingMotion capturePoint (geometry)MetadataState of matterRule of inferenceTraffic reportingSpacetimeConfiguration spaceBitRaw image formatTimestampComputer fileProcess (computing)Elasticity (physics)Group actionLogin2 (number)Kernel (computing)Beat (acoustics)AuthenticationSource codeComputer animation
03:24
Group actionCountingConvex hullEvent horizonBeat (acoustics)Configuration spaceInformation securityComputer filePattern languageMultiplication signIP addressModule (mathematics)Group actionOperator (mathematics)Sinc functionDisk read-and-write headParsingBitDefault (computer science)Operating systemLatent heatSequelComputer animation
05:15
Limit (category theory)Message passingEvent horizonComputer fileConfiguration spaceFile formatAsynchronous Transfer ModeParameter (computer programming)Rule of inferenceLimit (category theory)Module (mathematics)Group actionAttribute grammarMiniDiscBit rateSoftware developerDisk read-and-write headBeat (acoustics)Hecke operatorFerry CorstenComputer animation
06:55
Multiplication signConnected spacePoint cloudDisk read-and-write headVulnerability (computing)Information securityBitEvent horizonLatent heatInformationInstance (computer science)MetadataPhysical systemFigurate numberRevision controlProcess (computing)Computer fileOpen setMedical imagingFocus (optics)NamespaceCloud computingMereologyComputer animation
08:23
LoginRootMenu (computing)Elasticity (physics)PasswordDemonCuboidMoment (mathematics)Information securityPasswordTrailComputer animation
08:54
CountingPhysical systemFrame problemPasswordMultiplication signGeometryIP addressLoginCoefficient of determinationForcing (mathematics)Instance (computer science)System administratorAuthorizationComputer animation
10:45
PasswordElasticity (physics)Information securityPhysical systemLoginPatch (Unix)Canonical ensembleKernel (computing)Reduction of orderComputer-assisted translationEvent horizonComputer fileRule of inferenceComputer animation
11:12
Source codeBitEvent horizonRule of inference1 (number)Online helpComputer animation
12:01
LoginElasticity (physics)Computer fileFile formatModule (mathematics)FlagDirectory serviceDigital filterBitOperator (mathematics)Reading (process)Software developerProcess (computing)Rule of inferenceComputer filePhysical systemTrailSensitivity analysisInformationBit rateIntrusion detection systemComputer animation
13:00
Computer fileFile formatModule (mathematics)FlagDirectory serviceAuthenticationIdentity managementPasswordCountingPasswordService (economics)System administratorElasticity (physics)Group actionElectronic mailing listPhysical systemProcess (computing)Beat (acoustics)Computer animation
13:58
AuthenticationIdentity managementPasswordLoginPatch (Unix)Canonical ensembleInformation securityPhysical systemKernel (computing)Reduction of orderElasticity (physics)Convex hullEvent horizonSystem administratorElasticity (physics)Computer fileRule of inferenceType theoryRootDirectory serviceComputer animation
15:18
Digital filterSource codeMenu (computing)System callRule of inferencePower (physics)Event horizonProof theory
16:25
PasswordElasticity (physics)TelnetEscape characterComputer virusPhysical systemProcess (computing)Network socketPlasma displayModul <Datentyp>Module (mathematics)Socket-SchnittstelleConfiguration spacePhysical systemFile formatOptical disc driveTelnetInstance (computer science)Process (computing)BitForm (programming)Incidence algebraComputer animation
17:47
Source codePoint cloudDreizehnSource codeProcess (computing)Event horizonGreatest elementNetwork socketInstance (computer science)Control flowComputer file
18:57
Computer virusElasticity (physics)PasswordMenu (computing)File systemSubject indexingComputer fileContent (media)Multiplication signWebsiteWeb 2.0INTEGRALHash functionGroup actionLine (geometry)Web pageMathematicsDirectory serviceComputer animation
20:02
Web pageInheritance (object-oriented programming)Computer animation
20:24
ExplosionTotal S.A.Event horizonMusical ensembleINTEGRALComputer fileEvent horizonMathematicsSlide rulePattern languageModule (mathematics)File systemRootComputer animation
21:35
Group actionEvent horizonLoginRule of inferenceMultiplication signPhysical systemPasswordRootInformation overloadConfiguration spaceInheritance (object-oriented programming)Process (computing)Virtual machineAutomationAtomic numberTime seriesPresentation of a groupMachine learningPattern languageSlide ruleCloud computingBeat (acoustics)Term (mathematics)Instance (computer science)OutlierInclusion mapStack (abstract data type)Extension (kinesiology)
25:02
Physical systemComputer animation
Transcript: English(auto-generated)
00:14
OK, I think. Hello, everyone. Let's welcome our next speaker, Philip Grant, with the talk,
00:24
Scale Your Auditing Events. Thank you. Hi, everyone. I do have a couple of slides. But since we only have 25 minutes, these are the slides. You've seen them, they're online.
00:42
We'll forget the slides, we'll do stuff live. Otherwise, we're just wasting too much time. So who is using auditd? OK, that's very few. So just to give you a quick idea of what auditd is, it's something that lives in the user space, can capture kernel events. Basically, you can have configuration rules
01:01
of things that come from user space and that you want to capture. It could be like system processes, it can be user actions, it can be file events. Those you can capture and then work with auditd. And you have something that looks like this. You have our report. It just gives you an overview of what has been happening
01:22
or what is the state of your system. So you can see this has been running just very shortly on my system, just a few seconds, basically. But you can still see logins, failed logins, authentications, users. So this just gives you an overview of what is happening in your system. But obviously, this has raw events in the background as well.
01:42
So that would be all search. And then if you add dash raw, it would just give you a lot of raw events. And these are basically the raw events that are happening in your system. Every line starts with a type. You have lots of different types. So for example, a user started something,
02:00
then you have in a bracket that is a Unix timestamp, and after the colon, there is a unique identifier. If one event is causing multiple things that you want to capture, you can have duplicate timestamps and IDs. And then you have a lot of meta information depending on what is happening in your system. So for example here,
02:21
you can see something was ended in success, and I opened a session with a specific user and a process ID, and we can capture all of these. And that is audit D, basically. The one thing that you can already see here a bit is that this line looks very different than this. And while this is nice to capture a lot of events,
02:42
it is a pain in the ass to capture that in a central fashion. You can try to parse it, but it's kind of semi-nice. So just to give you a quick idea, also, why am I talking about that? I work for Elastic, the company behind Elasticsearch, Logstash, Kibana, Beats,
03:01
and we have a cloud service as well. And we kind of had this need at some point that we wanted to audit security events for our own cloud service. And then we looked at, how can we capture security events in general? And the first thing we did is we wanted to capture this file and then basically parse it. So that's something, no, not slack.
03:24
If you've never seen Kibana, this is Kibana. We'll head over to a pre-built dashboard. We have so-called Filebeat modules. They're basically configured, and they know, I want to, let's head over to the Filebeat modules. You can see here that we have various Filebeat modules
03:42
that can collect specific log files. The one I'm interested in here is we have one for auditing events. And basically, we know on my operating system, Ubuntu 18.04, that log file is under that specific path. And by default, the pattern of that file has a specific layout.
04:01
And then we can just parse that and collect all the data. We can do that for NGINX, Apache, MySQL, Postgres, Kafka, lots of others. And we can do that for auditing events. So what we get out of these auditing events, and since I've only been running that yesterday for a short while, you can see here which users were doing which kinds of actions.
04:20
We could filter down on those. You can see we don't have any top-executed commands, but you can see here we were running this and we had a lot of events in a short amount of time. You can also see, this was probably me doing something yesterday. And I had a Belgian IP address probably. We could zoom into that to see where I was actually coming from.
04:40
And then you can see the actual commands and what we have captured on those. However, parsing all of this was still a pain and we didn't really want to do that. So we were thinking like, how can we make this easier? And basically, what we then did is we took what Auditee is doing and we packaged that in another beat. And we called that the Auditee beat.
05:01
So it's just a beat to collect security events. And this is using the same configuration like maybe you could say arcane configuration for Auditee. Just to give you a quick idea of what we have here. So like every other beat, it's living in ECT, whatever beat name you have.
05:21
So we have Auditee beat here. We have Auditee beat YAML. So we have the YAML file here. You see we have the Auditee module that we are running here. These are some configuration parameters that you might know that how failure modes should be handled, how many backlog items do you want to capture, if you want to rate limit this,
05:42
if this is capturing or generating too many events. And then here, I've written myself some notes because I always forget how these filter rules actually work. They define like how this is working. And for example here, I'm saying like watch, we're watching a file, ETC pass WD
06:02
with the actions read, write, execute or change any attribute. And we tag this event with pass WD access. This is one rule for example. Or here, we have a slightly more complicated definition where I say always on exit, follow the file etcpen.conf when somebody reads the file
06:24
but only this uses in the group 1001 and tag that, K is always tagged that with developers PAM read that specific file. So these are just some configuration files. And basically what we've done is we've kept the same configuration format. We've just wrapped it into audit bead
06:41
and audit bead rather than writing it out to disk and then you try to parse it back, sends it as a structured format directly into Elasticsearch. And then you can just visualize that and see what is going on. So to show you, let's head over to this one here. We have some pre-built dashboards for those as well.
07:01
Let's head to the audit D overview. Maybe the last 24 hours are a bit too much. Let's say we just figure on the last or focus on the last 15 minutes. You can see we had for example, 22 open file events or connections to or that somebody executed the process and you can see over time how they developed,
07:20
where those were coming from. For example, that somebody tried to log in and further down you can see the actual events. And we capture a lot of metadata. Something that might be nice for you to see is for example, you can see we have a host information. So we know on what kind of host this is running and what operating system. If you're running in a cloud provider on some cloud,
07:43
you can see this is running on AWS and we enriched that with all the information so you know it's just one specific instance that has a problem for example, security wise. Or you could filter down to say like, oh I know only Ubuntu has a specific vulnerability in one specific version and then I'm only interested in those hosts. So you could filter down to those.
08:00
If you're running in Docker or Kubernetes, you could enrich that with the same metadata here so then you could filter down to a specific part or name space or a specific base image if you know that there is a security issue to look into that. So this is what we have running here. So let's see if we can actually find something.
08:22
So I will quickly head out here, leave this user, let's say we want to log in with an elastic user. Well, what happens if we forget the password? That would not be good.
08:41
But that would be kind of a nice opportunity to actually figure out if anybody is trying to SSH into my box. Since this is a security track, did anybody try to SSH into my box? Otherwise we'll see in a moment. So let's see, I always try to throw that in. So let's see what has been happening in SSH. Well, maybe not just the last 15 minutes
09:01
but let's say the last 24 hours. And then we can see how many people tried to log in. This time we were lucky. It's not even that many login attempts. And you could see like public key password. You can see where they're coming from, which users. So these are me probably, admin is not me. We could filter down to the last 15 minutes, for example,
09:22
just to see, okay, something didn't work here. You can see, okay, this is too small. This time the IP is coming from somewhere else. Anybody has any idea where this might be coming from and why? Okay, somebody, that's surprising,
09:42
somebody from Norway or so is trying to log in. The other one is from Vienna. That's where I'm from. I'm using my phone because people are doing funny stuff with the wifi. And if you're roaming with your phone, you will always get an IP address from home. And then it looks as if with a geo IP lookup, as if that was coming from home. But this is just a coincidence.
10:01
This was me trying to log in. Then down here you could see the actual failed login attempts. And if you zoom out and make the timeframe large enough, you would normally have either Russia or China trying to brute force into your instances or probably you now. Ruth me, yeah?
10:22
So let's see, where are you coming from? You can see five from here, 10 from here. And we seem to be doing better than China and Russia today. Or India is also trying to catch up. But you just get the idea of where stuff is coming from. But this is just the auth log and it's still educational to see what people are trying to accomplish here. So let's actually log in with our user
10:44
and do something useful. So for example, I've shown you the rule when somebody's trying to access ETC-passwd. So let's say my user, bless you, bless you again.
11:02
We're just running a cat on ETC-passwd. We're showing the file because this is something we're trying to capture. So let's see if we actually found that. So I'm heading over to the raw events. These are not the ones I want. And you see in the last 24 hours,
11:21
we didn't have just 31 events, but this will be, yeah, we had 600,000 events. But we set this nice little tag. And with this tag, we can filter down and say like, I think it's called ETC-passwd-access. You could filter down on that one and instead of 600,000 events, you just have 131
11:41
because this is run very frequently. So this is still not helping you too much. You can still see which user is calling it and what command and was it successful or not successful. But this is still a bit wide. So the first rule that we set was maybe not super helpful. The other one that we set for the pem conf
12:04
was a bit more limited because it was just bound to that specific user and just a read operation. So let's see if we switch down that filter to that one if we find something more meaningful. And we have developers pem read. So only the developer user is trying to read that file.
12:22
If you drill into that, you can see, this was just me now. If we open that one, you can actually see we were running bincat on that file. You can see which users, which operating system. You can see the exact process information, user IDs, et cetera.
12:40
Obviously, this makes sense if you have some sensitive files or something that you want to keep track of very well that you configure your rules accordingly for that. Like, you can see pem conf is not making much sense here, but it's just to show you the general approach of what you could do. Something else you could do, for example, is if I run to restart nginx,
13:00
so we have nginx running here. So I say service nginx restart. I pick my Elastic Admin user because I know the password for that one. We would be capturing those as well. So for example, you would have multiple places where you could find those. But for example, we have a dashboard
13:21
for process executions in audit beat. We have the process executions. Here you can see, for example, this is the stuff that is being run. Let's just filter down on the Elastic user, whatever that user is up to. Then down here in that list you can see here, this was actually restarting nginx.
13:41
This was one of the actions that the user has been up to. And that way you could just fit, like if somebody broke into your system with a specific user, you could just see what have they been running and what have they been up to to follow along, whatever stuff they tried out. You could also do some other fun stuff. So let's log out from that user and let's say we're an admin user.
14:04
And the admin user is kind of curious. And the admin user tries to look in what do we have in our home directory. And then he sees like, oh, we have an Elastic user. Well, let's see what the Elastic user is up to.
14:24
And the Elastic user has a file secret txt. And obviously our admin cannot resist looking at the file. So let's take a look. Oops, there's not where we want to go. Elastic user, secret, will this work?
14:44
Yes, no, maybe? No, we need sudo. We can run this with sudo, if I type correctly, to secure. And you can see my secret.
15:02
So this was collected. You can, by the way, we have set up a rule to collect those events. So we said like, if somebody who has root permissions looks into the home directory of a non-root user, we want to log that event. And that is something we have in the raw events as well. So if I add my filter rule here,
15:23
I think I called it privilege or elevated, oh no, sorry, not elevated briefs. It is called elevated briefs. Oh yeah, sorry, power abuse, yes. This is the one I'm looking for. But elevated briefs will capture this as well since we ran it with sudo.
15:41
Elevated briefs is just, in my example, anything that you run with sudo, we are capturing here as well. So we might probably find the elevated briefs here if we open that one. So you can see we have a syscall and you can still see this, but this was only found now because it was run with sudo,
16:00
but we're still capturing any sudo event. But yes, power abuse is the proper one if I remembered my own tags. And with power abuse, if we scroll down to those, you see, okay, this was tagged with power abuse, and you can see this was as well the command that we have been running.
16:21
So you find out what somebody is up to. Something else that you might be doing is that, let's say, anybody wants to talk to me, but I think the Wi-Fi here is filtering that out. But generally, if I connect by telnet
16:41
to my instance on that port and I write hello, it will arrive here. And anybody who wants to can just chat, but I think the Wi-Fi might be filtering this out, so you might need to reuse your phone. How would we find out that somebody has opened a port and is doing stuff now? For that, we have slightly extended
17:01
our configuration format, because I said initially the ODD format is a bit crude. And in the very latest release of the stack that came out this week, we added another way to configure things like that, which might be slightly nicer to look at. So here, if I scroll down a bit,
17:22
no, not so far. This is the one I want. Here, for example, we have a so-called system module that is looking for processes and sockets every second and is capturing those. And then you could just say which processes have been run by a user or which sockets have been opened, which also might be interesting
17:41
if somebody opens weird sockets on your system. And to show you what that might look like is, let's throw all of this away, and let's assume somebody opened something on port 1025. I'm just randomly searching for that. You can see we found a bunch of events,
18:01
and you can see, okay, somebody opened an outbound socket. You could just open that event and then see, okay, this is the destination IP. The process has now exited. But this was the specific thing that happened on your instance. You could also see which source port was hit. You could see we would also have a process.
18:22
So somewhere you would find the netcat command that we have been running, depending on which event we're looking at right now. I'm just capturing this in various ways to show you the multiple ways that you could see that. For example, here you can see, okay, netcat was stopped, but we opened that port,
18:41
and this was the command that we had been running, and this was also what opened the port. So you could just figure out like why are suddenly new ports listening, and which user was that, what has been compromised, or where is stuff happening and breaking. One final thing that we have at the bottom here is we can check the file integrity. In the file integrity,
19:00
basically we can just monitor a folder or a file, hash it, and then every time the file changes, we will compare the hash and see that somebody changed the file. Which can also be interesting if suddenly your website is starting to serve some weird content. You might want to figure out when was it changed, and maybe which user changed it, or what were the actual actions they did on the file system.
19:21
So I'm watching my web directory, and to actually see something, we'll need to change that. So let's say, no, var, var lib, oh, var lib, HTML, index HTML.
19:42
Let's say, no, this is the wrong line. Undo, undo, undo. Let's say we're adding something here, so what this looks like is,
20:01
right now, it's just a welcome page. Not super fancy, but let's say somebody wants to add an emoji. This should work. Ah, crap.
20:21
And now, when you reload your page, ah, did it, okay, doesn't look that professional anymore,
20:41
but we do have another dashboard to actually figure out what happened here. So let's say, audit bit, audit bit, and we have a file integrity module. Let's say we just want to change this in the last 15 minutes. And then you can see here, we moved files,
21:02
and we created files. This is because VI's are having this weird replacement pattern. If you used nano or ed or something like that, you would just see a changed file. And if you scroll down, you can see which file's changed, and you can see the actual events where the file was created and replaced, et cetera.
21:23
So you can see all the changes on the file system for that as well. And I think that's pretty much it. You did very well on the slides. If you want to try it out yourself, I'm not giving you the root user, but you can use the other user. It's the elastic user and the password is secret.
21:41
And you cannot create or change anything on the dashboards, but you can look at all the existing dashboards and just filter down on them if you want to play around with that. I let that run for another day or so, then I'll just throw it away. Please don't start sending spam or anything else. Just do some proper research. If you want to get all the configurations,
22:01
I have those on GitHub. You can just look at the configurations and set it up. It's just Terraform and Ansible to set up my instance automatically, and I will just destroy it tomorrow again with that. Yeah, there are some similar solutions. Look at those if you're interested, or look at Auditbeat. That's what we are trying to do and put in our stack.
22:21
And with that, we have two minutes for questions. Any questions? Okay, while the microphone is on the way to you, is anybody using Auditbeat already, just out of curiosity? Okay, not that many, not yet.
22:43
Hi, thank you for the slides and a great presentation. What are the next steps? So we have collected a lot of audit events, and are there automations or some further stuff that you reach automatically? So it's a kind of a manual stuff until,
23:01
so we have to get into Kibana, and let's try to find some events. Yes, so there are two things. However, they are how my salary is being paid. So those are commercial features, so you get them on our cloud service included. We have something called alerting, where you just specify rules, and it will send you Slack, SMS,
23:22
Pager, YouTube, whatever. The other thing that is very helpful in that regard probably is we have something called machine learning, but machine learning is a super overloaded term. It's anomaly detection of time series. So it basically learns what is normal. That could be like logins or failed logins per user, or started processes, or which ports are open. And it just learns that over time it knows, for example,
23:42
during the week you have like a regular curve, and on the weekend it's much lower. And then if you have an outlier, for example, if a Saturday looks like a weekday, it would just alert you because it knows what the pattern is that it is expecting. So for example, if you have normally very few login attempts over the weekend because you're kind of in cooperation and nobody's working on the weekend, it could just alert you and tell you,
24:01
somebody is trying to do as many logins as during the week, which might otherwise with strict alerting rules be very hard to find. So we have these commercial add-ons, but admittedly they're commercial because that is how we finance the rest of the stack. But to get the events, that is all free
24:21
and you can just go crazy with that. All the stuff I've shown you is Apache 2 licensed. Thank you. Cool, I think we have one more minute. If anybody else wants to, okay. I was wondering, so you have very nice rules you can give.
24:45
Sorry, yeah. Now if you end up with quite an extensive rule set, how heavy would that be for your system? I mean, it would not be the first time that all the thing or mounting would like take down the system.
25:01
Obviously if you have too many rules it could take down your system.