Generating seccomp profiles for containers using podman and eBPF
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 44 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46117 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
All Systems Go! 201924 / 44
5
10
17
20
21
30
32
36
44
00:00
Physical systemSystem programmingMetropolitan area network2 (number)Profil (magazine)Disk read-and-write headComputer animation
00:30
XMLMeeting/InterviewComputer animation
00:54
Windows RegistryComputer-generated imageryXMLUMLJSON
01:20
System callWordSystem callSurfaceVulnerability (computing)Kernel (computing)Goodness of fitInformation securityCompilation albumVirtual machineSet (mathematics)Physical systemJSONXMLComputer animation
02:38
View (database)Reading (process)Asynchronous Transfer ModePhysical systemInformation securityComputerKernel (computing)Process (computing)System programmingNetwork topologyGrid ComputingComputer programRevision controlComa BerenicesParameter (computer programming)Set (mathematics)Link (knot theory)Web pageContent (media)Event horizonRandom numberOnline helpFreewareInformationWritingExtension (kinesiology)System callFunction (mathematics)Digital filterRule of inferenceGoogolWeb browserGraphical user interfaceSet (mathematics)Compilation albumForcing (mathematics)1 (number)Computer animation
03:06
State of matterDigital filterSystem callPoint (geometry)Computer fontInformation securityUsabilityDependent and independent variablesMultiplication signFilter <Stochastik>UsabilityInformation securitySystem callState of matterProcess (computing)Type theoryProfil (magazine)Default (computer science)Computer animation
03:36
Default (computer science)User profileHacker (term)Web pagePermianStress (mechanics)System callSoftware testingInheritance (object-oriented programming)Profil (magazine)CausalityCartesian coordinate systemSystem callFilter <Stochastik>Default (computer science)QuicksortRight angleXMLComputer animation
04:12
Digital filterInformation securityBitSet (mathematics)Filter <Stochastik>Multiplication signCompilation albumForcing (mathematics)Information securityComputer animation
04:45
Rule of inferenceProcess (computing)Software developerAsynchronous Transfer ModeSoftware testingKernel (computing)Multiplication signDisk read-and-write headCartesian coordinate systemSoftware bugEntire functionTraffic reportingRight angleDatabaseTerm (mathematics)Software testingPhysical systemNeuroinformatikSuite (music)QuicksortInformation securityAsynchronous Transfer ModeComputer programmingRule of inferenceLoginSoftware frameworkKernel (computing)System callXMLComputer animation
06:58
Walsh functionCodeGoogolProcess (computing)Rule of inferenceQuicksortMeeting/Interview
07:26
Digital filterSystem callBlogLoginMiniDiscTracing (software)Figurate numberProcess (computing)Compilation albumTraffic reportingComputer-assisted translationSet (mathematics)Kernel (computing)Computer animation
08:57
Process (computing)Electric currentDigital filterPrincipal ideal domainRow (database)CodePoint (geometry)Process (computing)System callPersonal identification numberPresentation of a groupMultiplication signNamespaceMereologyKernel (computing)Group actionRadio-frequency identificationDifferent (Kate Ryan album)FamilyConditional-access moduleType theoryElectric generatorFilter <Stochastik>Figurate numberPrincipal ideal domainComputer animation
11:02
User profileRobotComputer fileCodeSystem callRun time (program lifecycle phase)Metropolitan area networkFlow separationComputer animation
11:40
Tracing (software)StatisticsOpen setTexture mappingLatent heatProcess (computing)Software testingSystem programmingComputer fileRun time (program lifecycle phase)Process (computing)Computer programmingPhysical systemSystem callHookingFilter <Stochastik>MereologyCondition numberDemo (music)Phase transitionIntegrated development environmentPrincipal ideal domainPoint (geometry)Compilation albumMappingPersonal identification numberSoftware testingSoftware developeroutputMetropolitan area networkLangevin-GleichungRadio-frequency identification
13:58
System callCompilation albumBitProcess (computing)DrawingComputer animation
14:56
PasswordBootingHypermediaInformation securityRootLatent heatGroup actionRight angleMultiplication signMetropolitan area networkSystem callLoginQuicksortDefault (computer science)Flash memoryComputer configurationSet (mathematics)Rule of inferenceComputer fileFilter <Stochastik>HookingRoundness (object)XMLComputer animation
16:49
StatisticsLink (knot theory)Computer fileDomain nameSystem callNetwork socketScripting languageAttribute grammar1 (number)Thermal expansionFerry CorstenPhysical systemGroup actionComputer animation
17:50
StatisticsComputer fontNetwork socketInformation securityRule of inferenceCompilation albumComputer animation
18:13
RootBootingHypermediaLink (knot theory)Information securityTotal S.A.StatisticsNetwork socketDemo (music)Rule of inferencePhysical systemComputer animation
18:35
Demo (music)Software testingCompilation albumSuite (music)Run time (program lifecycle phase)Physical systemSet (mathematics)Rule of inferenceComputer animation
19:03
Digital filterPermianPhysical systemElectric generatorDefault (computer science)Filter <Stochastik>Multiplication signVector potentialSound effectSheaf (mathematics)CodeCompilation albumSpacetimeStreaming mediaSystem callSpectrum (functional analysis)Computer animation
20:09
Physical systemSystem programmingSound effectGroup actionConfiguration spaceSystem callCodeInformation securityStack (abstract data type)Physical systemDirect numerical simulationDifferent (Kate Ryan album)Resolvent formalismMathematicsVector potentialReading (process)PermianCartesian coordinate systemQuicksortComputer filePrime idealMeeting/Interview
21:56
Hill differential equationPermianPauli exclusion principleMultitier architectureEmailIRIS-TUniform boundedness principleMIDIExecution unitCNNFundamental theorem of algebraXINGCodeCompilation albumFigurate numberSystem callComputer configurationSoftware developerCartesian coordinate systemProfil (magazine)Right angleProcess (computing)Fluid staticsPhysical systemMetropolitan area networkView (database)Phase transitionComputer fileKernel (computing)Line (geometry)Point (geometry)Information securityError messageMultiplication signWeb pageComputer programmingNamespaceOffice suiteOperating systemData loggerMathematical analysisWorkstation <Musikinstrument>Block (periodic table)InformationMathematicsGame controllerDirectory serviceLoginDifferent (Kate Ryan album)Regular graphComputer animation
26:49
WebsiteSystem programmingLattice (order)Computer animation
Transcript: English(auto-generated)
00:05
The next talk will be given by Dan Walsh from Red Hat about generating a second profiles for containers using Portman and EBPF. All right, thank you. Thanks for having me. I was here last year gave a talk on replacing a darker with pod man and this is not that much about pod man, but
00:26
pod man is the tool that we're using to do this, but I don't know if I did this last year, but I usually do this before, since it drives me nuts. Everybody please stand up.
00:43
Please read out loud all text that is in.
01:25
This is my Dan Chiodi movement and I'm just trying to get people to expand behind, not have to prefix every word with D-O-C-K-E-I, unless you want to call all Linuxes Red Hat Linux, but you know.
01:41
So the goal of this talk or with this tool that we're developing is to basically run a container with tighter syscall filtering with seccomp. Does everybody know what seccomp is? All right, good. What seccomp does is it allows us to shrink the attack surface on the
02:01
kernel, so it's one of the, when I give talks on container security, I used to talk about things like SELinux and capabilities, and you get to seccomp, and seccomp's really cool in that we can basically take a Linux system, there's 600, the x86-64 machine, there's about 650 syscalls.
02:22
And if we can cut out a whole bunch of those syscalls, then a vulnerability in one of those syscalls that leads to a kernel exploit, if you don't have that syscall, you can't get the kernel exploit. And so what we really want to do is cut down on syscalls. But when we're looking at this, so this really, this
02:41
is what Wikipedia defines as seccomp, but this is really talking about the initial seccomp, which is cut down to, I think, four syscalls. I think it was read-write, actually it's in there somewhere. But we're really talking about seccomp-BPF, which is the thing that Google further enhanced the original seccomp to basically allow you to have all syscalls, and then allow you to choose which ones you want to control.
03:07
So the goal, again, is to run containers with tighter syscall filtering. So the state-of-the-art in containers was actually developed by Jessie Frisell. Jessie Frisell at the time was working for Docker, and she wrote a really, what she called a rant on usability, usable security.
03:26
So really what we needed to do is we wanted to take advantage of seccomp, but the type of jobs that run inside of containers are so varied. And she said, they haven't written a default seccomp profile for Docker. I'm pretty familiar with how hard it is for people to use.
03:43
And it requires a deep knowledge of application, being contained in syscalls. You know, what syscalls does a container require? And then she says down here, turning on something that will cause EPRM by default if we left it important, syscall is terrifying. So really what she's saying here is if we had tight syscall filters running
04:02
inside of all these containers, then it would just cause people to turn off seccomp. So we had to have sort of a general, fairly loose capability for seccomps. The idea is to build, what she wanted was a build time generated and applied on the run seccomp filters.
04:26
And then she basically says down here, if it causes any problems, people will just turn off the security. And I'm going to talk a little bit about that at the end, and this happens all the time with, and of course I see Linux dimensions, because I'm sure everybody in this room has it set in force one.
04:45
But anyways. So, ooh, my picture didn't show up. That's too bad. I always had the problem with the term one size fits all, so there's supposed to be a picture there with a little tiny hat on its head. So imagine, I have a size 8 head, right, so it's a very large head, and any time I show up for a hat, it's always a hassle.
05:08
And you pick up these hats that say one size fits all, and I put them on my head and it looks like a little beanie on me. So security is the same thing. We have one size, with our container world, we have one size fits all for the security. And really, unless you're going to go deep into analyzing how containers run,
05:24
how do you figure out what size syscalls your application is going to do? So what we decided to do this summer is, oh, so I'll talk quickly about what SELinux did. SELinux really had the same problem. SELinux policy is basically picking out a certain application, say an Apache application or a database.
05:45
And what SELinux did is we would have sort of general rules. We'd have a general idea of how an application would run, and we'd write a generalized framework for what the application did. Then we would basically put that policy in place and run it in what we call permissive mode, and just basically run it through an entire test suite.
06:03
And then we'd go to auditing system. We'd go to the audit logs and basically continuously pull out the rules as they came along. And there was a tool called audit to allow that translated the ABC rules that were showing up in audit logs into allow rules. And then we'd go back to step two and just keep on.
06:23
And I don't know if people, my first computer course I ever took was explaining how you would write a program for taking a shower in the morning with shampoo. And it used to say lather, rinse, repeat. And of course you'd be in the shower until the thing ran out of shampoo and the kernel crashed, right?
06:41
So basically you'd keep on doing this. Then we would take that policy and we'd actually put it out into Fedora, into rawhide, and we'd get bug reports and continuously get bug reports. And over time we figured out what the application did. So it's not easy to figure out applications, how they work. So this summer we decided to do a Google, try to build tools that we could at least do what we did with SELinux to figure out seccomp rules.
07:09
So this guy here, Divyesh, did most of the work on this. And Valentin Rothberg is on my team and he really sort of mentored him and I was involved in the process as well.
07:21
I wanted to have one of those two guys give this talk, but neither one of them could make it here. So that's why you get me. So the first step we wanted to do is we wanted to investigate how we would figure out what syscalls were happening inside of a container. And the first idea everybody has is, oh, just use this strace or ptrace.
07:42
And if you saw our inspector gadget yesterday, you'd realize that that's slow and very difficult to filter. It's also hard to figure out what the processes inside of the container are and how to follow along with that. So the next step is we could do what SELinux did, which is basically turn on seccomp filtering, auditing,
08:06
and then just keep on going to the audit logs and get the information out of the audit log about which syscalls are being called. So the kernel can report those to the thing. But the problem is the kernel is just going to start spewing all syscalls to it, or you could get it down to a process.
08:23
But there's no idea in the audit log what a container is. The Linux kernel has no idea what a container is. And there's been a pull request in the kernel for five, six, seven years now to add what's called a container ID to the audit log. So you could actually go into an audit log and figure out at least all these processes coming from the same container
08:44
and maybe eventually trace a bunch of processes back to an individual container. So because of that, it became very quickly difficult to figure out with the audit log which container generated them.
09:00
So we basically went and looked at eBPF at this point. And eBPF to me was a really cool thing. And as I said, I'm giving a lot of credit to Inspector Gadget. That was really an awesome presentation to show all the different things you can start to do with eBPF, basically revealing information out of the kernel. So what we're doing is very similar.
09:22
Basically we're watching for sysenterer of the process, by basically the process every time you go into sysenterer on a PID ID, and you're able to watch all the syscalls that basically that process generates. And we can generate, watch all of its children and grandchildren.
09:42
The other thing that we did, so the talk on Inspector Gadget yesterday talked about, they were taking advantage of cgroups v2 to basically figure out which processes are inside of the container. The problem is we didn't have cgroup v2 everywhere. So we relied on, we're basically looking at the syscalls as they come in and looking at the PID namespace
10:02
and basically deciding whether or not the process is inside the container based on the PID namespace. So it's a little bit different, but eventually once cgroups v2 becomes prevalent, I think we probably want to take advantage of that filtering. So this is basically what's going on inside of this thing.
10:20
So one of the things we needed to do is figure out, an interesting problem in this is runc creates the process, the PID1 of the container early in the process, but then it continues to do stuff in that process's state. And those things that it's doing, we don't want to record as being part of the seccomp filter.
10:42
So what we really needed to do was figure out, there's three or four syscalls that come in at the beginning that we basically need to dump out because we don't want those being allowed because they're very, very privileged. Because if you imagine runc is setting up user namespaces and all that type of stuff, we didn't want it being allowed to the container process.
11:02
So he's opened up a pull request, and originally this talk even says that this is associated with Podman. It's being done underneath Podman, but he's basically creating an OCI runtime. So an OCI runtime that can be run inside of any container engine.
11:21
So you could run it inside your Kubernetes underneath cryo container d. Theoretically you can run it with Docker, Podman, anything can do it because it's really separate from the tool, but we'll be using Podman to generate it. Right now it's a pull request on Podman, but I think we're going to put it in as a separate package that people can just download and play with.
11:42
So how to start tracing? Well, we need to know when the container starts. So you run Podman, the container doesn't start when Podman. Podman's going to go out and create an OCI runtime. OCI runtime is going to be read by runc. Runc eventually is going to do the fork and exec to create the PID1 on the container, and then it's going to launch, actually exec the container
12:01
so the container doesn't start until that last point. So the best way to do that is with OCI hooks because an OCI hook is in the OCI spec. The OCI runtime will call out to individual hooks at certain parts of the development phase. So basically after it creates the PID, PID1 of the container,
12:22
it actually stops and calls out to the OCI hook and basically hands us at that point the actual PID that's going to be the PID of the container, but the container's not doing anything yet. So history of OCI hooks. They've been around for a little while. They allow you to do things like pre-start and post-stop.
12:41
So the OCI tools will call in as the container starts and as the container stops, different phases of it. So what happens is we run the tracer when the OCI on the pre-start attaches the eBPF program to it, watch for the anti-trace, and then start mapping. And then when we're done, we basically send...
13:00
Right now he sends a signal to the process to say the container is done, so save out to your file. So we use Podman for testing, and one of the cool things we did with OCI hooks, and I'm going to show you what an OCI hook looks like in a second, is you can actually set up OCI hooks so that they only run on certain conditions. So the first one we built was OCI system D hooks,
13:23
so we could set up system D environment before it runs a container, but you'd want to know if the container's going to be running PID 1 of system D. So another thing you can do is you can do this annotations. So this is the way you trigger a container to basically start doing the set comp filtering, and it basically says using IO container syscall,
13:44
and then we're outputting to a certain file. And that's the file we're going to look at generating the syscall filters. Okay, so let's actually do a demo.
14:02
Okay, so when I talked about one-size-fits-all, just before I sat down to do this, start talking to you guys, I went and grabbed... Remember I said there's 650 syscalls in x86-64? Well, just turning on set comp, you eliminate half of them because you don't tend to run 32-bit code inside of a container,
14:23
so we can turn off all the 32-bit syscalls. So that drops us down to around 325 syscalls. If you go through Jesse Frizzell's syscall thing, it drops it down to about 313 syscalls. So as much as set comp seems like it's going to be powerful, we require all these hundreds of syscalls
14:43
just to have general purpose and not everybody turning it off. So it's good, we went from 650 down to 313, but it could be better. Okay, so this is when you're running hooks
15:10
inside of OCI hooks, this is sort of what they look like. You have the definition of the hook, the executable, and then you can basically tell the specification
15:20
that I only want to run it if I have an annotation that looks like that, otherwise the executable won't start, so you don't have overhead if you're not filtering. So that's basically what a specification looks like. So right now I'm running, so up here you see I'm running the container, I'm running podman, annotation of that, writing out the syscall,
15:42
and I just did an ls of flash to generate the syscall, and that generated something that'll look like this. So basically it says I'm generating, it's a JSON file that the container, that OCI understands, and it basically says
16:01
default action is to return an error, but if I allow it, then just allow it, and these are the rules, the syscalls that it actually found when it was running. So now, I'm going to run the container, so this time I changed the option on podman to actually use my newly generated seccomp filter. So this time, the original one,
16:22
I used the default, which is the one Jesse Frizzell wrote, now I'm running the same exact command, and sure enough, it fully allows it. So I decide to change the round down here, I'm about to run it, and I've got to run it with an ls dash l. Okay, so just slightly more, but I'm still running with that nice, tight policy, and guess what, I've got permission denied. So because I'm running
16:42
seccomp filters on it, now just adding the dash l causes a problem. If I go into the audit logs, you'll see all the audit logs generating all the syscalls that happen. So basically, if you look at this, the first one didn't do it, this one's looking for
17:00
xaddits, basically trying to read the attributes of the individual files inside of it. Down here is actually the ones that were missing. So there's actually connect, futex, getpids, gettids. So basically what's happening here is this is actually going out and creating a socket. So just doing an ls dash
17:22
l of a file actually goes out when it's doing the getpwusers, basically looking at the user ID, that's actually going to talk to sssd on the system, and actually get connect a socket, a unix domain socket, to that. So you can imagine that there's just suddenly an expansion of the amount of syscalls just by adding
17:42
that. So now I'm going to take these syscalls that got basically I have bash scripts that grabbed them. This is when you do the unthinkable and actually run a vi on a json in front of 50 people, and
18:00
hope that you don't screw up. So I just added those rules to my seccomp filter, seccomp filter, and voila. So it just took those rules, added them into the system, and basically allowed me to run, and now
18:22
I'm just going to give you, so here's the original rules versus the new rules added to it. And that's it. So basically the idea here is that we could take this tool and start to run, say in your CICD
18:41
system, you might want to run all of your tests, a full test suite on top of an individual container, and then generate your seccomp rules based on that, and then when you're shipping it, you would ship that seccomp filter and you can use with any container runtime, you can specify I have one minute left, holy smokes.
19:05
Okay. So anyways, you could ship that seccomp filter. Obviously there's going to be problems, and that's why we turn on the audit, so any syscalls, you can start to monitor those audits to see if you're getting any denials, but this gives you the opportunity, potential opportunity to generate seccomp
19:20
filters at a very small phase. Couple of problems with seccomp, this one, a lot of people aren't using seccomp right now because they think it is slow, but we actually found that for some reason live seccomp by default turns on the Spectre melt-on protections, which is a huge performance hit, like a 25% performance hit, so a lot of people are saying that seccomp
19:42
is slow because this is a side effect, so the latest code, we just got it into the OCI screen, it allows us to specify whether we want the Spectre melt-on to go on, and it allows us to turn on that, the auditing system to be able to do it. So this quick section, I'm going to ask for questions,
20:00
and anybody who wants to talk about friendly EPIRM, I'd love to talk about that, but I'm out of time. Any questions? Yes. I guess when you run an application, you are not going to take all the possible code paths, is there always going to be possibly a syscall
20:23
that you miss? Correct. So the question is, basically, there's always going to be additional syscalls, and the funny thing is, I can actually, with containers it's less likely, but traditionally in SELinux, it was more about people modifying
20:41
say, switch files, so all of a sudden they add this full LDAP stack that's happening because I did an ls-l, so I'm calling out the DNS resolvers and all this stuff, so there's all these side effects potentially. Yeah, but it's a fundamental problem as we tried to tighten the security on things that depending on the different code paths, there's going to be issues,
21:02
there's potential issues. One of the things we did with SELinux is we actually, and maybe we'll do this eventually, is there's certain actions that we know, we'll see one syscall, like ls, getstat, and all of a sudden we know that read involves like five syscalls. There's like lock,
21:22
get adder, read, so as soon as you see a read, we should just instantaneously give all five of those syscalls to it. So we could start to build up the understanding of the system to add a whole group of syscalls. But my goal is to get, if I can get from 333 down to 200, that would be a huge improvement.
21:41
But I do take the risk of suddenly a change in configuration is going to cause get adder. One of the reasons I wanted to talk about Friendly Eperm is exactly that. So I'll just quickly go into Friendly Eperm before he throws me out of here. So Friendly Eperm, this was actually a proposal we had back in 2010. We saw
22:01
this coming. Right now, when you run a process on a Linux system, there's about ten different ways you can get permission denied. So you can get permission denied by seccomp, you can get it from user namespace, you can get it from regular Unix ownership permissions, that's called discretionary access control. You can get it from
22:22
SELinux, you can get it from Appon, you can get it from like five other LSMs, you can get it from user namespace. The tools basically get permission denied. You as a user, what do you do when your application gets permission denied? OK, pseudo. OK, that's one thing you do. What's the other thing if you're doing when you're running containers
22:40
with Podman, what are you going to do? Dash dash permissive, right? Instantaneously, we turn off all of the protections. And the reason for that is because you can't figure it out. Even if you contact me, and I pretty much know why you're going to get permission denied, I can't figure it out without making you go and do like ten different rock fetches. OK, turn off SELinux.
23:01
Nope, that didn't do it. Turn SELinux back on, turn off seccomp. No, that didn't do it. Turn off capabilities. Start adding these certain capabilities. Who knows why you get permission denied? The frigging kernel knows. And the kernel ain't telling you. Or if it's telling you, it's putting it in some random place in the operating system that you don't know. SELinux will issue, and now seccomp issues go to the audit log.
23:23
So Apache gets an error, permission denied. It can't write into its log file saying, I'm not allowed to do this because SELinux blocked me. Or I'm not allowed to do this because I don't have the capabilities. So we have this fundamental problem with security, and the only option people have is to turn it off. So we opened a friendly EPIRM years ago because we wanted the frigging kernel
23:43
to tell us why you're giving us permission denied. We wanted to allow Apache to go to the kernel and say, why did you just deny me that? And it's inherently racy. The whole syscall, the only thing the syscall can return you is EPIRM. So originally we were saying, well maybe we could return to like a secondary thing that's a, you know, here's a little tag
24:01
that says why. After the secondary phase we wanted to go to look into the system and say, say the proc system and say, give me a proc status why'd you give me a frigging EPIRM? And I come up with an SELinux line that's saying SELinux blocked it and now you could write it to your log file. And that's racy because me asking the
24:22
proc, I could get permission denied. So basically we went back and forth on this for a while and Linus finally told us, you know, get lost. And now we come eight years later, nine years later and it's ten times worse. Now everybody that's running containers is facing this problem. So with
24:40
the stuff that he showed Inspector Gadget yesterday or with this feature, my goal now is to get to the point where we potentially could use eBPF to go to the kernel and say, you know, so you could basically run a container, get permission denied and say, okay let me set an annotation, run the container again and have the kernel tell me why I'm getting permission denied. And you could actually
25:01
figure out what's going on, figure out, you know, right now as I said, you know, that. So anyways, yes. When's the next talk? As a developer on some of these things, can I get a friendly einval as well? Yeah, well,
25:20
einval and eperma, two ways of saying the same thing. But the goal would be to basically, when we have this, I don't know if it goes through Inspector Gadget, but basically this tool and again this is in my brain, not anything in reality, if it could figure out, yeah, allow you to specify which error node from a syscall you would want
25:43
to basically ask the kernel why did you give me that, you know, that error node and have the kernel reveal some information. Last question. Would it make sense to rely on static code analysis to figure out which second profile you need, kind of as an answer to not triggering all the code paths? I mean, we haven't talked about that
26:03
from a syscall point of view, but that used to be asked all the time with SELinux. The problem is that it's not, you can look at your code until you're blue in the face, but you have to basically go and analyze all of glibc. I mean, maybe if you have a static program that might be impossible, but the issues with SELinux, we
26:21
kind of knew what the application was doing, it was when people would change the underlying tooling on the system that you would end up with problems just by, as I said, setting up NFS. Put an NFS home directory on there and all of a sudden the whole world changes. And you don't have yellow pages anymore, but yellow pages used to be like, everything's open?
26:42
All ports, all things, so it's tough. Okay, thanks for having me.