Hello, and welcome to my talk, Post and Pre and Security here at the OpenSUSE conference. So let's get started. First, the obligatory who am I slide.

My name is Johannes Seggets. I'm a security engineer at SUSE. I am based in Germany near Nuremberg. So if there is no global pandemic, then I actually go to the office from time to time. My main tasks are code review and product pen testing. This is what we will cover today.

I will give you a quick introduction into the SUSE security team. Then we will talk about the security implications of post and similar hooks in RPM packaging. And then I will give you some tips on how to not get a CVE assigned. Good. So the SUSE security team, we have two groups in the team.

We have the so-called reactive group, where Marcus, Alexander, Robert, and Gianluca work. And there we mostly do incident handling. So as you know, every day, a lot of security issues are discovered. We handle sometimes over 100 issues per day. And for each of these CVEs or other security reports,

we need to analyze is OpenSUSE or SUSE affected. And especially for SUSE, this is a bit more work because there we have older code streams where a lot could have changed. So some packages change programming language or do major changes over the years.

And then having a vulnerability report for current version forces you to go into the code and to analyze in detail if something is a problem in the older code streams or not. After a while, we make sure that a security update gets out and we do this both for SUSE and for OpenSUSE. Then we have the so-called proactive group,

where Mathias, Wolfgang, Paolo, and I work. There we do most of the security work that is done before shipment. So a lot of that is hardening. So we want to make sure that a product that comes out is secure by default. And that security issues that arise later don't affect us if possible.

This is also done by doing code reviews. So especially for more critical code, like setUID binaries, we review the individual code, but of course we can only do this to a given extent because you probably know how much code is contained in a single web browser, how much time it takes to review a bit of code.

And this is just not doable for distribution. I like this comic strip a lot because it covers the conflict between security and normal people very well. Most of the time programmers create something and then the security guy comes along and tells them,

well, do this, do this, do this. And they feel super cool by doing this. I really hope that we are not this way. If we are, then please give us feedback. To a certain extent, it's kind of hard to avoid because we obviously often have to point out problems that is hard to avoid in our line of work,

but we try to be as helpful as possible. And I hope this talk is part of being helpful. Good. I will show you some real world vulnerabilities we found during the last few months. And the examples are obfuscated. So I removed the package names

so that it's not directly clear where this is coming from. But in general, it will not be too hard to identify the package where this is coming from. First of all, sometimes there are some commands that are very specific for a given package, but also the CVs are public. So you can just go and Google them.

So yeah, that would be easy. But it's really important for me to state that this is not a blame game. I myself managed to create issues like this, and I'm pretty sure that almost everyone will sooner or later do this. I really hope that this talk will help in preventing this to some extent, but creating a security problem

is not something to be ashamed of. It's just a fact of life given the current software world we live in. So this is how it started. Ludwig Nusselt contacted us a while ago. I am pretty sure every one of you knows Ludwig because he is very active in the OpenSUSE community. And he used to be a member of the security team,

and he has a really good security mindset. And because of that, we quite often get good reports from him. And he told us that a certain package has a very complex postscript, and that it seems to be vulnerable to a symlink attack. And that could be used for privilege escalation.

So after that hint, we decided, well, if it happens in this package, it might make sense to look at other packages. And over the last one and a half years, we reviewed all of these scripts, and we found quite some issues, and I will show you some of them now. So this was the example that Ludwig sent us in the first place.

So we see here the ownership of a file is changed to a given user and a given group. And the problem here is that the directory where this file resides also belongs to this user. So we have root operating on a user-owned directory,

and that is in general a really, really bad idea. So why is this a problem? Let's give it a try to illustrate this nicely. Let's say I am logged in as Johannes. I'm in the test directory I created.

Then I showed that I'm Johannes and no other user, and I have this test file in this test directory. So everything is well, everything is fine. Now, if root decides to change the ownership of this file, and in this case, I changed the ownership to the user Johannes. The file already belongs to Johannes,

so that doesn't make a lot of sense, but it's just for showing you this. Then you get the results you would expect. You probably all use this command. So the file now belongs to Johannes, everything like we want to see it. But now Johannes becomes the evil mastermind he always was

and turns into a great hacker. And he deletes the test file and then links etc shadow to the test file. Now, if you look at the directory, you will see that test file links to etc shadow. And now if root redoes the steps that I showed before,

so it changes the ownership of this file to Johannes, then you will see the link still is there. But if you also list the file etc shadow, you will see that it now belongs to Johannes. And with that Johannes, so me, has the full control over the system.

And that means we have a local privilege escalation from user to root. Privilege escalation is a term that describes that a privilege is escalated to a different level. And that usually is, especially in the security realm, a security problem in itself,

because you don't want to have the ability for normal users to escalate their privileges to a different level, unless you specify exactly what happens there. And in this case, the friendly security team will assign you a CVE from our pool. So we are a CNA, a CVE numbering authority,

and we have a certain pool of CVEs already assigned that we can use for issues like that. So if something is SUSE or open SUSE specific, we can just take one of our CVEs, assign it, and then you get a friendly bug report asking you to fix this.

So why did this actually happen? The main problem here is that operating on anything, a user controls, and especially a directory is really dangerous. You will operate in an environment that can change while you work there. And just think of working on a ladder where the ground where the ladder stands on

changes while you're up there. That is not a very pleasant experience. And basically the same happens if you operate as a privileged user or in general, as a user with different privileges in a directory or anywhere where a different user can control the environment.

Now you could say, well, I can just create a check to ensure that everything is fine before I actually do the change I want to do. And then you will probably hear the term toctou from us, which stands for time of check, time of use. And that is also a very common security issue.

The problem here is that you first have a check and after you checked something, you then in a second step do a certain operation. And between those two operations, there's a certain amount of time. So let's look at this example that we found a while ago.

So the packager iterated over various log files and first he tested if the log file exists and if it doesn't exist, then he creates it and then changes the owner of the file. There's a small deviation that we also see sometimes.

So here the test is not if the file exists, but here the packager apparently already was aware that security issues could arise by siblings. So he tests here if this file is not assembling and if it's not assembling, then the owner has changed.

What you need to make sure of is that operations are atomic. That means they happen in one step because TOC issues are harder to exploit, but in the end, it's just a matter of optimization. No matter how small that time window is between those two operations, the attacker will have a chance to get in between.

The smaller it is, the harder it is of course, but most often it's just a question of optimization, not if it's generally possible to do it. So by making sure that operations are atomic, that means they happen either in one step or they don't happen at all. You prevent this because there is no time window

between different operations. So I changed the example a little bit to make sure that we actually managed to jump in here. And in this case, I added the sleep statement. And then as a user, we go to the directory,

we delete all the files. So we bypass the first check that the file exists. And then we use inotify-wait. And inotify is the ability of the Linux kernel to notify you of changes. And inotify-wait is one of the utilities you can use on the shell

to get notified of changes on files or directories. So what we do here is we delete all the files. Then we wait until something happens. So that will trigger as soon as the file is created by the touch command. Then we delete everything again. And then we create our symlink to etc shadow.

In this case, this will work always because we have this helpful sleep command in here. And this might seem a little bit constructed, but I assure you it's not. First of all, the exploit that I showed you here below is very crude. We also use shell constructs.

They are really slow. You can do something similar in C and there you can become really, really fast. And then you have a good chance that you manage to get between those shell commands just because you're so much faster. And if that doesn't work, then you can still use other tricks. So for example, you can create IO load

so that the initial touch is slow. And with that, you will have a very much higher chance of actually winning this race. Just because you can't figure out a way on how to win this race doesn't mean that someone else can't. This is one of the problems in security.

Just one guy has to figure out a way to get around this and we all have a problem. So changing the owner or changing the access mode of a file are obviously bad, but let's say you just want to update the configuration file, then you should be good here, right?

And if I'm asking like that, then the answer is obviously no because basically everything is dangerous. Let's take this example here. Here a packager tried to use set to replace something in a configuration file. So the new version renamed a configuration option and they copied the existing configuration file

to a temporary file, probably to prevent an error from killing the old configuration file and then moved the old configuration file back over the, sorry, copied the temporary configuration file over the old configuration file.

So that looks pretty unproblematic, but that gives us quite some options here. So an attacker can hear with root privileges if this runs as post or prescript, either create new files. So we can create a Zimlink from the temporary file

to some other file. And once this command I showed you earlier runs, it will create this file. And that might not seem so bad, but quite often you have the ability to trigger some action by creating a file. Some demons look for certain files to trigger actions.

Sometimes you can write something to devices or whatever. So this is definitely not harmless. You can also overwrite existing files. So just link the temporary file to etc shadow and it will be overwritten. So you basically denied of service the system.

Or in this case, you can even overwrite with chosen content. So you delete the configuration file, you add into the configuration file what you want, considering that there's the set command running. So you might have to fiddle it around a bit that it is not destroyed what you actually want to write.

And then you link it to the bug file. And in this case, you will again, overwrite etc shadow with something that you control, which gives you control of the system. One thing you need to be really wary of are upgrade helpers. It's really great that you want to help the user

and sometimes demons change a lot and you want to make sure that if someone goes from version two to version three, that their life is easier. But really often we find these snippets to be problematic. So here's one example. There was obviously a bigger change in the configuration.

So the package had decided to use Org to change the configuration to fit the new format. And for that, they create their first test if the configuration file exists, then they create this Org script in temp, have some Org commands in there, and then they dump the old configuration,

change it with Org and then write it to someplace. And this again is unfortunately problematic. This is a direct root exploit that you can gain by pre-creating the file, monitoring it and overwriting it at the right moment. And you do this by pre-creating the file

in the temporary directory. With that, you are the owner of this file and you can control it. Then you use inotify-wait to wait for a change on this file. So once root, I go back one slide. So as soon as root creates this file or wants to change this file here with cat,

then this inotify-wait will trigger. And then you can jump in and just overwrite this file since you control it with an arbitrary Org script. And Org has the system function that allows you to then take control of the system again. In general, slash temp and slash var temp are evil

and to be avoided. That holds generally true, but especially in post and pre-scripts, you don't have to do this because your root, you can just use some directory and run or wherever and be safe there. In general, as you probably know,

these temporary directories are problematic because everyone can create files in there and you have a very special situation there. Modern kernels have some countermeasures to prevent certain exploits in there, but still, if you don't have to, then don't operate in these directories. And if you have to,

then create secure temporary files with mktemp. You can either create directly as a temporary file or you can create a directory that is created in a secure way with secure permissions and then operate in this directory. That is definitely the way to go if you need to do something in temp.

So now we've discussed quite a lot of problems that might arise here, let's get to the solutions. The first solution might seem a bit trivial, but just don't use it. And I know this might seem like a useless suggestion, but you would really be surprised how often we find a dangerous construct

and then we approach the packager and they just tell us, well, yeah, I don't actually need it anyway, let's just delete it. And from our point of view, this is pretty nice because that makes the security fix pretty easy. Sometimes they just replace it with a certain RPM packaging construct and also then don't need it.

So really consider if you need to take the risk of having these post and pre-scripts, as long as you operate on only root controlled files and directories and don't include any user controlled content, then you're fine. So let's say a call to update alternatives

or whatever, that is fine. But as soon as you start to do more complex stuff, you have to be aware that this is actually a security risk. And then you need to consider, is that worth the risk? In general, again, use RPM as much as possible. RPM gives you a lot of power and you can just delegate certain tasks to RPM directly.

Then you don't have to script it yourself and you don't run into some of these issues. The next suggestion is run it as an unprivileged user if possible. And there's a nice helper called set-prif. You can use it like this.

So here I have a root shell and then I use set-prif. First, I clear all of the groups that I might potentially have. Then I set the no new-prifs flag that specifies that the child that is now created is never able to gain any additional privileges.

This is a rather new kernel feature and is really helpful to constrain processes. Then I specify a certain group. So here it's the users group. I specify a user ID, in this case, my name. I drop all of the capabilities that I might have and I reset the environment

and stop the parsing of arguments with a double minus. And then I specify the command. In this case, it's just a simple idea to show what we did here. And with that, you can easily run commands as a certain user. The advantage of set-prif is also that it doesn't depend on PAM or anything. It's a really simple tool

and can be used for tasks like these. This obviously does not work if you have to change permissions or anything else privileged. So if you need these privileges, then you can't just drop them. But for example, if you want to fiddle with config files like we've seen before, you don't need these root privileges to do this.

So if you run all of these commands as the user that is already able to control the config file, then you can't run into a problem because everything that the user might do to trick you to do something different, he can already do himself because he's running on the same privileges.

So this is definitely a really good way if you don't need to do anything where you necessarily need root permissions. And maybe you need just some capabilities, then you can also do this and limit your attack surface. Next is to consider your permissions model.

Is it really necessary that a certain user owns a directory or file? We also had that quite a few times that we talked to the packager and he said, well, then let's just make root the owner and everything worked as before. So sometimes it's just not necessary. You really have to find a good justification for that.

If you can't do this, if you really need the user to have these permissions, then think about if you can structure it in a way that root doesn't have to operate in there. So let's say a user needs to be able to control a given file. Maybe you can structure a package in a way that the operations you need to do as root

happen somewhere else so you don't clash there. Then if you are able to use proper languages that mostly means not shell, then you can try to work with file descriptors and not pass. A lot of these problems arise

because the way that a path can be constructed in Unix are very flexible, which is nice and helpful, but it's also dangerous from a security point of view. So if you start from a safe, that mostly means root own starting place, then with certain functions, you can work with file descriptors a lot

and don't have to use paths that much. If you want to see a concrete example because we don't have the time to go into detail here, then check out source check start CPP in the OpenSUSE permissions package on GitHub, and especially here the safe open function.

We had the problem a while ago in check start that we noticed that this is doing also unsafe operations and then Malte Kraus and Matthias Gassner worked on this and refactored the code quite heavily to make sure that this code runs in a safe way.

And so you can look at the source code directly. Also helpful is look at the evolution of the commits because there you see how tricky it is at times to get this right. Another way to deal with this is to ensure that you set proper permissions starting from a safe base.

So let's say you need to operate somewhere where a user has control, then you can, starting from a safe base, change control back to root and then commit the change and then start to change the permissions back in reverse order. So let's say you need to change the permissions

of a log file that lives in a directory that belongs to a user. And if you do this directly, then we run into the problems that I showed you before. If you first start from var log, which is root owned and change the permissions of the directory back to root, then the user loses the ability

to arbitrarily change this directory while you're working in there. Then you go into the directory, do what you need to do there, then go back, change the permissions of the directory also back to this user. So for a short while, the user loses the ability control this directory and that is also a way to work around this.

And if everything else fails and you really need to use bash, then have a look at the main page of the commands you're using. For example, Joan has the flag minus H that prevents Zimlink following. And this is by no means a perfect fix.

In the end, you will still have a security vulnerability because there again, you have this time of check, time of use issue. Joan checks if there's a Zimlink and if not, then it continues. This is not done in a perfect way, but it keeps this window of attack very small. And sometimes we have to settle for an imperfect solution

if there's absolutely no alternative, but this should really be the last resort. And optimally, you would discuss that with us first. Again, use RPM as much as possible. And if you don't have to, then just unscript it yourself. It's more work for you.

It's dangerous for users and you might get a CVE from us. So with that, thank you for your attention. In general, if you have questions, I will be in one of these chat rooms now to answer any questions. I will also be around for the rest of the event in some shape or form. If you see there's a recording,

then just reach out to the security team. We are reachable via email. If you want to reach us this way, we are also available in ISE or whatever, but email is usually the best form to reach us. Thank you very much and enjoy your OpenSUSE conference.