Hello everyone, my name is Nikolay Kondrashov. I'm a software engineer at Red Hat. I'm going to be talking about this project that we have of implementing user session recording in a way that is suitable for using the enterprise such as banks or

hospitals or just commercial organizations. I work at the common login team, which has started not so long ago.

And they are focusing on getting the logs together from across all the Red Hat products and putting them in one place that is usable. I work mostly on this project, user session recording project, and I maintain three RedHats packages as well.

In my free time, I maintain the DigiMed project, which is about supporting graphics tablets on Linux and also, I play with embedded well in what's left of my time. So has anybody ever used anything like user session recording before?

Oh wow. Okay, has anybody set it up? Cool. Has anybody been recorded? Okay, has anybody

ever wished that he has recorded user sessions? Oh, nice. Okay, great. Let's see. So the premise is basic. Some companies need to comply with government regulations.

Some need to just track contractors or some visitors on their systems, and they need to know what happened to their servers if something goes wrong and who did it. So in an ideal world, people want to record everything that users do. Some

companies even put cameras behind people to see what they do, and they want to store that somewhere else, somewhere safe, and they want to access that, search that, analyze and correlate, and they want to play everything back exactly like it happened when they were trying to figure out what happened.

There's plenty of commercial offerings, and there are pretty good ones too, and they go from dedicated hardware, which you could put on your network and plug one cable and another cable in it, and then it will intercept everything provided they have the keys.

Then there are just software that you can install on your own hardware, and there are things which are simply like jump hosts, where you log in and then you log into your target system, and the intermediate system records everything, and then there are just programs which are running exactly on the target host,

which is being recorded. Many of them integrate with identity management, with access control, where you can for example, watch the session and interrupt it when you see that something is going wrong. And again, most of them have some sort of central storage and searching and playback, of course.

But those are all commercial offerings, so they are expensive at some time, very expensive, with some really difficult licensing, like per server or even per host, per client host, and then it's commercial software, it's usually closed source, and you can't fix it, and you can't see how it works, and you can't improve it.

So, customers were asking us for something that would come from Red Hat or from open source, just something that they could get for free, and they could take a look inside and figure out how it works and fix it or maybe improve it, and

they also want support, especially the bigger organizations. So what we have so far in open source, it's not that much. It's just script, which is the classic, but you have to work a lot if you want to have something central or secure with it.

Then there is sudoio login. It is security-oriented, it has searching, it has playback, but it's not centralized, and you can't stream it. It just writes the recording on the local host. And the closest thing that there is, is the

TTY audit in the kernel, in the audit subsystem, which is also supported by ODT, but it is security-oriented, and it can be centralized as any logs could be, but it's only for input. It doesn't record output, and from what I heard from the developers, with the current

architecture, it won't perform very well if we try to use it for output. So what we are going to do, so, the basics, the basic requirements record, what's happening, what you can see, what the user enters, what the

user executes, accesses, get it over there, the machine where it is recorded as soon as possible, and store it centrally and securely. Then you would need to correlate this with other events on the system, where you, because that's the only way to really figure out what's happening, because just the recording of the screen is not enough,

and you will need, of course, to play it back to see what the user saw, and you need to control all of that centrally. So, we had an idea to just shove it with the logs, because we already have the audit events in the logs, we have login servers which know very well how to deliver, and they have this whole infrastructure ready,

and there is a whole bunch of systems which allow you to correlate those logs, search them, and analyze, and graph, and everything. And that allows us to really save on the maintenance and on the setup costs, because we just set up logging, and we have the all the delivery,

infrastructure, and storage analytics there. And it's not just like just video recording, it's closely related to those logs, and you can correlate them easily. So,

we wanted to decide on something to use for that central storage and searching solution, and so far, it seems it's going to be Elasticsearch and Kibana. It's pretty hip, pretty popular, and our common login team is working on bringing that to our products at Redhead, and it's already in OpenShift, and

it's going to OpenStack, and we are building basically a pre-cooked log forwarding solution where you basically say install, and it all goes to one place, and it's all parsed into structures, into JSON, in Elasticsearch, and you can search it

in a more or less structural way as long, like, as far as the logs can go. So, it's a sort of a turnkey solution which is being worked on by the other part of our, of my team. So, we have the central storage. Now, we need to figure out how to control the rest, and how to log everything, and how to

make sense, especially of audit logs, because they are not very well structured. They're not ready to be ingested by Elasticsearch. We need to deliver that, we need to choose something for delivery, and we need to play everything back.

So, we also have a big team working on 3APA and SSSD at Redhead, which is basically our analog of Active Directory. And, well, it's an open source project. It's just that we work on it a lot. And, among other things, it allows you to join Windows domains, to control domains, make your own domains, have trust

relations between domains, and cache credentials on the laptops, for example, where you can log in, and then disconnect from the network, and still be able to log in. And, they have the, that's outdated.

We have this session, yeah, not outdated. We are designing this session recording control right now. We have some solution on the SSSD part already. So, for actually logging the user terminal, the input and output, we made a tool called TLog. And, we considered doing it in the kernel, but again, that would have been pretty

long process, and the audit system wasn't ready for that anyway. So, we made a user space program that is started in place of user login shell, sits between the user shell and the terminal, and records everything that passes between.

It converts it to JSON, and then logs to syslog or journal, and it also has a tool that allows you to play it back on the terminal. Pretty basic. So, to deal with audit logs, to get them to Elasticsearch, we made a tool called Oshape, and we worked together with the audit team, and we're still working.

We made that tool to convert the audit logs to JSON, which would be suitable for Elasticsearch and to XML for other tools, because we have customers who prefer XML.

We built in the schemas, which you can use to validate the output, so it's more or less strictly defined as far as we have gone so far, at least. It just logs to syslog, and that's it for now. So, for delivery to Elasticsearch, we already have rsyslog, which supports that,

in Red Hat, and you can, of course, install Fluentd or Logstash, or if you are going the OpenShifter or OpenStack route, you can use our iQ solution. So, for the actual playback for the

auditors and administrators, we are building a web UI, which will be connecting to Elasticsearch playing back from there. You should be able to see input, output, and comments, and file success, and search all those, and we are planning to build it as a reusable component, which you can perhaps build into your web UI.

We already started making it, and we are starting with Cockpit. Cockpit is a web UI for managing your servers, like particular servers. Not only just a specific server, which is running on that, where the Cockpit is running, but also others, but mostly it is just this server.

We started with that because it is simpler and easier, as you will see. Okay, so this is the basic shape of things. We get the terminal IO through Tlog and other events through ODT, convert them with Oshape, push them to any

combinations of login servers, then to Elasticsearch, and then you can use web UI, or Kibana, or something else, which can be used with Elasticsearch to view them. Okay, so I'm going to show you a little demo.

I log in into a system, which has recorded an applet for a particular user, and we will start the playback at the same time, and see how well it goes, and Elasticsearch, and I'm going to also show you how the proof-of-concept web UI works in Cockpit.

Okay, okay, I forgot this. Sorry, just a moment. I guess it worked.

Okay, I wonder if it's the same session.

Yeah, it's the same session. There is a delay between the playback and the recording, because I started it much later, so it's going to take a while to catch up.

We are working on that right now. The most difficult part is jumping to arbitrary time, because yeah, you will have to basically build keyframes like in video, because the state of the terminal depends on what has happened all the way before.

Yeah, that's going to be one of the first controls that we are doing. I'm just writing code for that right now. So it's quite easy to speed it up, to play, and to put a pause on it, but not random. Not yet. We have several ideas about that, and we can talk about that if somebody is interested.

Okay, so this is the... Just a moment. This is the locks that came from this recording. This is the O-shape locks, which are connected to JSON already.

And you can see the previous audit records here, which probably resulted in the following O-shape lines. There is the T-log stuff here. These are the recordings

of user input and output. You can see some control characters here. And this is how it looks in Kibana.

Let's see, that was session four. So here, for example, I look up the audit events with session four, which we just executed, and the T-log events with session four. And I'm looking for the VIM in them. So this is from T-log.

You can see the VIM command here. This is me trying to look for that VIM command. And more on VIM command from audit.

For example, if we search for this, exactly, we can see all the comments that were executed.

This is mixed up with some older sessions. But anyway, you can see the comments right in Elasticsearch and Kibana and search them and correlate. Now let's take a look at Cockpit.

So we can already see our session here in Cockpit, because it was also locked to journal. I can come in here and see it. It's going to take long, because I took long pauses. But I'm going to make a new one. So as soon as I start and get some output going, this session should appear on the right in Cockpit.

There it is. I can come in and see what's going on.

T-log locks the window sizes, and you can resize the player here.

But at this moment, we are not replaying the window resizes, we are just working on that along with the playback. So you can make it bigger, but we just need to pass that information from playback. Okay.

So, if you would set up T-log with just the T-log and on a typical Linux system without any control from anything else, it would work about this way.

So, for example, take a login session on a console. user authenticates to login via PAM, as usually, and it's the job of NSS to tell login that the T-log is the shell. You can do that by basically putting the shell into ATC pass WED.

Then login starts T-log, because it's the shell, and T-log gets the actual shell to start from its configuration or environment. T-log starts it under PTY and then passes everything between the PTY and the actual terminal, and locks whatever it passes to syslog or journal.

So, if you want to control with SSD, which was actually merged not long ago, it hasn't made it into any releases yet, but it's in master, and it works. You can configure SSD to tell it in configuration that session recording is enabled, and these users and these groups should be recorded.

Then when a user logs in, SSD serves the PAM requests and an assess request, and for an assess request, it returns the shell being T-log, substitutes it, and when the session is set up using PAM,

SSD adds an environment variable saying which shell to start. So, when T-log starts, it picks up that environment variable and starts the shell. That way you can have shells and users specify somewhere in an adapt directory or an inactive directory, or in free APA, and don't care about the shells, and turn on the session recording somewhere on the host,

and the user will still get the right shell. And for free APA, we have the plan to use systems similar to a CL Linux mappings, where you assign configuration to H-back rules, and in this case, it's going to be configuration for T-log.

Then the SSD on the client machine fetches both the configurations and H-back rules, matches them, sees if it's the right host, if it's the right user, and then basically proceeds to apply the configuration and passes it to T-log via environment again.

yes, T-log records, input, output, and window resizes. You can configure the notice if you'd like, and you can write a journal or to syslog or to file for testing, for example, and you can also configure the site, like, do you want more interactive, more real-time recording, or do you want to save on

the extra overhead of all the headers and that information? And yes, there's, T-log has a tool that can play back from Elasticsearch, as we just saw, or from Journal, as we just saw in cockpit, or from file, if you want to test it. So, this is

how the T-log messages are structured. They are basically chopping the stream into messages, because it's login, we have to have messages. We store input and output separately in separate fields, because we want them to remain searchable. They are coming in time at the same time,

and if you just store them in the same field, they will be interspersed, and you won't be able to search it. We preserve everything, even if it's binary, and if there is an invalid UTF-8, which cannot be encoded in JSON, then we just put it in an array of bytes, so we can extract everything. If somebody just dumped an archive on their screen,

you can extract that. We also store timing with millisecond precision in a separate field, and the window resizes. Oshape functions quite simply,

kernel supplies the net, the audit messages to auditd through netlink, and auditd then has a special daemon for plugins, like a dispatcher daemon. You can have multiple plugins. One of them could be Oshape, which would then encode it in JSON, and just log it to syslog.

And then you can use whatever login server to push it to Elasticsearch. So we are trying to keep the audit event structure. I mean, all the same names, all the same hierarchy, but we have to encode it into something more strictly defined than just plain text.

We have both the XML and JSON schema similar. They're just encoding. They're basically having the same structure. And here's, for example, a user executing PS command. This is very heavily trimmed, but

basically, that's how it could look. And one event can be logged by auditd as a series of log messages. They can span several, like there could be plenty of detail about the event, but we record everything in a single object

so that you don't have to merge them somewhere where you're searching for them. So at the moment, Cockpit UI is very basic, and we're resorting to a trick to get it off quickly. Tlog records, a special synthetic recording ID, which is unique on the host and

records it as a journal field so that then Cockpit could request all the Tlog messages and then just aggregate them by recording ID and provide this list here. This one.

And then to play the recording, we basically run Tlog play on host. It's a playback tool, and Cockpit runs it under PTY and then forwards everything it sees to the JavaScript-based terminal that is on the screen here.

And that's going to be probably the first release, and then we will have to do the the most difficult part of making the playback in the JavaScript and browser. So it will probably be some of the terminal emulators that are available, probably Xterm.js or something. And we're going to modify that to provide the timing and

playback controls and decoding and everything. So there is the feature request that we often hear where people ask for, can we just please avoid recording passwords?

So I'm not aware of a way to do that purely from user space, which we're doing. The audit subsystem has the TTY audit system detecting when there is the echo off mode in the kernel and then not logging anything.

But we can do that from user space as far as I know. So the plan is either just turn off recording input because we also record audit. It doesn't matter that much that we don't record input because we get all the underlying events. Or

there is an option which we will implement. Maybe we just can turn off the input record and then see log and enable audit, TTY audit in audit, and then we can just play that back and pick it up from the audit. We're thinking about not recording graphical sessions because

we cannot separate easily the multiple terminals that can be started there. So we just don't record that, but we don't know how to detect that. Well, we're going to do that. And in general, I think that the best idea is to record the whole graphical session screen

if you need to record a graphical session. Then, yeah, there's the charset conversion because some people still don't use UTF-8. Weird, right?

We'll need to convert that somehow, but of course there's going to be mismatches between encodings and it might produce garbage. So we will need to keep the original anyway in binary form and provide the converted text just for searching purposes so it could search for that. And if you find something, you can

reconvert the binary data to recover what was there, if necessary. And of course the playback controls, which I talked about already. So Oshape, the main challenge with Oshape is that audit log is a mess.

It's basically built in the kernel. Kernel basically formats little strings, which then puts together basically like primfs, which is supposed to be logs. And there are periodically some kind of issues between userspace and the kernel, where userspace suddenly finds out that the kernel has changed some format and they can no longer parse it. And the kernel doesn't

consider that binary API, they just say like, we are within the API, it's Netlink, so whatever, deal with it. And well, in general, audit developers do a good job in dealing with it, but still

probably nobody knows all the possible logs that the kernel can produce. There is a plan to eventually switch to some new binary protocol for audit login, but even from the most enthusiastic people, it's just not going to happen very soon.

So we are going to do with what we have, and the audit developers, they keep the dictionary of all the records and type and fields that they know can happen, but we need more information than that. We need to know the types and everything, so we're going to work with them to

improve that catalog, so we can make a more strict schema, therefore making it more useful, because you can know exactly what will happen. And again, there is the needs to be character encoding conversion, because some file names can be in weird encodings, etc. And we'll have to somehow deal with miss translations and garbage and encode that in JSON.

Okay, and for Web UI, yeah, so we are using journal as a storage right now. Basically, they're just a session recording, it's just a series of journal messages, and we need to search

that, and we need to correlate, and we need to list the sessions. And so far, with just a few sessions, even though they could be long ones, it is working, but it might become very, very slow. So, regarding that, our team is going to cockpit hackfest, which is going to happen in October,

in Berlin, in the Red Hat offices. If you'd like to come, suddenly somebody wants to come there, you need to write to the mail list and ask. And right after that, in Berlin, there's going to be

All Systems Go conference, and I'm going to go there and maybe talk about this journal thing, if it makes sense or not, or we should switch to something else. Yeah, again, the playback control. So, while we have the playback done on the host and forwarded to the terminal, it is easier to start, but it is harder to continue with implementing the

playback control, like the random positioning, for example, because we don't have the access to the terminal emulator that easily there. And yeah, to correlate with audit logs, so that you could see what was happening behind the scenes,

like you would see the screen and the audit events happening as the session goes, we need to somehow communicate the current playback position, and that might be interesting. Okay, so, if you'd like to try it, an older version of T-Log is in Fedora repos,

and you can always, of course, build it from source. The journal support in particular wasn't released yet, it's just in master, and I'm going to release it soon, I expect. So, the basic thing to try is just install T-Log as it is,

like maybe from RPM and the log to file, and see how it looks, and try to play back. There are also instructions in readme about setting up with Elasticsearch, so everything from setting up the user, and setting up your RCS log, and how to forward it, and we would be glad

to hear any issue reports or, of course, pull requests. There are actually people asking for rate control, where they want to avoid generating a ton of logs when a person just dumps dev view random or something on the screen, so there is a pull request from a student at

some government organization trying to implement that, and I think we're going to do it in autumn, this autumn. Then all shape is even easier, you just, well, you have to build it, and there's

actually been a release, it's relatively fresh, so you can just get a RPM and feed it your own audit log, it won't upload it anywhere, but it will convert it to JSON or XML as you want, as you will ask, and you can see what's the structure and how it looks and if it's of use to

and again, there are instructions for forwarding to Elasticsearch and readme. Okay, and this is the most difficult part, if somebody is really, really determined, they can get our proof of concept branch and read the upstream instructions on building

and running cockpit from source, they can then install CLog, configure it to write a journal, create a user to be recorded, log in with that user and the stuff should magically appear on the cockpit UI and the session as you just saw there, but that magic might require some work.

Okay, I'm early. Thank you. Anybody curious about anything? Sure.

Thanks for your talk. Are there any plans to implement Elasticsearch support in the web interface or at least the backing JavaScript library, so we do not have to read from journal? In cockpit. Or well, I think you said something

about releasing a separate JavaScript library for the web interface maybe? Yeah, there's going to be some component which you can use and the playback from journal is just for local recording just to get started and eventually we will have perhaps even in cockpit support for Elasticsearch, although I'm not sure if that makes sense because cockpit is mostly for

managing this one server and you would log to some other server and then to playback session recorded on this server you have to look into that server to get the session from there, but we are going to do that anyway, we'll need to playback from Elasticsearch and you can use the log play to playback from Elasticsearch right now, it's just that there is no web UI for that.

I'm just talking about more like a central solution for reading or displaying the recorded sessions for multiple servers and maybe not integrated into cockpit, building your own stuff. Yeah, we are planning for that, we just not at the moment haven't selected one place

we're going to put this, so it might be cloud forms or free API web UI or something like that and to decide that we are actually just working it around to make it embeddable, so it's easier. So absolutely that's our present target, we are trying to work hard towards it,

we got the new hire who is working the web UI and it's going to be better. I have a couple of questions, first is correctly that the same message is sent twice, one from Oddity and one from your daemon? You don't have to send them anywhere.

What you saw is that JournalD picks up its own audit messages from the kernel, you can turn that off, I think in JournalD configuration and just set up Oddity which

can then be set up to discard its own messages and just put it to the plugins. Yeah, but you have to do that manually, that's what I wanted to ask. Yeah, but we are going to implement maybe like an Ansible playbook or something for setting it up.

The problem with Elasticsearch is always when you have access to Elasticsearch you can search for everything, so you see everything. Is there any possibility that we can say not everybody is allowed to see everybody else's recordings?

That's what the viaQ project is working on in our other part, so they are doing access control based on indexes for various tenants in OpenShift cluster, so they have separate access and you can control that thing. The other question is, is it very easy to bypass simply by using change shell

and give me the bash again? I'm sorry? When you use change shell, the command line and specifying hey I want to use bash, then you disable the logging. Is it that easy? Well, if you're root then everything is easy. Yeah, but no, change shell is everybody can do that, everybody can specify his own login shell. Can they? Yes.

Let's say it's normally when you don't disable. And the last question is, does any regulatory body accept that, or have you talked with any regulatory body if that is something that can, for example, for a government or banking application that is acceptable?

Well, we have several banks like talking and asking for this and making requests, and the Australian security services are interested in the audit part, so there is some interest. And regarding the shell, you can't always do that,

change the shell. You can have the users send free IP, you can have policies on that. Okay, thanks. My question is, is there any way to make it stealth for the user that is being

recorded? Stealth? Yeah, I mean not him to be aware of it, because if I'm aware that I'm being recorded so I behave, if I'm aware that I'm not being recorded so I'll do whatever I want to. Well, if the users behave, that's already good.

Yeah, but then you will actually, if the user is clueless, you can just disable the notice, and they might not know, but if they look at the process table, they will see that. And another part which you can do, you can set up a jump host, a transparent jump host where a user logs into

one system, like a duplicate, a shadow system, where it automatically logs him in into another system, and the recording is happening on the jump host, transparently, and they won't see that. Okay, thanks. Anyone else? All right, thank you very much, thank you for coming,

and for the questions.