User Session Recording for the Enterprise
Kondrashov, Nikolai
A presentation on an effort to implement Linux User Session Recording for the Enterprise using new and existing open-source software.
 Hello, everyone. My name is Nikolai Kondrashov. I'm a software engineer at Redhead. I'm going to be talking about this project that we have of implementing user session recording in a way that is suitable for using the enterprise such as banks or hospitals or just commercial organizations. I work at the common login team, which has started not so long ago, and they're focusing on getting the logs together from across all the Redhead products and putting them in one place that is usable. I work mostly on this project, user session recording project, and I maintain free RedHouse packages as well. In my free time, I maintain the DigiMent project, which is about supporting graphics tablets on Linux, and also I play with embedded, well, in what's left of my time. So, has anybody ever used anything like user session recording before? Oh, wow. Okay. Has anybody set it up? Cool. Has anybody been recorded? Okay. Has anybody ever wished that he has recorded user sessions? Oh, nice. Okay, great. Let's see. So, the premise is basic. Some companies need to comply with government regulations. Some need to just track contractors or some visitors on their systems, and they need to know what happened to their servers if something goes wrong and who did it. So, in ideal world, people want to record everything that users do. Some companies even put cameras behind people to see what they do, and they want to store that somewhere else. Some are safe, and they want to access that, search that, analyze and correlate, and they want to play everything back exactly like it happened when they tried to figure out what happened. There's plenty of commercial offerings, and there are pretty good ones, too, and they go from dedicated hardware, which you could put on your network and plug one cable and another cable in it, and then it will intercept everything provided they have the keys. Then there are just software that you can install on your own hardware, and there are things which are simply like jump hosts where you log in and then you log into your target system and the intermediate system records everything, and then there are just programs which are running on the exactly on the target host, which is being recorded. And many of them integrate with identity management, with access control, where you can, for example, watch the session and interrupt it when you see that something is going wrong. And again, most of them have some sort of central storage and searching and playback, of course. But those are all commercial offerings, so they are expensive at some time, very expensive with some really difficult licensing, like per server or even per host, per client host. And then it's commercial softwares, usually closed source, and you can't fix it, and you can't see how it works and you can't improve it. So the customers were asking us for something that would come from Red Hat and from Open Source, just something that they could get for free, and they could take a look inside and figure out how it works and fix it or maybe improve it. And they also want support, especially the bigger organizations. So what we have so far in Open Source, it's not that much. It's just script, which is the classic, but you have to work a lot if you want to have something central or secure with it. Then there is sudoio login. It is security oriented. It has searching and has playback, but it's not centralized and you can't stream it. It just writes the recording on the local host. And the closest thing that there is is the DTY audit in the kernel, in the audit subsystem, which is also supported by audit.t, but it is security oriented and it can't be centralized as any logs could be, but it's only for input. It doesn't record the output and from what I heard from the developers with the current architecture, it won't perform very well if we try to use it for output. So what we are going to do? So the basic requirements record, what's happening, what you can see, what the user enters, what the user executes, accesses, get it off the machine where it is recorded as soon as possible and store it centrally and securely. Then you would need to correlate this with other events on the system, because that's the only way to really figure out what's happening, because just the recording of the screen is not enough. And you will need, of course, to play it back to see what the user saw and you need to control all of that centrally. So we had an idea to just shove it with the logs. Because we already have the audit events in the logs, we have login servers which know very well how to deliver and they have this whole infrastructure ready and there is a whole bunch of systems which allow you to correlate those logs, search them and analyze and graph and everything. And that allows us to really save on the maintenance and on the set up costs because we just set up logging and we have the old delivery infrastructure and storage analytics there. And it's not just like just video recording, it's closely related to those logs and you can correlate them easily. So we wanted to decide on something to use for that central storage and searching solution. And so far it seems it's going to be elastic search on Kibana. It's pretty hip, pretty popular. And our common login team is working on bringing that to our products at Red Hat. And it's already in OpenShift. And it's going to OpenStack and we are building basically a pre-cooked log forwarding solution where you basically say install and it all goes to one place and it's all parsed into structures and to JSON in elastic search and you can search it in a more or less structural way as long as far as the logs can go. So it's a sort of a turnkey solution which is being worked on by the other part of my team. So we have the central storage. Now we need to figure out how to control the rest and how to log everything and how to make sense especially of audit logs because they are not very well structured. They are not ready to be ingested by elastic search. We need to deliver that. We need to choose something for delivery and we need to play everything back. So we also have a big team working on 3AP and SSSD at Red Hat. Which is basically our analog of Active Directory. And well, it's an open source project. It's just that we work on it a lot. And among other things, it allows you to join Windows domains, to control domains, make your own domains, have trust relations between domains and cache credentials on the laptops, for example, where you can log in and then disconnect from the network and still be able to log in. And we have the... That's outdated. We have this session... Yeah, not outdated. We are designing this session recording control right now. We have some solution on the SSSD part already. So for actually logging the user terminal, the input and output, we made a tool called T-Log. And we considered doing it in the kernel, but again, that would have been a pretty long process and the audit system wasn't ready for that anyway. So we made a user space program that started in place of user login shell. It's between the user shell and the terminal and records everything that passes between. It converts it to JSON and then logs to this local journal. And it also has a tool that allows you to play it back on a terminal. Pretty basic. So to deal with audit logs, to get them to Elasticsearch, we made a tool called Oshap. And we worked together with the audit team and we're still working. We made that tool to convert the audit logs to JSON, which would be suitable for Elasticsearch and to XML for other tools because we have customers who prefer XML. We built in the schemas, which you can use to validate the output. So it's more or less strictly defined as far as we have gone so far, at least. It just logs to SSLog and that's it for now. So for delivery to Elasticsearch, we already have RSSLog, which supports that in Red Hat. And you can of course install Fluendy or LogStash. Or if you're going to open the shifter or open stack route, you can use our IEQ solution. So for the actual playback for the auditors and administrators who are building a web UI, which will be connecting to Elasticsearch playing back from there, you should be able to see input output and commons and file success and search all those. And we are planning to build it as a reusable component, which you can perhaps build into your web UI. And we already started making it and we are starting with Cockpit. Cockpit is a web UI for managing your servers, like particular servers. Not only just a specific server which is running on that, where the Cockpit is running but also others, but mostly it is just this server. We started with that because it is simpler and easier as you will see. Okay, so this is the basic shape of things. We get the terminal IEQ through CLog and other events through AuditD, convert them with OShape, push them to any combinations of login servers, then the Elasticsearch and then you can use web UI or Kibana or something else, which can be used with Elasticsearch to view them. Okay, so I am going to show you a little demo. I log in into a system which has recorded an applet for a particular user. And we will start the playback at the same time and see how well it goes and then we will take a quick look at the logs in Journal and Elasticsearch. And I am going to also show you how the proof of concept web UI works in Cockpit. Okay. I forgot this. Sorry, just a moment. I guess it worked. Okay. I wonder if it is the same session. Yeah, it is the same session. There is a delay between the playback and the recording because I started it much later. So it is going to take a while to catch up. We are working on that right now. The most difficult part is the playback. The most difficult part is jumping to arbitrary time because you will have to basically build keyframes like in video because the state of the terminal depends on what has happened all the way before. Yeah, that is going to be one of the first controls that we are doing. I am just writing code for that right now. So it is quite easy to speed it up, to play and to put a pause on it, but not random, not yet. We have several ideas about that and we can talk about that if somebody is interested. Okay. So this is the... Just a moment. This is the locks that came from this recording. This is the O-shape locks which are called as JSON already. And you can see the previous audit records here which probably resulted in the following O-shape lines. There is the T-log stuff here. This is the recordings of user input and output. You can see some control characters here. And this is how it looks in Kibana. Let's see. That was session four. So here, for example, I look up the audit events with session four which we just executed and the T-log events with session four and I am looking for the beam in them. So this is from T-log. You can see the beam command here. This is me trying to look for that beam command and more on the beam command from audit. For example, if we search for... We can see all the commands that were executed. This is mixed up with some older sessions. But anyway, you can see the commands in the last search in Kibana and search them and correlate. Now let's take a look at cockpit. So we can already see our session here in cockpit because it was also locked to journal. I can come in here and see it. It's going to take long because it took long poses. But I am going to make a new one. So as soon as I start and get some output going, this session should appear on the right in cockpit. There it is. I can come in and see what's going on. T-log locks the window sizes and you can resize the player here. But at this moment we are not replaying the window resizes. We are just working on that along with the playback. You can make it bigger but we just need to pass the information from playback. So if you would set up T-log with just the log and typical Linux system without any control from anything else, it would work about this way. So for example, take login session on a console. So user authentication to login via PAM as usually. And it's the job of NSS to tell login that T-log is the shell. You can do that by putting the shell into ATC pass to WD. Then login starts T-log because it's the shell and T-log gets the actual shell to start from configuration or environment. T-log starts under PTY and then passes everything between the PTY and the actual terminal and locks whatever it passes to Syslog or Journal. If you want to control with SSD which was actually merged not long ago, it hasn't made it into any releases yet but it's in master and it works. You can configure SSD to tell it in configuration that session recording is enabled and these users and these groups should be recorded. Then when a user logs in, SSD serves the PAM requests and SSS requests and for an SSS request it returns the shell being T-log substitutes it and when the session is set up using PAM, SSD adds an environment variable saying which shell to start. So when T-log starts, it picks up that environment variable and starts the shell. That way you can have shells and users specify somewhere in an adaptive directory or in free APA and don't care about the shells and turn on the session recording somewhere on the host and the user will still get the right shell. For free APA we have the plan to use a system similar to a CL Linux mapping where you assign configuration to HPEC rules and in this case it's going to be configuration for T-log. Then the SSD on the client machine fetches both the configurations and HPEC rules, matches them, sees if it's the right host, if it's the right user and then basically proceeds to apply the configuration and passes it to T-log where environment again. So yes, T-log records input, output and window resizes. You can configure the notice if you'd like and you can write a journal or to syslog or to file for testing for example and you can also configure the site like do you want more interactive, more real-time recording or do you want to save on the extra overhead of the old headers and meta-information. And yes, T-log has a tool that can play back from elastic searches we just saw or from journals we just saw in cockpit or from file if you want to test it. So this is how the T-log messages are structured. They are basically chopping the stream into messages because it's logging, we have to have messages. We store input and output separately in separate fields because we want them to remain searchable. They are coming in time at the same time and if you just store them in the same field they will be interspersed and you won't be able to search it. We preserve everything even if it's binary and if there is invalid UTF-8 which cannot be encoded in JSON then we just put it in a array of bytes so we can extract everything if somebody just dumped an archive on their screen you can extract that. They also store time in this millisecond precision in a separate field and window resizes. O-shape functions quite simply. Kernel supplies the net, the audit messages to audit D through netlink and audit D then has a special demon for plugins like a dispatcher demon. You can have multiple plugins. One of them could be O-shape which would then encode it in JSON and just log it to syslog and then you can use whatever login server to push it to Elastic Search. So we are trying to keep the audit event structure. I mean all the same names, all the same hierarchy but we have to encode it into something more strictly defined and just plain text. We have the both the XML and JSON schemas similar. They just encode them. They are basically having the same structure. And here is for example a user executing PS command. This is very heavily trimmed but basically that's how it could look. And one event can be logged by audit D as a series of log messages. They can spend several like there could be plenty of detail about the event but we record everything in a single object that you don't have to merge them somewhere where you are searching for them. So the moment cockpit UI is very basic and we are resorted to a trick to get it off quickly. To log rack records, special synthetic recording ID which is unique on the host and records it as a journal field so that then cockpit could request by the all the to log messages and then aggregate them by recording ID and provide this list here. This one. And then to play the recording we basically run the log play on host. It's a play back tool and cockpit runs it under PTY and forwards everything it sees to the JavaScript based terminal that is on the screen here. That's going to be probably the first release. And then we will have to do the most difficult part of making the play back in the JavaScript and browser so it will probably be some of the terminal emulators that are available, probably X-term JS or something. And we are going to modify that to provide the timing and play back controls and decoding and everything. So there is the feature request that we often hear where people ask for, can we just please avoid recording passwords? So I'm not aware of a way to do that purely from user space which we are doing. The audit subsystem has the TTY audit system detecting when there is the echo off mode in the kernel and then not logging anything but we can do that from user space as far as I know. So the plan is either just turn off recording input because we are also recording audit. It doesn't matter that much that we don't record input because we get all the underlying events. Or there is an option which we will implement. Maybe we just can turn off the input recording and enable audit to I audit and audit and then we can just play that back and pick it up from the audit. We are thinking about not recording graphical sessions because we cannot separate easily these multiple terminals that can be started there. So we just don't record that but we don't know how to detect that well. We are going to do that. And in general I think that the best idea is to record the whole graphical session screen if you need to record a graphical session. Then there is the char set conversion because some people still don't use UTF-8. We will need to convert that somehow but there will be mismatches between encodings and it might produce garbage. So we will need to keep the original anyway in binary form and provide the converted text just for search and purposes. So it could search for that. If you find something you can reconvert the binary data to recover what was there if necessary. And of course the playback controls which I talked about already. So O-shape, the main challenge with O-shape is that audit log is a mess. It is basically built in the kernel. Kernel basically formats little strings which then puts together basically like prints which is supposed to be logs and there are periodically some kind of issues between user space and the kernel where user space suddenly finds out that the kernel has changed some format and they can no longer parse it. And the kernel doesn't consider that binary API, they just say like, wow, we are within the API, it's not linked. So whatever, deal with it. And well, in general, developers do a good job of dealing with it but still probably nobody knows all the possible logs that the kernel can produce. There is a plan to eventually switch to some new binary protocol for audit login but it's even from the most enthusiastic people it's just not going to happen very soon. So we are going to do with what we have and the audit developers, they keep the dictionary folder records and type and fields that they know can happen but we need more information than that, we need to know the types and everything. So we are going to work with them to improve that catalog so we can make a more strict schema therefore making it more useful because you can know exactly what will happen. And again, there needs to be character encoding conversion because some file names can be in weird encodings, et cetera. And we will have to somehow deal with the mistranslations and garbage and encode that in JSON. Okay, and for web UI, yeah, so we are using journal as a storage right now, basically they are just a session recording, it's just a series of journal messages and we need to search that and we need to correlate and we need to list the sessions and so far with just a few sessions even though they could be long ones, it is working but it might become very, very slow. So regarding that, our team is going to cockpit hack fest which is going to happen in October in Berlin in red hat offices if you would like to come, suddenly somebody wants to come there, you need to write to the mail list and ask. And right after that in Berlin there is going to be all systems go conference and I am going to go there and maybe talk about this journal thing if it makes sense or not or we should switch to something else. Yeah, again, the playback control, so while we have the playback done on the host and forwarded to the terminal, it is easier to start but it is harder to continue with implementing the playback control like the random positioning for example because we don't have the access to the terminal emulator that is in there. And yeah, to correlate with audit logs so that you could see what was happening behind the scenes like you would see the screen and the audit events happening as the session goes, we need to somehow communicate the current playback position and that might be interesting. Okay, so if you would like to try an older version of telog is in Fedora repos and you can always of course build it from source. The journal support in particular was not released yet, it is just a master, I am going to release it soon, I expect. So the basic thing to try is just install telog as it is like maybe from RPM and the log to file and see how it looks and try to playback. There are also instructions in readme about setting up is elastic search, so everything from setting up the user and setting up your RC log and how to forward it and we would be glad to hear any issue reports or of course pull requests. There are actually people asking for rate control where they want to avoid generating a ton of logs when a person just dumps the view random or something on the screen. So there is a pull request from a student at some government organization trying to implement that and I think we are going to do it in autumn this autumn. Then O-shape is even easier, you just, well you have to build it. After it has actually been released, it is relatively fresh so you can just get a RPM and feed it to your own audit log, it won't upload it anywhere but it will convert it to JSON or XML as you will ask and you can see what is the structure and how it looks and if it is of use to you. And again there is instructions for forwarding to elastic search in readme. Okay, and this is the most difficult part. If somebody is really, really determined, they can get our proof of concept branch and read the upstream instructions on building and running cockpit from source. They can then install C log, configure it to write the journal, create a user to be recorded. Well, again with that user and the stuff should magically appear on the cockpit UI and the session as you just saw there but that magic might require some work. Okay, I am early. Thank you. Anybody curious about anything? Sure. Thanks for your talk. Are there any plans to implement elastic search support in the web interface or at least back in JavaScript library? So you do not have to read from journal? In cockpit. Or well, I think you said something about releasing a separate JavaScript library for the web interface maybe. Yeah, there is going to be some component which you can use and the playback from journal is just for local recording just to get started and eventually we will have perhaps even in cockpit support for elastic search. Although I am not sure if that makes sense because cockpit is mostly for managing this one server and you would log to some other server and then to playback session recorded on this server you have to log into that server to get the session from there. But we are going to do that anyway. We will need to playback from elastic search and you can use the log play to playback from elastic search right now. It is just that there is no web UI for that. I am just talking about more like a central solution for reading or displaying the recorded sessions for multiple servers and maybe not integrated into cockpit, building your own stuff. Yeah, we are planning for that. We just not at the moment haven't selected one place we are going to put this. So it might be cloud forms or free APA web UI or something like that and to decide that we are actually just working it around to make it embeddable. So it is easier. So absolutely that is our present target. We are trying to work hard towards it. We got the new hire who is working the web UI and it is going to be better. I have a couple of questions. First is correctly that the same message is sent twice, one from Audity, one from Audispti and one from your daemon? You don't have to send them anywhere. Yeah, but it is automatically added to your phone. What you saw is that JournalD picks up its own audit messages from the kernel. You can turn that off. I think in JournalD configuration and just set up Audity which can then be set up to discard its own messages and just put it to the plug-in. Yeah, but you have to do that manually. That is what I want to do. Yeah, but we are going to implement maybe a non-sable playbook or something for setting it up. The problem with elastic search is always when you have access to elastic search you can search for everything. You see everything. Is there any possibilities that we can say not everybody is allowed to see everybody else's recordings? That is what the Viacube project is working on in our other part. They are doing access control based on indexes for various tenants and open shift clusters. They have separate access and you can control that thing. The other question is very easy to bypass simply by using changeShell and give me the bash again. When you use changeShell, the command line, specifying, hey, I want to use bash, then you disable the logging. Is it that easy? If you are root, then everything is easy. But no, changeShell is everybody can do that. Everybody can specify its own lock and shell. Can they? Yes. Let's say it is normally when you don't disable. The last question is does any regulatory body accepted that or have you talked with any regulatory body if that is something that can, for example, for government or banking application that is that acceptable? Well, we have several banks talking and asking for this and making requests and the Australian security services are interested in the audit part. There is some interest. Regarding the shell, you can't always do that, change the shell. You can have the user's entry AP, you can have policies on that. Yeah. Okay. Thanks. My question is, is there any way to make it stealth for the user that is being recorded? Stealth? Yeah. I mean, not him to be aware of it. Because if I'm aware that I'm being recorded, so I behave. If I am aware that I'm not being recorded, so I'll do whatever I want to. Well, if the user's behave, that's already good. Yeah. Yeah. But then you will actually, if the user is clueless, you can just disable the notice and they might not know, but if they look at the process table, they will see that. And another part which you can do, you can set up a jump host, a transparent jump host where user logs into one system like a duplicate, a shadow system where it automatically logs him in into another system. And the recording is happening on the jump host transparently. And they won't see that. Okay. Thanks. Anyone else? All right. Thank you very much. Thank you for coming and for the questions.
 Hello, everyone. My name is Nikolai Kondrashov. I'm a software engineer at Redhead. I'm going  to be talking about this project that we have of implementing user session recording in  a way that is suitable for using the enterprise such as banks or hospitals or just commercial  organizations. I work at the common login team, which has started not so long ago, and  they're focusing on getting the logs together from across all the Redhead products and putting  them in one place that is usable. I work mostly on this project, user session recording project,  and I maintain free RedHouse packages as well. In my free time, I maintain the DigiMent project,  which is about supporting graphics tablets on Linux, and also I play with embedded,  well, in what's left of my time. So, has anybody ever used anything like user session recording  before? Oh, wow. Okay. Has anybody set it up? Cool. Has anybody been recorded? Okay. Has anybody  ever wished that he has recorded user sessions? Oh, nice. Okay, great. Let's see. So, the premise  is basic. Some companies need to comply with government regulations. Some need to just track  contractors or some visitors on their systems, and they need to know what happened to their  servers if something goes wrong and who did it. So, in ideal world, people want to record  everything that users do. Some companies even put cameras behind people to see what they  do, and they want to store that somewhere else. Some are safe, and they want to access  that, search that, analyze and correlate, and they want to play everything back exactly  like it happened when they tried to figure out what happened. There's plenty of commercial  offerings, and there are pretty good ones, too, and they go from dedicated hardware,  which you could put on your network and plug one cable and another cable in it, and then  it will intercept everything provided they have the keys. Then there are just software  that you can install on your own hardware, and there are things which are simply like  jump hosts where you log in and then you log into your target system and the intermediate  system records everything, and then there are just programs which are running on the  exactly on the target host, which is being recorded. And many of them integrate with  identity management, with access control, where you can, for example, watch the session  and interrupt it when you see that something is going wrong. And again, most of them have  some sort of central storage and searching and playback, of course. But those are all  commercial offerings, so they are expensive at some time, very expensive with some really  difficult licensing, like per server or even per host, per client host. And then it's  commercial softwares, usually closed source, and you can't fix it, and you can't see how  it works and you can't improve it. So the customers were asking us for something that  would come from Red Hat and from Open Source, just something that they could get for free,  and they could take a look inside and figure out how it works and fix it or maybe improve  it. And they also want support, especially the bigger organizations. So what we have  so far in Open Source, it's not that much. It's just script, which is the classic, but  you have to work a lot if you want to have something central or secure with it. Then  there is sudoio login. It is security oriented. It has searching and has playback, but it's  not centralized and you can't stream it. It just writes the recording on the local host.  And the closest thing that there is is the DTY audit in the kernel, in the audit subsystem,  which is also supported by audit.t, but it is security oriented and it can't be centralized  as any logs could be, but it's only for input. It doesn't record the output and from what  I heard from the developers with the current architecture, it won't perform very well if  we try to use it for output. So what we are going to do? So the basic requirements record,  what's happening, what you can see, what the user enters, what the user executes, accesses,  get it off the machine where it is recorded as soon as possible and store it centrally  and securely. Then you would need to correlate this with other events on the system, because  that's the only way to really figure out what's happening, because just the recording of the  screen is not enough. And you will need, of course, to play it back to see what the user  saw and you need to control all of that centrally. So we had an idea to just shove it with the  logs. Because we already have the audit events in the logs, we have login servers which know  very well how to deliver and they have this whole infrastructure ready and there is a  whole bunch of systems which allow you to correlate those logs, search them and analyze  and graph and everything. And that allows us to really save on the maintenance and on  the set up costs because we just set up logging and we have the old delivery infrastructure  and storage analytics there. And it's not just like just video recording, it's closely  related to those logs and you can correlate them easily. So we wanted to decide on something  to use for that central storage and searching solution. And so far it seems it's going  to be elastic search on Kibana. It's pretty hip, pretty popular. And our common login  team is working on bringing that to our products at Red Hat. And it's already in OpenShift.  And it's going to OpenStack and we are building basically a pre-cooked log forwarding solution  where you basically say install and it all goes to one place and it's all parsed into  structures and to JSON in elastic search and you can search it in a more or less structural  way as long as far as the logs can go. So it's a sort of a turnkey solution which is  being worked on by the other part of my team. So we have the central storage. Now we need  to figure out how to control the rest and how to log everything and how to make sense  especially of audit logs because they are not very well structured. They are not ready  to be ingested by elastic search. We need to deliver that. We need to choose something  for delivery and we need to play everything back. So we also have a big team working on  3AP and SSSD at Red Hat. Which is basically our analog of Active Directory. And well,  it's an open source project. It's just that we work on it a lot. And among other things,  it allows you to join Windows domains, to control domains, make your own domains, have  trust relations between domains and cache credentials on the laptops, for example, where  you can log in and then disconnect from the network and still be able to log in. And we  have the... That's outdated. We have this session... Yeah, not outdated. We are designing this  session recording control right now. We have some solution on the SSSD part already.  So for actually logging the user terminal, the input and output, we made a tool called  T-Log. And we considered doing it in the kernel, but again, that would have been a pretty long  process and the audit system wasn't ready for that anyway. So we made a user space program  that started in place of user login shell. It's between the user shell and the terminal  and records everything that passes between. It converts it to JSON and then logs to this  local journal. And it also has a tool that allows you to play it back on a terminal.  Pretty basic. So to deal with audit logs, to get them to Elasticsearch, we made a tool  called Oshap. And we worked together with the audit team and we're still working. We made  that tool to convert the audit logs to JSON, which would be suitable for Elasticsearch  and to XML for other tools because we have customers who prefer XML. We built in the  schemas, which you can use to validate the output. So it's more or less strictly defined  as far as we have gone so far, at least. It just logs to SSLog and that's it for now.  So for delivery to Elasticsearch, we already have RSSLog, which supports that in Red Hat.  And you can of course install Fluendy or LogStash. Or if you're going to open the shifter or  open stack route, you can use our IEQ solution. So for the actual playback for the auditors  and administrators who are building a web UI, which will be connecting to Elasticsearch  playing back from there, you should be able to see input output and commons and file success  and search all those. And we are planning to build it as a reusable component, which  you can perhaps build into your web UI. And we already started making it and we are starting  with Cockpit. Cockpit is a web UI for managing your servers, like particular servers. Not  only just a specific server which is running on that, where the Cockpit is running but  also others, but mostly it is just this server. We started with that because it is simpler  and easier as you will see. Okay, so this is the basic shape of things. We get the terminal  IEQ through CLog and other events through AuditD, convert them with OShape, push them  to any combinations of login servers, then the Elasticsearch and then you can use web  UI or Kibana or something else, which can be used with Elasticsearch to view them. Okay,  so I am going to show you a little demo. I log in into a system which has recorded an  applet for a particular user. And we will start the playback at the same time and see  how well it goes and then we will take a quick look at the logs in Journal and Elasticsearch.  And I am going to also show you how the proof of concept web UI works in Cockpit. Okay.  I forgot this. Sorry, just a moment.  I guess it worked. Okay. I wonder if it is the same session.  Yeah, it is the same session. There is a delay between the playback and the recording because  I started it much later. So it is going to take a while to catch up.  We are working on that right now. The most difficult part is the playback.  The most difficult part is jumping to arbitrary time because you will have to basically build  keyframes like in video because the state of the terminal depends on what has happened  all the way before. Yeah, that is going to be one of the first controls that we are doing.  I am just writing code for that right now. So it is quite easy to speed it up, to play  and to put a pause on it, but not random, not yet. We have several ideas about that  and we can talk about that if somebody is interested.  Okay. So this is the... Just a moment.  This is the locks that came from this recording. This is the O-shape locks which are called  as JSON already. And you can see the previous audit records here which probably resulted  in the following O-shape lines. There is the T-log stuff here. This is the  recordings of user input and output. You can see some control characters here.  And this is how it looks in Kibana. Let's see. That was session four.  So here, for example, I look up the audit events with session four which we just executed  and the T-log events with session four and I am looking for the beam in them.  So this is from T-log. You can see the beam command here.  This is me trying to look for that beam command and more on the beam command from audit.  For example, if we search for... We can see all the commands that were executed.  This is mixed up with some older sessions. But anyway, you can see the commands in the  last search in Kibana and search them and correlate.  Now let's take a look at cockpit. So we can already see our session here in cockpit because  it was also locked to journal. I can come in here and see it.  It's going to take long because it took long poses. But I am going to make a new one.  So as soon as I start and get some output going, this session should appear on the right  in cockpit. There it is.  I can come in and see what's going on.  T-log locks the window sizes and you can resize the player here.  But at this moment we are not replaying the window resizes. We are just working on that  along with the playback. You can make it bigger but we just need to  pass the information from playback. So if you would set up T-log with just the  log and typical Linux system without any control from anything else, it would work about this  way. So for example, take login session on a console.  So user authentication to login via PAM as usually.  And it's the job of NSS to tell login that T-log is the shell.  You can do that by putting the shell into ATC pass to WD.  Then login starts T-log because it's the shell and T-log gets the actual shell to start from  configuration or environment. T-log starts under PTY and then passes everything  between the PTY and the actual terminal and locks whatever it passes to Syslog or Journal.  If you want to control with SSD which was actually merged not long ago, it hasn't made  it into any releases yet but it's in master and it works.  You can configure SSD to tell it in configuration that session recording is enabled and these  users and these groups should be recorded. Then when a user logs in, SSD serves the PAM  requests and SSS requests and for an SSS request it returns the shell being T-log substitutes  it and when the session is set up using PAM, SSD adds an environment variable saying which  shell to start. So when T-log starts, it picks up that environment  variable and starts the shell. That way you can have shells and users specify  somewhere in an adaptive directory or in free APA and don't care about the shells and turn  on the session recording somewhere on the host and the user will still get the right  shell. For free APA we have the plan to use a system  similar to a CL Linux mapping where you assign configuration to HPEC rules and in this case  it's going to be configuration for T-log. Then the SSD on the client machine fetches  both the configurations and HPEC rules, matches them, sees if it's the right host, if it's  the right user and then basically proceeds to apply the configuration and passes it to  T-log where environment again. So yes, T-log records input, output and  window resizes. You can configure the notice if you'd like and you can write a journal or  to syslog or to file for testing for example and you can also configure the site like do  you want more interactive, more real-time recording or do you want to save on the extra  overhead of the old headers and meta-information. And yes, T-log has a tool that can play back  from elastic searches we just saw or from journals we just saw in cockpit or from file  if you want to test it. So this is how the T-log messages are structured.  They are basically chopping the stream into messages because it's logging, we have to  have messages. We store input and output separately in separate fields because we want them to  remain searchable. They are coming in time at the same time and if you just store them  in the same field they will be interspersed and you won't be able to search it. We preserve  everything even if it's binary and if there is invalid UTF-8 which cannot be encoded in  JSON then we just put it in a array of bytes so we can extract everything if somebody just  dumped an archive on their screen you can extract that.  They also store time in this millisecond precision in a separate field and window resizes.  O-shape functions quite simply. Kernel supplies the net, the audit messages to audit D through  netlink and audit D then has a special demon for plugins like a dispatcher demon. You can  have multiple plugins. One of them could be O-shape which would then encode it in JSON  and just log it to syslog and then you can use whatever login server to push it to Elastic  Search.  So we are trying to keep the audit event structure. I mean all the same names, all the same hierarchy  but we have to encode it into something more strictly defined and just plain text. We have  the both the XML and JSON schemas similar. They just encode them. They are basically having  the same structure. And here is for example a user executing PS command. This is very  heavily trimmed but basically that's how it could look.  And one event can be logged by audit D as a series of log messages. They can spend several  like there could be plenty of detail about the event but we record everything in a single  object that you don't have to merge them somewhere where you are searching for them.  So the moment cockpit UI is very basic and we are resorted to a trick to get it off quickly.  To log rack records, special synthetic recording ID which is unique on the host and records  it as a journal field so that then cockpit could request by the all the to log messages  and then aggregate them by recording ID and provide this list here. This one. And then  to play the recording we basically run the log play on host. It's a play back tool and  cockpit runs it under PTY and forwards everything it sees to the JavaScript based terminal that  is on the screen here. That's going to be probably the first release. And then we will  have to do the most difficult part of making the play back in the JavaScript and browser  so it will probably be some of the terminal emulators that are available, probably X-term  JS or something. And we are going to modify that to provide the timing and play back controls  and decoding and everything. So there is the feature request that we often hear where  people ask for, can we just please avoid recording passwords? So I'm not aware of a way to do  that purely from user space which we are doing. The audit subsystem has the TTY audit system  detecting when there is the echo off mode in the kernel and then not logging anything  but we can do that from user space as far as I know. So the plan is either just turn  off recording input because we are also recording audit. It doesn't matter that much that we  don't record input because we get all the underlying events. Or there is an option which  we will implement. Maybe we just can turn off the input recording and enable audit to  I audit and audit and then we can just play that back and pick it up from the audit. We  are thinking about not recording graphical sessions because we cannot separate easily  these multiple terminals that can be started there. So we just don't record that but we  don't know how to detect that well. We are going to do that. And in general I think that  the best idea is to record the whole graphical session screen if you need to record a graphical  session. Then there is the char set conversion because some people still don't use UTF-8.  We will need to convert that somehow but there will be mismatches between encodings and it  might produce garbage. So we will need to keep the original anyway in binary form and  provide the converted text just for search and purposes. So it could search for that.  If you find something you can reconvert the binary data to recover what was there if necessary.  And of course the playback controls which I talked about already. So O-shape, the main  challenge with O-shape is that audit log is a mess. It is basically built in the kernel.  Kernel basically formats little strings which then puts together basically like prints which  is supposed to be logs and there are periodically some kind of issues between user space and  the kernel where user space suddenly finds out that the kernel has changed some format  and they can no longer parse it. And the kernel doesn't consider that binary API, they just  say like, wow, we are within the API, it's not linked. So whatever, deal with it. And  well, in general, developers do a good job of dealing with it but still probably nobody  knows all the possible logs that the kernel can produce. There is a plan to eventually  switch to some new binary protocol for audit login but it's even from the most enthusiastic  people it's just not going to happen very soon. So we are going to do with what we have  and the audit developers, they keep the dictionary folder records and type and fields that they  know can happen but we need more information than that, we need to know the types and everything.  So we are going to work with them to improve that catalog so we can make a more strict  schema therefore making it more useful because you can know exactly what will happen. And  again, there needs to be character encoding conversion because some file names can be  in weird encodings, et cetera. And we will have to somehow deal with the mistranslations  and garbage and encode that in JSON. Okay, and for web UI, yeah, so we are using journal  as a storage right now, basically they are just a session recording, it's just a series  of journal messages and we need to search that and we need to correlate and we need  to list the sessions and so far with just a few sessions even though they could be long  ones, it is working but it might become very, very slow. So regarding that, our team is  going to cockpit hack fest which is going to happen in October in Berlin in red hat  offices if you would like to come, suddenly somebody wants to come there, you need to  write to the mail list and ask. And right after that in Berlin there is going to be  all systems go conference and I am going to go there and maybe talk about this journal  thing if it makes sense or not or we should switch to something else. Yeah, again, the  playback control, so while we have the playback done on the host and forwarded to the terminal,  it is easier to start but it is harder to continue with implementing the playback control like  the random positioning for example because we don't have the access to the terminal emulator  that is in there. And yeah, to correlate with audit logs so that you could see what was  happening behind the scenes like you would see the screen and the audit events happening  as the session goes, we need to somehow communicate the current playback position and that might  be interesting. Okay, so if you would like to try an older version of telog is in Fedora  repos and you can always of course build it from source. The journal support in particular  was not released yet, it is just a master, I am going to release it soon, I expect. So  the basic thing to try is just install telog as it is like maybe from RPM and the log to  file and see how it looks and try to playback. There are also instructions in readme about  setting up is elastic search, so everything from setting up the user and setting up your  RC log and how to forward it and we would be glad to hear any issue reports or of course  pull requests. There are actually people asking for rate control where they want to avoid generating  a ton of logs when a person just dumps the view random or something on the screen. So  there is a pull request from a student at some government organization trying to implement  that and I think we are going to do it in autumn this autumn. Then O-shape is even easier,  you just, well you have to build it. After it has actually been released, it is relatively  fresh so you can just get a RPM and feed it to your own audit log, it won't upload it  anywhere but it will convert it to JSON or XML as you will ask and you can see what is  the structure and how it looks and if it is of use to you. And again there is instructions  for forwarding to elastic search in readme. Okay, and this is the most difficult part.  If somebody is really, really determined, they can get our proof of concept branch and  read the upstream instructions on building and running cockpit from source. They can  then install C log, configure it to write the journal, create a user to be recorded.  Well, again with that user and the stuff should magically appear on the cockpit UI and the  session as you just saw there but that magic might require some work. Okay, I am early.  Thank you. Anybody curious about anything? Sure.  Thanks for your talk. Are there any plans to implement elastic search support in the  web interface or at least back in JavaScript library? So you do not have to read from  journal? In cockpit.  Or well, I think you said something about releasing a separate JavaScript library for  the web interface maybe. Yeah, there is going to be some component  which you can use and the playback from journal is just for local recording just to get started  and eventually we will have perhaps even in cockpit support for elastic search. Although  I am not sure if that makes sense because cockpit is mostly for managing this one server  and you would log to some other server and then to playback session recorded on this  server you have to log into that server to get the session from there. But we are going  to do that anyway. We will need to playback from elastic search and you can use the log  play to playback from elastic search right now. It is just that there is no web UI for  that. I am just talking about more like a central  solution for reading or displaying the recorded sessions for multiple servers and maybe not  integrated into cockpit, building your own stuff.  Yeah, we are planning for that. We just not at the moment haven't selected one place we  are going to put this. So it might be cloud forms or free APA web UI or something like  that and to decide that we are actually just working it around to make it embeddable. So  it is easier. So absolutely that is our present target. We are trying to work hard towards  it. We got the new hire who is working the web UI and it is going to be better.  I have a couple of questions. First is correctly that the same message is sent twice, one from  Audity, one from Audispti and one from your daemon?  You don't have to send them anywhere. Yeah, but it is automatically added to your  phone. What you saw is that JournalD picks up its own audit messages from the kernel.  You can turn that off. I think in JournalD configuration and just set up Audity which  can then be set up to discard its own messages and just put it to the plug-in.  Yeah, but you have to do that manually. That is what I want to do.  Yeah, but we are going to implement maybe a non-sable playbook or something for setting  it up.  The problem with elastic search is always when you have access to elastic search you  can search for everything. You see everything. Is there any possibilities that we can say  not everybody is allowed to see everybody else's recordings?  That is what the Viacube project is working on in our other part. They are doing access  control based on indexes for various tenants and open shift clusters. They have separate  access and you can control that thing.  The other question is very easy to bypass simply by using changeShell and give me the  bash again.  When you use changeShell, the command line, specifying, hey, I want to use bash, then  you disable the logging. Is it that easy?  If you are root, then everything is easy.  But no, changeShell is everybody can do that. Everybody can specify its own lock and shell.  Can they?  Yes.  Let's say it is normally when you don't disable. The last question is does any regulatory body  accepted that or have you talked with any regulatory body if that is something that  can, for example, for government or banking application that is that acceptable?  Well, we have several banks talking and asking for this and making requests and the Australian  security services are interested in the audit part. There is some interest. Regarding the  shell, you can't always do that, change the shell. You can have the user's entry AP,  you can have policies on that.  Yeah. Okay. Thanks.  My question is, is there any way to make it stealth for the user that is being recorded?  Stealth?  Yeah. I mean, not him to be aware of it. Because if I'm aware that I'm being recorded, so I  behave. If I am aware that I'm not being recorded, so I'll do whatever I want to.  Well, if the user's behave, that's already good. Yeah. Yeah. But then you will actually,  if the user is clueless, you can just disable the notice and they might not know, but if  they look at the process table, they will see that. And another part which you can do,  you can set up a jump host, a transparent jump host where user logs into one system  like a duplicate, a shadow system where it automatically logs him in into another system.  And the recording is happening on the jump host transparently. And they won't see that.  Okay. Thanks.  Anyone else?  All right. Thank you very much. Thank you for coming and for the questions.
froscon2017-1961-user_session_recording_for_the_enterprise_0_155.png froscon2017-1961-user_session_recording_for_the_enterprise_156_2927.png froscon2017-1961-user_session_recording_for_the_enterprise_2928_3479.png froscon2017-1961-user_session_recording_for_the_enterprise_3480_5195.png froscon2017-1961-user_session_recording_for_the_enterprise_5196_9251.png froscon2017-1961-user_session_recording_for_the_enterprise_9252_11975.png froscon2017-1961-user_session_recording_for_the_enterprise_11976_16379.png froscon2017-1961-user_session_recording_for_the_enterprise_16380_18311.png froscon2017-1961-user_session_recording_for_the_enterprise_18312_19163.png froscon2017-1961-user_session_recording_for_the_enterprise_19164_19247.png froscon2017-1961-user_session_recording_for_the_enterprise_19248_20723.png froscon2017-1961-user_session_recording_for_the_enterprise_20724_23279.png froscon2017-1961-user_session_recording_for_the_enterprise_23280_23435.png froscon2017-1961-user_session_recording_for_the_enterprise_23436_24323.png froscon2017-1961-user_session_recording_for_the_enterprise_24324_25187.png froscon2017-1961-user_session_recording_for_the_enterprise_25188_25259.png froscon2017-1961-user_session_recording_for_the_enterprise_25260_25499.png froscon2017-1961-user_session_recording_for_the_enterprise_25500_25643.png froscon2017-1961-user_session_recording_for_the_enterprise_25644_25727.png froscon2017-1961-user_session_recording_for_the_enterprise_25728_28499.png froscon2017-1961-user_session_recording_for_the_enterprise_28500_28559.png froscon2017-1961-user_session_recording_for_the_enterprise_28560_32759.png froscon2017-1961-user_session_recording_for_the_enterprise_32760_37931.png froscon2017-1961-user_session_recording_for_the_enterprise_37932_38039.png froscon2017-1961-user_session_recording_for_the_enterprise_38040_38363.png froscon2017-1961-user_session_recording_for_the_enterprise_38364_38831.png froscon2017-1961-user_session_recording_for_the_enterprise_38832_39155.png froscon2017-1961-user_session_recording_for_the_enterprise_39156_42839.png froscon2017-1961-user_session_recording_for_the_enterprise_42840_48431.png froscon2017-1961-user_session_recording_for_the_enterprise_48432_53867.png froscon2017-1961-user_session_recording_for_the_enterprise_53868_53915.png froscon2017-1961-user_session_recording_for_the_enterprise_53916_54107.png froscon2017-1961-user_session_recording_for_the_enterprise_54108_54863.png froscon2017-1961-user_session_recording_for_the_enterprise_54864_54911.png froscon2017-1961-user_session_recording_for_the_enterprise_54912_55679.png froscon2017-1961-user_session_recording_for_the_enterprise_55680_56399.png froscon2017-1961-user_session_recording_for_the_enterprise_56400_56579.png froscon2017-1961-user_session_recording_for_the_enterprise_56580_57863.png froscon2017-1961-user_session_recording_for_the_enterprise_57864_59027.png froscon2017-1961-user_session_recording_for_the_enterprise_59028_60095.png froscon2017-1961-user_session_recording_for_the_enterprise_60096_60599.png froscon2017-1961-user_session_recording_for_the_enterprise_60600_60863.png froscon2017-1961-user_session_recording_for_the_enterprise_60864_61415.png froscon2017-1961-user_session_recording_for_the_enterprise_61416_61700.png
Hello, everyone. My name is Nikolai Kondrashov. I'm a software engineer at Redhead. I'm going to be talking about this project that we have of implementing user session recording in a way that is suitable for using the enterprise such as banks or hospitals or just commercial organizations. I work at the common login team, which has started not so long ago, and they're focusing on getting the logs together from across all the Redhead products and putting them in one place that is usable. I work mostly on this project, user session recording project, and I maintain free RedHouse packages as well. In my free time, I maintain the DigiMent project, which is about supporting graphics tablets on Linux, and also I play with embedded, well, in what's left of my time. So, has anybody ever used anything like user session recording before? Oh, wow. Okay. Has anybody set it up? Cool. Has anybody been recorded? Okay. Has anybody ever wished that he has recorded user sessions? Oh, nice. Okay, great. Let's see. So, the premise is basic. Some companies need to comply with government regulations. Some need to just track contractors or some visitors on their systems, and they need to know what happened to their servers if something goes wrong and who did it. So, in ideal world, people want to record everything that users do. Some companies even put cameras behind people to see what they do, and they want to store that somewhere else. Some are safe, and they want to access that, search that, analyze and correlate, and they want to play everything back exactly like it happened when they tried to figure out what happened. There's plenty of commercial offerings, and there are pretty good ones, too, and they go from dedicated hardware, which you could put on your network and plug one cable and another cable in it, and then it will intercept everything provided they have the keys. Then there are just software that you can install on your own hardware, and there are things which are simply like jump hosts where you log in and then you log into your target system and the intermediate system records everything, and then there are just programs which are running on the exactly on the target host, which is being recorded. And many of them integrate with identity management, with access control, where you can, for example, watch the session and interrupt it when you see that something is going wrong. And again, most of them have some sort of central storage and searching and playback, of course. But those are all commercial offerings, so they are expensive at some time, very expensive with some really difficult licensing, like per server or even per host, per client host. And then it's commercial softwares, usually closed source, and you can't fix it, and you can't see how it works and you can't improve it. So the customers were asking us for something that would come from Red Hat and from Open Source, just something that they could get for free, and they could take a look inside and figure out how it works and fix it or maybe improve it. And they also want support, especially the bigger organizations. So what we have so far in Open Source, it's not that much. It's just script, which is the classic, but you have to work a lot if you want to have something central or secure with it. Then there is sudoio login. It is security oriented. It has searching and has playback, but it's not centralized and you can't stream it. It just writes the recording on the local host. And the closest thing that there is is the DTY audit in the kernel, in the audit subsystem, which is also supported by audit.t, but it is security oriented and it can't be centralized as any logs could be, but it's only for input. It doesn't record the output and from what I heard from the developers with the current architecture, it won't perform very well if we try to use it for output. So what we are going to do? So the basic requirements record, what's happening, what you can see, what the user enters, what the user executes, accesses, get it off the machine where it is recorded as soon as possible and store it centrally and securely. Then you would need to correlate this with other events on the system, because that's the only way to really figure out what's happening, because just the recording of the screen is not enough. And you will need, of course, to play it back to see what the user saw and you need to control all of that centrally. So we had an idea to just shove it with the logs. Because we already have the audit events in the logs, we have login servers which know very well how to deliver and they have this whole infrastructure ready and there is a whole bunch of systems which allow you to correlate those logs, search them and analyze and graph and everything. And that allows us to really save on the maintenance and on the set up costs because we just set up logging and we have the old delivery infrastructure and storage analytics there. And it's not just like just video recording, it's closely related to those logs and you can correlate them easily. So we wanted to decide on something to use for that central storage and searching solution. And so far it seems it's going to be elastic search on Kibana. It's pretty hip, pretty popular. And our common login team is working on bringing that to our products at Red Hat. And it's already in OpenShift. And it's going to OpenStack and we are building basically a pre-cooked log forwarding solution where you basically say install and it all goes to one place and it's all parsed into structures and to JSON in elastic search and you can search it in a more or less structural way as long as far as the logs can go. So it's a sort of a turnkey solution which is being worked on by the other part of my team. So we have the central storage. Now we need to figure out how to control the rest and how to log everything and how to make sense especially of audit logs because they are not very well structured. They are not ready to be ingested by elastic search. We need to deliver that. We need to choose something for delivery and we need to play everything back. So we also have a big team working on 3AP and SSSD at Red Hat. Which is basically our analog of Active Directory. And well, it's an open source project. It's just that we work on it a lot. And among other things, it allows you to join Windows domains, to control domains, make your own domains, have trust relations between domains and cache credentials on the laptops, for example, where you can log in and then disconnect from the network and still be able to log in. And we have the... That's outdated. We have this session... Yeah, not outdated. We are designing this session recording control right now. We have some solution on the SSSD part already. So for actually logging the user terminal, the input and output, we made a tool called T-Log. And we considered doing it in the kernel, but again, that would have been a pretty long process and the audit system wasn't ready for that anyway. So we made a user space program that started in place of user login shell. It's between the user shell and the terminal and records everything that passes between. It converts it to JSON and then logs to this local journal. And it also has a tool that allows you to play it back on a terminal. Pretty basic. So to deal with audit logs, to get them to Elasticsearch, we made a tool called Oshap. And we worked together with the audit team and we're still working. We made that tool to convert the audit logs to JSON, which would be suitable for Elasticsearch and to XML for other tools because we have customers who prefer XML. We built in the schemas, which you can use to validate the output. So it's more or less strictly defined as far as we have gone so far, at least. It just logs to SSLog and that's it for now. So for delivery to Elasticsearch, we already have RSSLog, which supports that in Red Hat. And you can of course install Fluendy or LogStash. Or if you're going to open the shifter or open stack route, you can use our IEQ solution. So for the actual playback for the auditors and administrators who are building a web UI, which will be connecting to Elasticsearch playing back from there, you should be able to see input output and commons and file success and search all those. And we are planning to build it as a reusable component, which you can perhaps build into your web UI. And we already started making it and we are starting with Cockpit. Cockpit is a web UI for managing your servers, like particular servers. Not only just a specific server which is running on that, where the Cockpit is running but also others, but mostly it is just this server. We started with that because it is simpler and easier as you will see. Okay, so this is the basic shape of things. We get the terminal IEQ through CLog and other events through AuditD, convert them with OShape, push them to any combinations of login servers, then the Elasticsearch and then you can use web UI or Kibana or something else, which can be used with Elasticsearch to view them. Okay, so I am going to show you a little demo. I log in into a system which has recorded an applet for a particular user. And we will start the playback at the same time and see how well it goes and then we will take a quick look at the logs in Journal and Elasticsearch. And I am going to also show you how the proof of concept web UI works in Cockpit. Okay. I forgot this. Sorry, just a moment. I guess it worked. Okay. I wonder if it is the same session. Yeah, it is the same session. There is a delay between the playback and the recording because I started it much later. So it is going to take a while to catch up. We are working on that right now. The most difficult part is the playback. The most difficult part is jumping to arbitrary time because you will have to basically build keyframes like in video because the state of the terminal depends on what has happened all the way before. Yeah, that is going to be one of the first controls that we are doing. I am just writing code for that right now. So it is quite easy to speed it up, to play and to put a pause on it, but not random, not yet. We have several ideas about that and we can talk about that if somebody is interested. Okay. So this is the... Just a moment. This is the locks that came from this recording. This is the O-shape locks which are called as JSON already. And you can see the previous audit records here which probably resulted in the following O-shape lines. There is the T-log stuff here. This is the recordings of user input and output. You can see some control characters here. And this is how it looks in Kibana. Let's see. That was session four. So here, for example, I look up the audit events with session four which we just executed and the T-log events with session four and I am looking for the beam in them. So this is from T-log. You can see the beam command here. This is me trying to look for that beam command and more on the beam command from audit. For example, if we search for... We can see all the commands that were executed. This is mixed up with some older sessions. But anyway, you can see the commands in the last search in Kibana and search them and correlate. Now let's take a look at cockpit. So we can already see our session here in cockpit because it was also locked to journal. I can come in here and see it. It's going to take long because it took long poses. But I am going to make a new one. So as soon as I start and get some output going, this session should appear on the right in cockpit. There it is. I can come in and see what's going on. T-log locks the window sizes and you can resize the player here. But at this moment we are not replaying the window resizes. We are just working on that along with the playback. You can make it bigger but we just need to pass the information from playback. So if you would set up T-log with just the log and typical Linux system without any control from anything else, it would work about this way. So for example, take login session on a console. So user authentication to login via PAM as usually. And it's the job of NSS to tell login that T-log is the shell. You can do that by putting the shell into ATC pass to WD. Then login starts T-log because it's the shell and T-log gets the actual shell to start from configuration or environment. T-log starts under PTY and then passes everything between the PTY and the actual terminal and locks whatever it passes to Syslog or Journal. If you want to control with SSD which was actually merged not long ago, it hasn't made it into any releases yet but it's in master and it works. You can configure SSD to tell it in configuration that session recording is enabled and these users and these groups should be recorded. Then when a user logs in, SSD serves the PAM requests and SSS requests and for an SSS request it returns the shell being T-log substitutes it and when the session is set up using PAM, SSD adds an environment variable saying which shell to start. So when T-log starts, it picks up that environment variable and starts the shell. That way you can have shells and users specify somewhere in an adaptive directory or in free APA and don't care about the shells and turn on the session recording somewhere on the host and the user will still get the right shell. For free APA we have the plan to use a system similar to a CL Linux mapping where you assign configuration to HPEC rules and in this case it's going to be configuration for T-log. Then the SSD on the client machine fetches both the configurations and HPEC rules, matches them, sees if it's the right host, if it's the right user and then basically proceeds to apply the configuration and passes it to T-log where environment again. So yes, T-log records input, output and window resizes. You can configure the notice if you'd like and you can write a journal or to syslog or to file for testing for example and you can also configure the site like do you want more interactive, more real-time recording or do you want to save on the extra overhead of the old headers and meta-information. And yes, T-log has a tool that can play back from elastic searches we just saw or from journals we just saw in cockpit or from file if you want to test it. So this is how the T-log messages are structured. They are basically chopping the stream into messages because it's logging, we have to have messages. We store input and output separately in separate fields because we want them to remain searchable. They are coming in time at the same time and if you just store them in the same field they will be interspersed and you won't be able to search it. We preserve everything even if it's binary and if there is invalid UTF-8 which cannot be encoded in JSON then we just put it in a array of bytes so we can extract everything if somebody just dumped an archive on their screen you can extract that. They also store time in this millisecond precision in a separate field and window resizes. O-shape functions quite simply. Kernel supplies the net, the audit messages to audit D through netlink and audit D then has a special demon for plugins like a dispatcher demon. You can have multiple plugins. One of them could be O-shape which would then encode it in JSON and just log it to syslog and then you can use whatever login server to push it to Elastic Search. So we are trying to keep the audit event structure. I mean all the same names, all the same hierarchy but we have to encode it into something more strictly defined and just plain text. We have the both the XML and JSON schemas similar. They just encode them. They are basically having the same structure. And here is for example a user executing PS command. This is very heavily trimmed but basically that's how it could look. And one event can be logged by audit D as a series of log messages. They can spend several like there could be plenty of detail about the event but we record everything in a single object that you don't have to merge them somewhere where you are searching for them. So the moment cockpit UI is very basic and we are resorted to a trick to get it off quickly. To log rack records, special synthetic recording ID which is unique on the host and records it as a journal field so that then cockpit could request by the all the to log messages and then aggregate them by recording ID and provide this list here. This one. And then to play the recording we basically run the log play on host. It's a play back tool and cockpit runs it under PTY and forwards everything it sees to the JavaScript based terminal that is on the screen here. That's going to be probably the first release. And then we will have to do the most difficult part of making the play back in the JavaScript and browser so it will probably be some of the terminal emulators that are available, probably X-term JS or something. And we are going to modify that to provide the timing and play back controls and decoding and everything. So there is the feature request that we often hear where people ask for, can we just please avoid recording passwords? So I'm not aware of a way to do that purely from user space which we are doing. The audit subsystem has the TTY audit system detecting when there is the echo off mode in the kernel and then not logging anything but we can do that from user space as far as I know. So the plan is either just turn off recording input because we are also recording audit. It doesn't matter that much that we don't record input because we get all the underlying events. Or there is an option which we will implement. Maybe we just can turn off the input recording and enable audit to I audit and audit and then we can just play that back and pick it up from the audit. We are thinking about not recording graphical sessions because we cannot separate easily these multiple terminals that can be started there. So we just don't record that but we don't know how to detect that well. We are going to do that. And in general I think that the best idea is to record the whole graphical session screen if you need to record a graphical session. Then there is the char set conversion because some people still don't use UTF-8. We will need to convert that somehow but there will be mismatches between encodings and it might produce garbage. So we will need to keep the original anyway in binary form and provide the converted text just for search and purposes. So it could search for that. If you find something you can reconvert the binary data to recover what was there if necessary. And of course the playback controls which I talked about already. So O-shape, the main challenge with O-shape is that audit log is a mess. It is basically built in the kernel. Kernel basically formats little strings which then puts together basically like prints which is supposed to be logs and there are periodically some kind of issues between user space and the kernel where user space suddenly finds out that the kernel has changed some format and they can no longer parse it. And the kernel doesn't consider that binary API, they just say like, wow, we are within the API, it's not linked. So whatever, deal with it. And well, in general, developers do a good job of dealing with it but still probably nobody knows all the possible logs that the kernel can produce. There is a plan to eventually switch to some new binary protocol for audit login but it's even from the most enthusiastic people it's just not going to happen very soon. So we are going to do with what we have and the audit developers, they keep the dictionary folder records and type and fields that they know can happen but we need more information than that, we need to know the types and everything. So we are going to work with them to improve that catalog so we can make a more strict schema therefore making it more useful because you can know exactly what will happen. And again, there needs to be character encoding conversion because some file names can be in weird encodings, et cetera. And we will have to somehow deal with the mistranslations and garbage and encode that in JSON. Okay, and for web UI, yeah, so we are using journal as a storage right now, basically they are just a session recording, it's just a series of journal messages and we need to search that and we need to correlate and we need to list the sessions and so far with just a few sessions even though they could be long ones, it is working but it might become very, very slow. So regarding that, our team is going to cockpit hack fest which is going to happen in October in Berlin in red hat offices if you would like to come, suddenly somebody wants to come there, you need to write to the mail list and ask. And right after that in Berlin there is going to be all systems go conference and I am going to go there and maybe talk about this journal thing if it makes sense or not or we should switch to something else. Yeah, again, the playback control, so while we have the playback done on the host and forwarded to the terminal, it is easier to start but it is harder to continue with implementing the playback control like the random positioning for example because we don't have the access to the terminal emulator that is in there. And yeah, to correlate with audit logs so that you could see what was happening behind the scenes like you would see the screen and the audit events happening as the session goes, we need to somehow communicate the current playback position and that might be interesting. Okay, so if you would like to try an older version of telog is in Fedora repos and you can always of course build it from source. The journal support in particular was not released yet, it is just a master, I am going to release it soon, I expect. So the basic thing to try is just install telog as it is like maybe from RPM and the log to file and see how it looks and try to playback. There are also instructions in readme about setting up is elastic search, so everything from setting up the user and setting up your RC log and how to forward it and we would be glad to hear any issue reports or of course pull requests. There are actually people asking for rate control where they want to avoid generating a ton of logs when a person just dumps the view random or something on the screen. So there is a pull request from a student at some government organization trying to implement that and I think we are going to do it in autumn this autumn. Then O-shape is even easier, you just, well you have to build it. After it has actually been released, it is relatively fresh so you can just get a RPM and feed it to your own audit log, it won't upload it anywhere but it will convert it to JSON or XML as you will ask and you can see what is the structure and how it looks and if it is of use to you. And again there is instructions for forwarding to elastic search in readme. Okay, and this is the most difficult part. If somebody is really, really determined, they can get our proof of concept branch and read the upstream instructions on building and running cockpit from source. They can then install C log, configure it to write the journal, create a user to be recorded. Well, again with that user and the stuff should magically appear on the cockpit UI and the session as you just saw there but that magic might require some work. Okay, I am early. Thank you. Anybody curious about anything? Sure. Thanks for your talk. Are there any plans to implement elastic search support in the web interface or at least back in JavaScript library? So you do not have to read from journal? In cockpit. Or well, I think you said something about releasing a separate JavaScript library for the web interface maybe. Yeah, there is going to be some component which you can use and the playback from journal is just for local recording just to get started and eventually we will have perhaps even in cockpit support for elastic search. Although I am not sure if that makes sense because cockpit is mostly for managing this one server and you would log to some other server and then to playback session recorded on this server you have to log into that server to get the session from there. But we are going to do that anyway. We will need to playback from elastic search and you can use the log play to playback from elastic search right now. It is just that there is no web UI for that. I am just talking about more like a central solution for reading or displaying the recorded sessions for multiple servers and maybe not integrated into cockpit, building your own stuff. Yeah, we are planning for that. We just not at the moment haven't selected one place we are going to put this. So it might be cloud forms or free APA web UI or something like that and to decide that we are actually just working it around to make it embeddable. So it is easier. So absolutely that is our present target. We are trying to work hard towards it. We got the new hire who is working the web UI and it is going to be better. I have a couple of questions. First is correctly that the same message is sent twice, one from Audity, one from Audispti and one from your daemon? You don't have to send them anywhere. Yeah, but it is automatically added to your phone. What you saw is that JournalD picks up its own audit messages from the kernel. You can turn that off. I think in JournalD configuration and just set up Audity which can then be set up to discard its own messages and just put it to the plug-in. Yeah, but you have to do that manually. That is what I want to do. Yeah, but we are going to implement maybe a non-sable playbook or something for setting it up. The problem with elastic search is always when you have access to elastic search you can search for everything. You see everything. Is there any possibilities that we can say not everybody is allowed to see everybody else's recordings? That is what the Viacube project is working on in our other part. They are doing access control based on indexes for various tenants and open shift clusters. They have separate access and you can control that thing. The other question is very easy to bypass simply by using changeShell and give me the bash again. When you use changeShell, the command line, specifying, hey, I want to use bash, then you disable the logging. Is it that easy? If you are root, then everything is easy. But no, changeShell is everybody can do that. Everybody can specify its own lock and shell. Can they? Yes. Let's say it is normally when you don't disable. The last question is does any regulatory body accepted that or have you talked with any regulatory body if that is something that can, for example, for government or banking application that is that acceptable? Well, we have several banks talking and asking for this and making requests and the Australian security services are interested in the audit part. There is some interest. Regarding the shell, you can't always do that, change the shell. You can have the user's entry AP, you can have policies on that. Yeah. Okay. Thanks. My question is, is there any way to make it stealth for the user that is being recorded? Stealth? Yeah. I mean, not him to be aware of it. Because if I'm aware that I'm being recorded, so I behave. If I am aware that I'm not being recorded, so I'll do whatever I want to. Well, if the user's behave, that's already good. Yeah. Yeah. But then you will actually, if the user is clueless, you can just disable the notice and they might not know, but if they look at the process table, they will see that. And another part which you can do, you can set up a jump host, a transparent jump host where user logs into one system like a duplicate, a shadow system where it automatically logs him in into another system. And the recording is happening on the jump host transparently. And they won't see that. Okay. Thanks. Anyone else? All right. Thank you very much. Thank you for coming and for the questions.
is about supporting graphics tablets on Linux, government organization trying to implement that coming and for the questions.