SPICE: New ways to remote desktops with GStreamer integration
Toso, Victor
In this presentation we will be talking about SPICE project and its goal to provide a complete open source solution for remote access to virtual machines. It will be described how SPICE works and the recent improvements over video streaming.
 Hello everyone, my name is Victor Tozo and I'm here to talk about spice. About me a little bit, I was born and raised in Brazil, sunny Brazil, and I came all the way to Czech Republic to work on this project for Red Hat. It's been almost three years now and I'm truly happy to be here to talk with spice with you and thank you for your interest in being here. So first of all, how many of you know something about spice? Okay, two people, that's fine. And about YouTube, do you use it frequently or no sometimes? Yeah? Okay. So, yeah, this couple of presentations is not going to go deep in the implementation on how spice works and, oh, or really low level, it's just like a high overview, so it's fine for everybody, I hope. I don't plan to take all the time that we have here. This presentation was planned to be like 20 minutes, 30 minutes, so feel free to raise your hands anytime to ask questions as they come. So, first of all, what is spice? So spice is a way to, similar, connect to some remote device, some remote machine. It could be a bare metal machine. It could be a virtual machine. Our main focus usually is virtual machines. It was designed for that, but there's several ways to connect to bare metal as well. So in these two images, quite small, but you have from Windows and from Linux, you are connecting to two different VMs there, from Windows to different Windows with spreadsheet or Linux with GNOME running, here from Linux, connecting to different Windows. And the main point of spice is being able to use this virtual machine, similar, right, like it was your real machine right there. So you can play video, share files, listen music, yeah, Redirector, USB. So that's our main goal. Now, to talk about spice, it's important to divide it in four different components, which are the client, which is the spice application that connects to this remote machine. The server, which is the library user to expose this machine, right, in a way that the client can connect. The guest, which is the machine itself. And the protocol, which is how we define all those components to communicate with each other. So the spice protocol will define a set of channels, a set of interfaces that the server will then expose to the client. So the server usually connects to this virtual machine and takes all this information and basically provides this interfaces to the client. Now we have, for instance, the display channel, which is basically what you see, the drawing. The input channel could be the keyboard, the mouse, so you can interact with it. Cursor channel, which is basically to send the shape position of the mouse and the position itself. Sound channel, actually there are two channels, the playback channel and the record channel, so you can do input and output. And USB channels that you can, for instance, redirect flash drives, webcams and any other device. In this presentation, we are mainly focused on the display channel to understand how it works and the optimizations that we did the last years or so. But if you have any other questions in regard to other channels and how it works, I'm happy to take them. So spice display, how it works. What does the server do here is that it takes all the images from the guest and has a way to send it to the client. So the client will just take the image and server is the one that will send it to the client. Now how usually it works. In a bare metal machine, you just want to, you have your application, you want to do some drowning and some rendering. So usually it asks for the driver, the video driver, to do the rendering for you. But when you are talking about remote, there is no real need to do the rendering on the remote machine. So what happens is that the video driver will provide these images to the spice server, which then will send it to the client. So basically it's like that, you get any update that you have, it will be sent to the client. Now we start to think about some optimizations there. So if you have just an update, do you want to send it everything? So this example is like, yeah, I'm sending that image plus the whole background, which is not really bandwidth resource smart, right? We just need that part, that position that is being updated to be sent. So besides that, we also can do some compression on the image itself. So that's what spice usually do in the display channel. It just take what is being handed in the guest, maybe compress that image and then send it to the client. And then I do have a small demo about it. So here I have a remote viewer, which is a client, a spice GTK client and also VNC client. I'm connecting to a machine here locally and I'm passing this argument in the, oh, you cannot see it. Can you? Yeah, that's better. Yeah. So GTK debug update, so what does it do? Our client is a GTK-based client. So with this argument, we can just say, hey, every time that you update something, give me some information about it. So you see it's red. Yeah, you see in the top right, every second, there is an update in the screen. So basically that's what happened in the guest updates that provides this information to the server. It sends just that part. So it's very good bandwidth wise and you can see if I move my mouse here, every time that there is update, there will be something. Yeah. Because it's client side, the, the, yes, so the question was why there is no drowning for the mouse. So you see that we are updating the guest there, but the mouse is not. So what happens is that the spice tries to, this is a smart way to avoid showing the delay that happens between the client and the guest. So we just get the position and through the cursor shape, you also know the shape of the mouse and we do the drowning on the client side. So this is in top of everything. This is like the widget itself being updated, but the mouse is in top of that. So that's the reason. You see that if I, you see that the mouse does not change here because it's like the client side of mouse. So good question. So that's generally how it works, but now, yeah, I had a video here. You see a lot of dates here. So my intention was to show you a video being playing the guest. They don't have internet. Just a second. So you can foresee what will happen when the video is being played. What will happen is a lot of updates in that region. So yeah, and then it becomes a little bit hard to optimize everything because it's a lot of updates being done. It's just raw image at this point. So what can we do about it? So basically, if we have a lot of updates in the same region on the display on the guest, we can just convert it to a stream, which is made for this, basically. So it does have a support for several years, but that's not really bandwidth wise, good, or it's not really something new or useful for different kind of type of clients. So with Jstreamer, we just got support for VP8, VP9 as well, H264, and it becomes very easy to expand that to new features, to new video codecs. But Jstreamer, so do you guys know what Jstreamer is? There's anyone? Yeah? Okay, cool. So just a basic overview. Jstreamer is a framework for multimedia applications. And if there is two takeaways from Jstreamer, it's a pipeline based, which means that it's very flexible. So if you want to build a pipeline to encode a video stream, which is our case, after you get it working, we just need to move something, replace it, and then we can get a totally different video stream. And also, it's very easy to stand because of that, because if you just need something, you plug in, for instance, you don't need to reinvent the wheel and write everything else, just that plug-in that you need. Yeah, a basic run for Jstreamer is this Jstreamer launch. So if you provide this, I can show you quickly as well. Like this. Yeah, it's very cool that you can do a lot with it already. Like there is this other pattern as well. And this pattern is called Snow, and it's very nice to know about it because it's like analog TV in the back old days. And why it's so interesting is because this is a nightmare for a video codec, right? So it means that it's like the worst case for trying to encode a stream. So that's something really nice to always use for testing and so on. So how it works with us, basically, it's about the same thing. We get all the images, but in the host, in the spy server, and then we use a pipeline, Jstreamer pipeline to convert those images in a video stream. Yeah, so basically the three main plugins that we use is this app source, app sync, and video encoder. That means that we just need to change video encoder to change it to different stream types, and then we can send it to the client. In the client, basically, it's just the opposite. We get the video stream, and then we use the video decoder to get the image or several images, and then we can just draw it on the client. Yeah, that was how we were doing, but in Jstreamer, actually, they have for this kind of applications, they have generic plugins called PlayBeam, for instance, that do everything. So you just need to provide them some input, and they will do the decoding. You don't need to deal with video decoder, for instance. You need to find it automatically and do it almost by itself. So that's something new in our latest release on the client. And it's very nice also because it can, without effort, support video accelerated, so hardware accelerated for decoding. You just need the plugins and the driver capable of that, and of course the hardware itself. So, streaming with Spice, who benefits with it? So, yeah, as I mentioned before, we just had MAMJPEG, and that's not much useful because if you think about, like, we do have web clients, for instance, and there is a lot of huge support for H264, VP8, VP9, and they usually are working in the back end to do hardware decode itself as well. Other thing, clients like Raspere PE or these small low end machines also usually do have some sort of hardware decoding. So if we can just send them the right video stream, they will have a better interface with Spice, right? It will be more smooth, the usage and so on. Of course, the CPU usage will go down a bit because of that. So this presentation, before, I did with a friend of mine, and we had two desktops, and we did a whole desktop streaming, connected to his desktop and so on. But I was not really in the mood to come from track republic here with two desktops, two desktops, so I just recorded. Let's see, yeah, I can do it with... Yeah, the user... The common... The common task case usually for Spice is Tux, right? Tux rays. So this is streaming, H264, the whole desktop. Yeah, it works quite well, and it's using a few new features from upstream that just got into Spice protocol. No, Virgil is here for quite some time, but Virgil only works locally. Yeah, it works. It works quite well. But this is actually guest streaming using PCI-PAS True, so the card goes to the guest, we get that buffer, use it for streaming. And yeah, that in Spice protocol just got accepted, and hopefully in the near future we have another component that will do it on the guest that might do for Intel, NVIDIA, and maybe others. Yeah, so for the future, we do have a lot of work. We start from below, actually, which is a lot of great importance. So Spice remotely works under TCP, and for real-time video streaming, it's not really good because of the X on the protocol. So we are really considering going for a new type of channel that will use RTP, top of UDP. So we hope to gain some reduced latency and so on. We do need some way to... We have those things implemented, but they don't work quite well. A way to do dynamic changes in the stream itself. So for instance, you are trying to stream a 4K display, which is a lot of data, even for the coding, sometimes even hardware decoding. So if it's not working well for the client, it's not good at all. You need to send some information to the server that is dropping a lot of frames, for instance, and the server should use it as input to change it to a lower quality or something like that. So we are just in this stage of getting everything in some good... Like working well, and then we are going to work on those things. Better handling for hardware encoding and decoding with JStreamer. So as I said, now in the client side, we just got this playbin, which does almost everything for us. So if you have VAPI plugins and an Intel-based system, it just works. But if you have a different processor, maybe any video card, it's something that needs to be developed even in JStreamer itself. Like new plugins or something that needs to get more mainstream. And hardware encoding, it's long run. It will take some time because it's something that we are not working on right now. We are working with different perspectives for now. So this will take at least another year to be in some interesting shape. Yeah, and that's it. I think a special thanks for CodeWeaver guys, Francois and Jeremy, because they were the ones that actually brought this JStreamer code to Spice. Check them, the site of them. They do some interesting stuff. And Pavel Grunt, which was my friend, and we did this presentation before. He is my friend, and we did the presentation before, and part of these slides is thanks to him. And yeah, that's it. Do you have any questions? Do you want to? I have a question. There are alternatives. There's on three, there's another, no machine, and there are alternatives for remote testing. Why do they choose Spice? It depends on your use case. So it depends on what you're comparing as well. There are some presentations that compare all of them. All of them have some benefits and some drawbacks. So Spice is really cool. There are several, well, I'm biased, right? I work for Spice, so I say, hey, team Spice. But there are pros and cons in every aspect. Why are you going to choose this? I think you have to consider the possibilities and what they provided to you. So I don't know if you want to compare. For instance, I did not try any proprietary version so far. I don't know, for instance, there is a few. They do very well in hardware encoding, hardware decoding. They have, there are companies, huge companies just based for this. So they will provide better solutions but are proprietary while we are open for software. So yeah, it depends. If you want to do, for instance, 4K rendering on a tablet, Spice will not be for you now. I'm following very well this USB configuration. So you can, like, start with the device and it goes to the device. Yeah. Yeah, basically, if, yes. Yeah, that was not a question. It was, he was saying about how Spice works well with USB, for instance. Yeah, I fixed some bugs on USB, for instance. There is this guy, he was redirecting his webcam across the ocean. So from, I don't know, I think, shock republic to North America. And yeah, it was a huge leak, a lot of memory being used. And, but yeah, in general, you can just do it like that. You just redirect your webcam, connect to Skype if you want, or other things. We have shared folders. We have this audio synchronization with video. I don't know, we have a few features that I think it's nice. Yeah. That's correct. It depends on your distro. In Fedora, for instance, we provided its building, it's built with it. So when you download the package, it usually comes together. It also works for Windows as well. So you have Windows, you need the USB DK, it's a different thing. And then you can share your USB devices from Windows client to any guest. Yeah. So the question was with the Spice and JStreamer, if we do have a possibility to not stream the whole desktop, it actually, yes, it depends on configuration. So we have three types of configuration in the host. So there are actually several solutions for streaming. We can do a guest side streaming. So we have someone in the guest that will provide this from maybe some device in the guest that has this frame buffer and maybe even encoded it so we can just use it. And we have the host side, which is this. And the host side streaming, the protocol, it's based on that. And it can actually, if you have two videos being played, it will be two different streams. It's usually like that. So, yeah, that's actually the default. Yeah, if you use different, so for instance, for connecting to bare metal, we do have this X11 Spice, which works only on XR. And it does using XR buffer to stream everything. And that's only the whole desktop. But if you are talking about virtual machines, it's more flexible. Yeah. There are mobile files. They work quite well. Actually, last year there were a company, I think, a company that was both from Telefonica, I think, and they provided two more clients, a web client that's not our Spice HTML5. It's a different one. And they own a demo that runs on the guest, which does a lot of optimizations. So it works quite well. So I think Android and iOS, I think iOS as well, client. Is there a base for the video stream for Jstreamer? That's, I did not check. Sorry. Yeah, for Jstreamer, I know that they're client, I think it's based on Spice SDK, which is the library that uses Jstreamer in the client. But I don't think at the point that they released this, we had Jstreamer on the code. So maybe not. Jstreamer works quite well in Android and iOS and so on. So it might not be so hard to have it, but yeah. Yeah. Anyone else? Okay. So thank you very much. Thank you.
 Hello everyone, my name is Victor Tozo and I'm here to talk about spice.  About me a little bit, I was born and raised in Brazil, sunny Brazil, and I came all the  way to Czech Republic to work on this project for Red Hat.  It's been almost three years now and I'm truly happy to be here to talk with spice with  you and thank you for your interest in being here.  So first of all, how many of you know something about spice?  Okay, two people, that's fine.  And about YouTube, do you use it frequently or no sometimes?  Yeah?  Okay.  So, yeah, this couple of presentations is not going to go deep in the implementation  on how spice works and, oh, or really low level, it's just like a high overview, so  it's fine for everybody, I hope.  I don't plan to take all the time that we have here.  This presentation was planned to be like 20 minutes, 30 minutes, so feel free to raise  your hands anytime to ask questions as they come.  So, first of all, what is spice?  So spice is a way to, similar, connect to some remote device, some remote machine.  It could be a bare metal machine.  It could be a virtual machine.  Our main focus usually is virtual machines.  It was designed for that, but there's several ways to connect to bare metal as well.  So in these two images, quite small, but you have from Windows and from Linux, you are  connecting to two different VMs there, from Windows to different Windows with spreadsheet  or Linux with GNOME running, here from Linux, connecting to different Windows.  And the main point of spice is being able to use this virtual machine, similar, right,  like it was your real machine right there.  So you can play video, share files, listen music, yeah, Redirector, USB.  So that's our main goal.  Now, to talk about spice, it's important to divide it in four different components, which  are the client, which is the spice application that connects to this remote machine.  The server, which is the library user to expose this machine, right, in a way that the client  can connect.  The guest, which is the machine itself.  And the protocol, which is how we define all those components to communicate with each  other.  So the spice protocol will define a set of channels, a set of interfaces that the server  will then expose to the client.  So the server usually connects to this virtual machine and takes all this information and  basically provides this interfaces to the client.  Now we have, for instance, the display channel, which is basically what you see, the drawing.  The input channel could be the keyboard, the mouse, so you can interact with it.  Cursor channel, which is basically to send the shape position of the mouse and the position  itself.  Sound channel, actually there are two channels, the playback channel and the record channel,  so you can do input and output.  And USB channels that you can, for instance, redirect flash drives, webcams and any other  device.  In this presentation, we are mainly focused on the display channel to understand how it  works and the optimizations that we did the last years or so.  But if you have any other questions in regard to other channels and how it works, I'm happy  to take them.  So spice display, how it works.  What does the server do here is that it takes all the images from the guest and has a way  to send it to the client.  So the client will just take the image and server is the one that will send it to the  client.  Now how usually it works.  In a bare metal machine, you just want to, you have your application, you want to do  some drowning and some rendering.  So usually it asks for the driver, the video driver, to do the rendering for you.  But when you are talking about remote, there is no real need to do the rendering on the  remote machine.  So what happens is that the video driver will provide these images to the spice server, which  then will send it to the client.  So basically it's like that, you get any update that you have, it will be sent to the client.  Now we start to think about some optimizations there.  So if you have just an update, do you want to send it everything?  So this example is like, yeah, I'm sending that image plus the whole background, which  is not really bandwidth resource smart, right?  We just need that part, that position that is being updated to be sent.  So besides that, we also can do some compression on the image itself.  So that's what spice usually do in the display channel.  It just take what is being handed in the guest, maybe compress that image and then send it  to the client.  And then I do have a small demo about it.  So here I have a remote viewer, which is a client, a spice GTK client and also VNC client.  I'm connecting to a machine here locally and I'm passing this argument in the, oh, you  cannot see it.  Can you?  Yeah, that's better.  Yeah.  So GTK debug update, so what does it do?  Our client is a GTK-based client.  So with this argument, we can just say, hey, every time that you update something, give  me some information about it.  So you see it's red.  Yeah, you see in the top right, every second, there is an update in the screen.  So basically that's what happened in the guest updates that provides this information to  the server.  It sends just that part.  So it's very good bandwidth wise and you can see if I move my mouse here, every time  that there is update, there will be something.  Yeah.  Because it's client side, the, the, yes, so the question was why there is no drowning  for the mouse.  So you see that we are updating the guest there, but the mouse is not.  So what happens is that the spice tries to, this is a smart way to avoid showing the delay  that happens between the client and the guest.  So we just get the position and through the cursor shape, you also know the shape of the  mouse and we do the drowning on the client side.  So this is in top of everything.  This is like the widget itself being updated, but the mouse is in top of that.  So that's the reason.  You see that if I, you see that the mouse does not change here because it's like the  client side of mouse.  So good question.  So that's generally how it works, but now, yeah, I had a video here.  You see a lot of dates here.  So my intention was to show you a video being playing the guest.  They don't have internet.  Just a second.  So you can foresee what will happen when the video is being played.  What will happen is a lot of updates in that region.  So yeah, and then it becomes a little bit hard to optimize everything because it's a  lot of updates being done.  It's just raw image at this point.  So what can we do about it?  So basically, if we have a lot of updates in the same region on the display on the guest,  we can just convert it to a stream, which is made for this, basically.  So it does have a support for several years, but that's not really bandwidth wise, good,  or it's not really something new or useful for different kind of type of clients.  So with Jstreamer, we just got support for VP8, VP9 as well, H264, and it becomes very  easy to expand that to new features, to new video codecs.  But Jstreamer, so do you guys know what Jstreamer is?  There's anyone?  Yeah?  Okay, cool.  So just a basic overview.  Jstreamer is a framework for multimedia applications.  And if there is two takeaways from Jstreamer, it's a pipeline based, which means that it's  very flexible.  So if you want to build a pipeline to encode a video stream, which is our case, after you  get it working, we just need to move something, replace it, and then we can get a totally  different video stream.  And also, it's very easy to stand because of that, because if you just need something,  you plug in, for instance, you don't need to reinvent the wheel and write everything else,  just that plug-in that you need.  Yeah, a basic run for Jstreamer is this Jstreamer launch.  So if you provide this, I can show you quickly as well.  Like this.  Yeah, it's very cool that you can do a lot with it already.  Like there is this other pattern as well.  And this pattern is called Snow, and it's very nice to know about it because it's like  analog TV in the back old days.  And why it's so interesting is because this is a nightmare for a video codec, right?  So it means that it's like the worst case for trying to encode a stream.  So that's something really nice to always use for testing and so on.  So how it works with us, basically, it's about the same thing.  We get all the images, but in the host, in the spy server, and then we use a pipeline,  Jstreamer pipeline to convert those images in a video stream.  Yeah, so basically the three main plugins that we use is this app source, app sync, and  video encoder.  That means that we just need to change video encoder to change it to different stream types,  and then we can send it to the client.  In the client, basically, it's just the opposite.  We get the video stream, and then we use the video decoder to get the image or several  images, and then we can just draw it on the client.  Yeah, that was how we were doing, but in Jstreamer, actually, they have for this kind of applications,  they have generic plugins called PlayBeam, for instance, that do everything.  So you just need to provide them some input, and they will do the decoding.  You don't need to deal with video decoder, for instance.  You need to find it automatically and do it almost by itself.  So that's something new in our latest release on the client.  And it's very nice also because it can, without effort, support video accelerated, so hardware  accelerated for decoding.  You just need the plugins and the driver capable of that, and of course the hardware itself.  So, streaming with Spice, who benefits with it?  So, yeah, as I mentioned before, we just had MAMJPEG, and that's not much useful because  if you think about, like, we do have web clients, for instance, and there is a lot of huge support  for H264, VP8, VP9, and they usually are working in the back end to do hardware decode  itself as well.  Other thing, clients like Raspere PE or these small low end machines also usually do have  some sort of hardware decoding.  So if we can just send them the right video stream, they will have a better interface  with Spice, right?  It will be more smooth, the usage and so on.  Of course, the CPU usage will go down a bit because of that.  So this presentation, before, I did with a friend of mine, and we had two desktops,  and we did a whole desktop streaming, connected to his desktop and so on.  But I was not really in the mood to come from track republic here with two desktops, two  desktops, so I just recorded.  Let's see, yeah, I can do it with...  Yeah, the user...  The common...  The common task case usually for Spice is Tux, right?  Tux rays.  So this is streaming, H264, the whole desktop.  Yeah, it works quite well, and it's using a few new features from upstream that just  got into Spice protocol.  No, Virgil is here for quite some time, but Virgil only works locally.  Yeah, it works.  It works quite well.  But this is actually guest streaming using PCI-PAS True, so the card goes to the guest,  we get that buffer, use it for streaming.  And yeah, that in Spice protocol just got accepted, and hopefully in the near future  we have another component that will do it on the guest that might do for Intel, NVIDIA,  and maybe others.  Yeah, so for the future, we do have a lot of work.  We start from below, actually, which is a lot of great importance.  So Spice remotely works under TCP, and for real-time video streaming, it's not really  good because of the X on the protocol.  So we are really considering going for a new type of channel that will use RTP, top of  UDP.  So we hope to gain some reduced latency and so on.  We do need some way to...  We have those things implemented, but they don't work quite well.  A way to do dynamic changes in the stream itself.  So for instance, you are trying to stream a 4K display, which is a lot of data, even  for the coding, sometimes even hardware decoding.  So if it's not working well for the client, it's not good at all.  You need to send some information to the server that is dropping a lot of frames, for instance,  and the server should use it as input to change it to a lower quality or something like that.  So we are just in this stage of getting everything in some good...  Like working well, and then we are going to work on those things.  Better handling for hardware encoding and decoding with JStreamer.  So as I said, now in the client side, we just got this playbin, which does almost everything  for us.  So if you have VAPI plugins and an Intel-based system, it just works.  But if you have a different processor, maybe any video card, it's something that needs  to be developed even in JStreamer itself.  Like new plugins or something that needs to get more mainstream.  And hardware encoding, it's long run.  It will take some time because it's something that we are not working on right now.  We are working with different perspectives for now.  So this will take at least another year to be in some interesting shape.  Yeah, and that's it.  I think a special thanks for CodeWeaver guys, Francois and Jeremy, because they were the  ones that actually brought this JStreamer code to Spice.  Check them, the site of them.  They do some interesting stuff.  And Pavel Grunt, which was my friend, and we did this presentation before.  He is my friend, and we did the presentation before, and part of these slides is thanks  to him.  And yeah, that's it.  Do you have any questions?  Do you want to?  I have a question.  There are alternatives.  There's on three, there's another, no machine, and there are alternatives for remote testing.  Why do they choose Spice?  It depends on your use case.  So it depends on what you're comparing as well.  There are some presentations that compare all of them.  All of them have some benefits and some drawbacks.  So Spice is really cool.  There are several, well, I'm biased, right?  I work for Spice, so I say, hey, team Spice.  But there are pros and cons in every aspect.  Why are you going to choose this?  I think you have to consider the possibilities and what they provided to you.  So I don't know if you want to compare.  For instance, I did not try any proprietary version so far.  I don't know, for instance, there is a few.  They do very well in hardware encoding, hardware decoding.  They have, there are companies, huge companies just based for this.  So they will provide better solutions but are proprietary while we are open for software.  So yeah, it depends.  If you want to do, for instance, 4K rendering on a tablet, Spice will not be for you now.  I'm following very well this USB configuration.  So you can, like, start with the device and it goes to the device.  Yeah.  Yeah, basically, if, yes.  Yeah, that was not a question.  It was, he was saying about how Spice works well with USB, for instance.  Yeah, I fixed some bugs on USB, for instance.  There is this guy, he was redirecting his webcam across the ocean.  So from, I don't know, I think, shock republic to North America.  And yeah, it was a huge leak, a lot of memory being used.  And, but yeah, in general, you can just do it like that.  You just redirect your webcam, connect to Skype if you want, or other things.  We have shared folders.  We have this audio synchronization with video.  I don't know, we have a few features that I think it's nice.  Yeah.  That's correct. It depends on your distro.  In Fedora, for instance, we provided its building, it's built with it.  So when you download the package, it usually comes together.  It also works for Windows as well.  So you have Windows, you need the USB DK, it's a different thing.  And then you can share your USB devices from Windows client to any guest.  Yeah.  So the question was with the Spice and JStreamer, if we do have a possibility to not stream the whole desktop,  it actually, yes, it depends on configuration.  So we have three types of configuration in the host.  So there are actually several solutions for streaming.  We can do a guest side streaming.  So we have someone in the guest that will provide this from maybe some device in the guest that has this frame buffer  and maybe even encoded it so we can just use it.  And we have the host side, which is this.  And the host side streaming, the protocol, it's based on that.  And it can actually, if you have two videos being played, it will be two different streams.  It's usually like that.  So, yeah, that's actually the default.  Yeah, if you use different, so for instance, for connecting to bare metal, we do have this X11 Spice,  which works only on XR.  And it does using XR buffer to stream everything.  And that's only the whole desktop.  But if you are talking about virtual machines, it's more flexible.  Yeah.  There are mobile files.  They work quite well. Actually, last year there were a company, I think, a company that was both from Telefonica, I think,  and they provided two more clients, a web client that's not our Spice HTML5.  It's a different one.  And they own a demo that runs on the guest, which does a lot of optimizations.  So it works quite well.  So I think Android and iOS, I think iOS as well, client.  Is there a base for the video stream for Jstreamer?  That's, I did not check. Sorry.  Yeah, for Jstreamer, I know that they're client, I think it's based on Spice SDK,  which is the library that uses Jstreamer in the client.  But I don't think at the point that they released this, we had Jstreamer on the code.  So maybe not.  Jstreamer works quite well in Android and iOS and so on.  So it might not be so hard to have it, but yeah.  Yeah. Anyone else?  Okay. So thank you very much.  Thank you.
froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_0_59.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_60_2147.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_2148_5147.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_5148_7943.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_7944_8051.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_8052_8099.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_8100_9107.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_9108_9575.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_9576_10139.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_10140_10955.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_10956_13595.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_13596_13775.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_13776_14039.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_14040_14615.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_14616_15023.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_15024_15071.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_15072_15131.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_15132_15371.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_15372_20003.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_20004_20531.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_20532_20627.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_20628_21551.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_21552_21743.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_21744_24635.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_24636_25319.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_25320_25523.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_25524_26591.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_26592_27815.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_27816_30371.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_30372_40919.png froscon2017-1988-spice_new_ways_to_remote_desktops_with_gstreamer_integration_40920_41299.png
Hello everyone, my name is Victor Tozo and I'm here to talk about spice. About me a little bit, I was born and raised in Brazil, sunny Brazil, and I came all the way to Czech Republic to work on this project for Red Hat. It's been almost three years now and I'm truly happy to be here to talk with spice with you and thank you for your interest in being here. So first of all, how many of you know something about spice? Okay, two people, that's fine. And about YouTube, do you use it frequently or no sometimes? Yeah? Okay. So, yeah, this couple of presentations is not going to go deep in the implementation on how spice works and, oh, or really low level, it's just like a high overview, so it's fine for everybody, I hope. I don't plan to take all the time that we have here. This presentation was planned to be like 20 minutes, 30 minutes, so feel free to raise your hands anytime to ask questions as they come. So, first of all, what is spice? So spice is a way to, similar, connect to some remote device, some remote machine. It could be a bare metal machine. It could be a virtual machine. Our main focus usually is virtual machines. It was designed for that, but there's several ways to connect to bare metal as well. So in these two images, quite small, but you have from Windows and from Linux, you are connecting to two different VMs there, from Windows to different Windows with spreadsheet or Linux with GNOME running, here from Linux, connecting to different Windows. And the main point of spice is being able to use this virtual machine, similar, right, like it was your real machine right there. So you can play video, share files, listen music, yeah, Redirector, USB. So that's our main goal. Now, to talk about spice, it's important to divide it in four different components, which are the client, which is the spice application that connects to this remote machine. The server, which is the library user to expose this machine, right, in a way that the client can connect. The guest, which is the machine itself. And the protocol, which is how we define all those components to communicate with each other. So the spice protocol will define a set of channels, a set of interfaces that the server will then expose to the client. So the server usually connects to this virtual machine and takes all this information and basically provides this interfaces to the client. Now we have, for instance, the display channel, which is basically what you see, the drawing. The input channel could be the keyboard, the mouse, so you can interact with it. Cursor channel, which is basically to send the shape position of the mouse and the position itself. Sound channel, actually there are two channels, the playback channel and the record channel, so you can do input and output. And USB channels that you can, for instance, redirect flash drives, webcams and any other device. In this presentation, we are mainly focused on the display channel to understand how it works and the optimizations that we did the last years or so. But if you have any other questions in regard to other channels and how it works, I'm happy to take them. So spice display, how it works. What does the server do here is that it takes all the images from the guest and has a way to send it to the client. So the client will just take the image and server is the one that will send it to the client. Now how usually it works. In a bare metal machine, you just want to, you have your application, you want to do some drowning and some rendering. So usually it asks for the driver, the video driver, to do the rendering for you. But when you are talking about remote, there is no real need to do the rendering on the remote machine. So what happens is that the video driver will provide these images to the spice server, which then will send it to the client. So basically it's like that, you get any update that you have, it will be sent to the client. Now we start to think about some optimizations there. So if you have just an update, do you want to send it everything? So this example is like, yeah, I'm sending that image plus the whole background, which is not really bandwidth resource smart, right? We just need that part, that position that is being updated to be sent. So besides that, we also can do some compression on the image itself. So that's what spice usually do in the display channel. It just take what is being handed in the guest, maybe compress that image and then send it to the client. And then I do have a small demo about it. So here I have a remote viewer, which is a client, a spice GTK client and also VNC client. I'm connecting to a machine here locally and I'm passing this argument in the, oh, you cannot see it. Can you? Yeah, that's better. Yeah. So GTK debug update, so what does it do? Our client is a GTK-based client. So with this argument, we can just say, hey, every time that you update something, give me some information about it. So you see it's red. Yeah, you see in the top right, every second, there is an update in the screen. So basically that's what happened in the guest updates that provides this information to the server. It sends just that part. So it's very good bandwidth wise and you can see if I move my mouse here, every time that there is update, there will be something. Yeah. Because it's client side, the, the, yes, so the question was why there is no drowning for the mouse. So you see that we are updating the guest there, but the mouse is not. So what happens is that the spice tries to, this is a smart way to avoid showing the delay that happens between the client and the guest. So we just get the position and through the cursor shape, you also know the shape of the mouse and we do the drowning on the client side. So this is in top of everything. This is like the widget itself being updated, but the mouse is in top of that. So that's the reason. You see that if I, you see that the mouse does not change here because it's like the client side of mouse. So good question. So that's generally how it works, but now, yeah, I had a video here. You see a lot of dates here. So my intention was to show you a video being playing the guest. They don't have internet. Just a second. So you can foresee what will happen when the video is being played. What will happen is a lot of updates in that region. So yeah, and then it becomes a little bit hard to optimize everything because it's a lot of updates being done. It's just raw image at this point. So what can we do about it? So basically, if we have a lot of updates in the same region on the display on the guest, we can just convert it to a stream, which is made for this, basically. So it does have a support for several years, but that's not really bandwidth wise, good, or it's not really something new or useful for different kind of type of clients. So with Jstreamer, we just got support for VP8, VP9 as well, H264, and it becomes very easy to expand that to new features, to new video codecs. But Jstreamer, so do you guys know what Jstreamer is? There's anyone? Yeah? Okay, cool. So just a basic overview. Jstreamer is a framework for multimedia applications. And if there is two takeaways from Jstreamer, it's a pipeline based, which means that it's very flexible. So if you want to build a pipeline to encode a video stream, which is our case, after you get it working, we just need to move something, replace it, and then we can get a totally different video stream. And also, it's very easy to stand because of that, because if you just need something, you plug in, for instance, you don't need to reinvent the wheel and write everything else, just that plug-in that you need. Yeah, a basic run for Jstreamer is this Jstreamer launch. So if you provide this, I can show you quickly as well. Like this. Yeah, it's very cool that you can do a lot with it already. Like there is this other pattern as well. And this pattern is called Snow, and it's very nice to know about it because it's like analog TV in the back old days. And why it's so interesting is because this is a nightmare for a video codec, right? So it means that it's like the worst case for trying to encode a stream. So that's something really nice to always use for testing and so on. So how it works with us, basically, it's about the same thing. We get all the images, but in the host, in the spy server, and then we use a pipeline, Jstreamer pipeline to convert those images in a video stream. Yeah, so basically the three main plugins that we use is this app source, app sync, and video encoder. That means that we just need to change video encoder to change it to different stream types, and then we can send it to the client. In the client, basically, it's just the opposite. We get the video stream, and then we use the video decoder to get the image or several images, and then we can just draw it on the client. Yeah, that was how we were doing, but in Jstreamer, actually, they have for this kind of applications, they have generic plugins called PlayBeam, for instance, that do everything. So you just need to provide them some input, and they will do the decoding. You don't need to deal with video decoder, for instance. You need to find it automatically and do it almost by itself. So that's something new in our latest release on the client. And it's very nice also because it can, without effort, support video accelerated, so hardware accelerated for decoding. You just need the plugins and the driver capable of that, and of course the hardware itself. So, streaming with Spice, who benefits with it? So, yeah, as I mentioned before, we just had MAMJPEG, and that's not much useful because if you think about, like, we do have web clients, for instance, and there is a lot of huge support for H264, VP8, VP9, and they usually are working in the back end to do hardware decode itself as well. Other thing, clients like Raspere PE or these small low end machines also usually do have some sort of hardware decoding. So if we can just send them the right video stream, they will have a better interface with Spice, right? It will be more smooth, the usage and so on. Of course, the CPU usage will go down a bit because of that. So this presentation, before, I did with a friend of mine, and we had two desktops, and we did a whole desktop streaming, connected to his desktop and so on. But I was not really in the mood to come from track republic here with two desktops, two desktops, so I just recorded. Let's see, yeah, I can do it with... Yeah, the user... The common... The common task case usually for Spice is Tux, right? Tux rays. So this is streaming, H264, the whole desktop. Yeah, it works quite well, and it's using a few new features from upstream that just got into Spice protocol. No, Virgil is here for quite some time, but Virgil only works locally. Yeah, it works. It works quite well. But this is actually guest streaming using PCI-PAS True, so the card goes to the guest, we get that buffer, use it for streaming. And yeah, that in Spice protocol just got accepted, and hopefully in the near future we have another component that will do it on the guest that might do for Intel, NVIDIA, and maybe others. Yeah, so for the future, we do have a lot of work. We start from below, actually, which is a lot of great importance. So Spice remotely works under TCP, and for real-time video streaming, it's not really good because of the X on the protocol. So we are really considering going for a new type of channel that will use RTP, top of UDP. So we hope to gain some reduced latency and so on. We do need some way to... We have those things implemented, but they don't work quite well. A way to do dynamic changes in the stream itself. So for instance, you are trying to stream a 4K display, which is a lot of data, even for the coding, sometimes even hardware decoding. So if it's not working well for the client, it's not good at all. You need to send some information to the server that is dropping a lot of frames, for instance, and the server should use it as input to change it to a lower quality or something like that. So we are just in this stage of getting everything in some good... Like working well, and then we are going to work on those things. Better handling for hardware encoding and decoding with JStreamer. So as I said, now in the client side, we just got this playbin, which does almost everything for us. So if you have VAPI plugins and an Intel-based system, it just works. But if you have a different processor, maybe any video card, it's something that needs to be developed even in JStreamer itself. Like new plugins or something that needs to get more mainstream. And hardware encoding, it's long run. It will take some time because it's something that we are not working on right now. We are working with different perspectives for now. So this will take at least another year to be in some interesting shape. Yeah, and that's it. I think a special thanks for CodeWeaver guys, Francois and Jeremy, because they were the ones that actually brought this JStreamer code to Spice. Check them, the site of them. They do some interesting stuff. And Pavel Grunt, which was my friend, and we did this presentation before. He is my friend, and we did the presentation before, and part of these slides is thanks to him. And yeah, that's it. Do you have any questions? Do you want to? I have a question. There are alternatives. There's on three, there's another, no machine, and there are alternatives for remote testing. Why do they choose Spice? It depends on your use case. So it depends on what you're comparing as well. There are some presentations that compare all of them. All of them have some benefits and some drawbacks. So Spice is really cool. There are several, well, I'm biased, right? I work for Spice, so I say, hey, team Spice. But there are pros and cons in every aspect. Why are you going to choose this? I think you have to consider the possibilities and what they provided to you. So I don't know if you want to compare. For instance, I did not try any proprietary version so far. I don't know, for instance, there is a few. They do very well in hardware encoding, hardware decoding. They have, there are companies, huge companies just based for this. So they will provide better solutions but are proprietary while we are open for software. So yeah, it depends. If you want to do, for instance, 4K rendering on a tablet, Spice will not be for you now. I'm following very well this USB configuration. So you can, like, start with the device and it goes to the device. Yeah. Yeah, basically, if, yes. Yeah, that was not a question. It was, he was saying about how Spice works well with USB, for instance. Yeah, I fixed some bugs on USB, for instance. There is this guy, he was redirecting his webcam across the ocean. So from, I don't know, I think, shock republic to North America. And yeah, it was a huge leak, a lot of memory being used. And, but yeah, in general, you can just do it like that. You just redirect your webcam, connect to Skype if you want, or other things. We have shared folders. We have this audio synchronization with video. I don't know, we have a few features that I think it's nice. Yeah. That's correct. It depends on your distro. In Fedora, for instance, we provided its building, it's built with it. So when you download the package, it usually comes together. It also works for Windows as well. So you have Windows, you need the USB DK, it's a different thing. And then you can share your USB devices from Windows client to any guest. Yeah. So the question was with the Spice and JStreamer, if we do have a possibility to not stream the whole desktop, it actually, yes, it depends on configuration. So we have three types of configuration in the host. So there are actually several solutions for streaming. We can do a guest side streaming. So we have someone in the guest that will provide this from maybe some device in the guest that has this frame buffer and maybe even encoded it so we can just use it. And we have the host side, which is this. And the host side streaming, the protocol, it's based on that. And it can actually, if you have two videos being played, it will be two different streams. It's usually like that. So, yeah, that's actually the default. Yeah, if you use different, so for instance, for connecting to bare metal, we do have this X11 Spice, which works only on XR. And it does using XR buffer to stream everything. And that's only the whole desktop. But if you are talking about virtual machines, it's more flexible. Yeah. There are mobile files. They work quite well. Actually, last year there were a company, I think, a company that was both from Telefonica, I think, and they provided two more clients, a web client that's not our Spice HTML5. It's a different one. And they own a demo that runs on the guest, which does a lot of optimizations. So it works quite well. So I think Android and iOS, I think iOS as well, client. Is there a base for the video stream for Jstreamer? That's, I did not check. Sorry. Yeah, for Jstreamer, I know that they're client, I think it's based on Spice SDK, which is the library that uses Jstreamer in the client. But I don't think at the point that they released this, we had Jstreamer on the code. So maybe not. Jstreamer works quite well in Android and iOS and so on. So it might not be so hard to have it, but yeah. Yeah. Anyone else? Okay. So thank you very much. Thank you.
other device. In this presentation, we are mainly it works and the optimizations that we did the interface with Spice, right? It will be more and for real-time video streaming, it's not talking about virtual machines, it's more