Movit: High-speed, high-quality video filters on the GPU
Gunderson, Steinar H.
Movit (the "Modern VIdeo Toolkit") is a high-performance, high-quality, open-source library for video filters, running on the GPU. Come see what the future holds when open-source video editing steps into 2014! Movit contains everything you need and expect for basic video filtering requirements (scale, blur, sharpen, mix, overlay, simple color grading, and others). Movit is supported in MLT, has full support in Shotcut and is on track for Kdenlive support. In this talk, I'll talk a bit about: * What does Movit do, and what doesn't it do? * What do we mean by "high-performance"? Why the GPU? * What do we mean by "high-quality"? How hard can a blend be anyway? * What goes on behind the scenes in Movit's filter graph? * How do you embed Movit in your own application? There will also be a demo of Movit in action, and of course Q&A if time permits.
 The way back after the second lunch break. My name is Stanna. I will be talking about MoveIt, which is my newest hobby project, more or less. And as you can see, it has this sort of pun in it. But it stands for the modern video toolkit. And this is, of course, dangerous because anything called modern usually isn't. Anything called toolkit is usually hideous. But it tries to be a high-performance, high-quality open-source library for video filters. And let me actually talk a bit first. What do I mean by high-performance? And to do that, we'll go back 25 years in time and see. This is NCC. I'm sorry to find this a bit small because there will be more things. But this is actually 25 years ago. NCC has held up remarkably well. But this is how you do a fade if you were programming this in 1989. I've stripped out all the complexity. This is only gray scale. There's no alpha. There's no nothing. But you take in an image A and image B. You want to fade between them. You multiply one by some fade constant F. And you multiply one by the other constant. And then you add them together. It's plain and simple. And then someone figures out, ah, maybe this is a bit slow. So we rearrange it a bit. We move things around. So we only have one multiply. And this is also a standard trick. And then people come along and say, it's still too slow. And the reason why it's so slow is we're using floating point. And it's not really the floating point itself. It's slow. Floating point has been really, really fast for most of the last at least the last 20 years. The thing is, slow is moving things back and forth from its two floats. So we're doing fixed point. We're just multiplying things a bit more tricky. We need to round a bit. We need to shift a bit. But it's still like the same thing. Now you come to MMX. 1997 around then Intel figured out that people want to do the same things to many different values. We have all these pixels. We have all these pixels over here. Let's do them four at a time. The problem, of course, is it becomes very, very complicated. I wrote this. I used at least half an hour to get it to work at all. It's probably not very efficient. It's even worse if you want to write like pure assembler. And before you talk about auto vectorization, I can tell you no compiler in the world will auto vectorize the previous things, this one, because we reduced position. And in general, auto vectorization is not something you can really rely on. So you need to write something like this. And of course, right at the bottom is still our old friend, the scalar thing for fallback for the last few pixels. And then we invent not only MMX, but SSE, SC2, AVX, AVX2. I hear there's now coming AVX512. And you need to write code for every single one of these. And then Intel comes along and says, hey, we have multiple cores. We cannot do more slow anymore. We need you to split your work. OK, so you need to do like P threads, whatever, fire up threads. And if one of them is slow, you will have a problem. So you need to distribute it somehow more intelligent than I have done. And this is, of course, on top of all the SSE, SC2, AVX, whatever, whatever, whatever. OK, so we've figured this is 10 years ago, right? We have multicore. We've figured out how to do this. Let me now show you how people actually do this. This is 2006. This is from a project called Frylick. If you have ever edited video on your Linux machine, you are most likely fading using this software or something very much like it. It is still, as you can see, completely scalar. There's no AVX. That's not support multicore. So what the heck? It turns out this model of doing things efficiently on the CPU is very, very complicated. So people don't do it. At least we don't in the free software world. Maybe you do so if you're on Windows, but this is not reality. So fast forwarded to 2014 in GPUs, you cannot really buy a desktop machine anymore without a GPU. And if you're lucky, you get a cat. Actually, you cannot buy a laptop anymore without having a GPU and maybe a cat. You can't buy a cell phone anymore without a reasonably powerful GPU. And this is a bottle-deposited machine. It has a GPU. So they are no longer uncommon. We should not anymore assume the user does not have a GPU. So how does it program it? This is GLSL. This is the most common shading language that you would use to program your GPU. And this is the four lines you need to do everything the CPU version did with full vectorization, with full splitting, with full handling of all edge cases, yada, yada, yada. And Movid uses GLSL. If you would write this in Movid, you need only a little more glue around it. This is basically because Movid is able to chain many of these things together. So it just needs some namespacing and things like this. It is refreshingly simple. And I think this is maybe the thing about using GPUs for anything like this. It's not necessarily that it's so much faster by default. Well, it depends a lot on your GPU. But I'm pretty sure that on the laptop, I could probably need the GPU for most of these things if I really, really want to. But it's so simple to get high performance. Refreshing. So let's see what we can do about this. And when I say high quality, this is, of course, now I'll try not to diss a lot of other people too much. This is one of the wrong things you can do. I'm sure, I mean, this is the graphics deck of the bedroom, right? I'm sure everyone knows this, but I will say it nevertheless. The pixel values that we store in our images are not linear amounts of light. They are a gamma compressed format, which means that they have some sort of curve. And if you do any sort of arithmetic on them, you need to convert them into the real value first. If you don't do this, you end up in like a situation here on the right. Or I've faded it like 50-50 together an image that is dark and light. And you can especially observe that sort of the wrong things come through. Like this right spot here is way too visible. These edges here are just way too visible. It's not right. The left one is how it's supposed to look like. This is how you'd look if you took physical grains of film and put them on top of each other. So this is one wrong thing you can do, not converting in the right place. And of course, the example code that I showed does not do this correctly, of course, because it does the simple thing. You can use the wrong color spaces. Now, I never trust beamers. I sort of exaggerated the effect here. I've done it three times. But we talk about RGB. We never really say what red, what green, what blue. And it turns out there are a number of different interpretations of this. And if you don't use the right one, your image on the left can end up like the image on the right with wrong colors in some way. So you really need to care about this sort of thing. This is not that visible, so I've zoomed it in. This is what happens if you put your chroma planes at the wrong site. There are a lot of different standards, or eight standards. If you put them wrong, you get these color fringes around here. So you need to really get this right. And before you ask, a lot of current in this video software does this wrong. The effects are subtle, but you will see them if you know where to look. And finally, when I talk about quality, I mean more than just image output quality. Quality is not a feature in itself. It really is a process. It really is a commitment to how we want to do things. Move it as a unit test. Then ask the tests that you can actually say, if the test runs in your machine, most likely everything would work. And I really, really try to say this as, when you think about you want to make something that works and doesn't give people the wrong idea of how to use it, you really want to convince yourself you're doing the wrong things. A lot of these things seem really, really simple. They're like, don't do mistakes. But they are mind-numbly hard. They're so easy to get a one-pixel placement wrong. Like, do I add one? Do I subtract one? Do I divide by two? Do I multiply by two? My education is in multimedia DSP, and I still get these things wrong. I try to have tests, and every time I write a test, I feel like this is a way too stupid test. This cannot possibly fail, and it fails. You might be perfect programmers. I'm not. So again, 90% cold coverage. So let's see how it is to use. Move it. Use a chain of effects. I usually say that if your hello world is more than five lines long, you've failed. So in a sense, I've failed. But I'll walk you through a bit what you do. First, you define an input. You say that it's in some color space. You say it's in some gamma. You define, well, OK, the order is VGA. It's unsigned by all this. You add it. You add an effect. In this case, a glow effect. You add an output of some format. You set how many output bits you want. And you finalize it. Now the chain is locked. Now it gets compiled, uploaded to your GPU. You can run it very, very fast, many times you want to run. You just set, oh, here are some new pixels. Render just green, this is how your main loop typically looks like. And doing this will create this chain here internally. This is a debug dump from where move it works. You have an input. You have a glow effect. And you have an output. And then internally, the first thing move it does, it's right is off to some big hideous monster. This is, well, a glow effect. That's sort of, I want to cut out the highlights. I want to blur it horizontally. I want to blur it vertically. And I want to add it back to the first thing. And then, this is, of course, way, way, way too small to read. I'm not expecting you to read this. But after 19 passes of this mini compiler thing, it has inserted all these gamma conversions that you need. It's split things up the faces. You can see the different colors here. These are different faces, because sometimes it's advantageous not to do everything change. And it's eventually created programs for you. And all of that will turn this image here to this image here. More effects. I don't have a lot of effects in it. And the reason is I would rather have good effects than many effects. So each one of them, there are only like 10 or 12 currently. There will be more. But each of them are fast. They work. They don't give you any surprises. They look good. Vignette, fair and simple. Just dark in the corners of it through some cosine squared law. This is sharpened. This is not an unsharpened mask. This is actually a deconvolution that tries to invert the transfer function of the lens. So you don't get this halo effect that you get. I don't think you can see it to the well on the slide, though, unfortunately. But usually it's better than a sharp mask. Saturation, desaturation. Observe again that since we are operating in linear space, which we should do, the image as a whole does not get more darker or lighter. It just becomes more or less desaturated. And finally, this isn't actually in yet. This is lens blur through FFT. It actually does a full convolve with whatever you want. And of course, when you have convolve, you can also go crazy and convolve with a star or whatever you want to. If you want a hexagonal blur, then fine by me. So demo time. And I really hope that my demo actually works in this resolution. It does. This is some sort of, I mean, these widgets here are just like something I threw together. They're not meant to be a UI thing. But they're like the standard color correction. So we'll say you can find in almost anything except Linux. So you can sort of drag your mid-tones around, your highlights around, and you can make it as ugly as you want to. But of course, the demo app is boring. This is KDN Live. This is, at least if you ask me, the only usable video editor on Linux right now. I hope there's no blender of people here to take offense. And this is like my early alpha prehacked version that actually uses MoveIt through a project called MLT. Of course, I mean, I'm pulling it down a clip here. It plays right now. MoveIt isn't really doing a lot. It's only doing color space conversion and things like that. But I can pull out an effect here. Let's say I want to blur it. And I can blur more or less. And you can see this happens in 60 FPS. This is 720p. It's not, I mean, it's a small window, but it's not cheating. It's actually editing in 720p and just sailing down for display. If I'm bored with the blur, of course, I can do some sort of fade, which will work properly. I can, of course, also do white balance. I just need to find some gray point in this clip here. Let's just focus on here. This is gray. And now gray actually stays gray. There are multiple Linux white balance things where gray does not become gray. Because no one ever wrote a test for it and it never worked. I'm not kidding you. This is frustrating. You can change the color temperature. It's super, super warm. It still doesn't become warped in any way. It doesn't clip. Again, because we're working in linear space. And you can now sort of play. I think I forgot to sort of reverse lag on the clip, so it's fading the wrong way. This is a UISU, right? So again, this is, I think, well, in the fade, I don't think I can do 60 FPS. I think this is 30 FPS, because this is a four-year-old ultra-portable laptop. This is not something high-end in any way. And I think that's mostly because we're decoding the video on the CPU. So really, I mean, if I wanted to do this in the current software world, this would be five FPS, maybe. And now it's 60. And this also means, of course, that if you have a much better machine, like say, any mid-range and media card, you can probably go 4K video. Directly on the GPU, no tricks. Just do things, move around, do whatever you want. I think this was the demo. Yes, future work is always the best part, of course. There will be more filters, but they will be added again slowly, because I don't want a lot of filters that don't work. There are some limitations in the YCBTR handling, because it's actually surprisingly subtle to get all the scaling correctly, all the interpolation. I want better interpolation in today's contents. How many people here like Interlaced? No hands, right? And unfortunately, of course, Interlaced's video is a reality many of us still have to deal with. And even though it's sort of the modern video toolkit, I think I might actually have to sort of succumb a bit here and support it eventually. And then, of course, whatever clients need, because Moved is not a program. Moved is used by currently MLT, which is like this generic video library thing that can do all sorts of things, which is turned used by ShotCuts. KDN Live, probably a few others, Flowblade. So again, whatever sort of clients come to me and say, oh, I need a shared library. This is the kind of things that, obviously, I want to support. So you will find that this KDN Live patches have not been released until now. I just uploaded them like an hour ago. So if you're really, really brave and happen to have a machine that's sort of like mine, I can promise you it's stopped hanging the Intel drivers in the kernel. It's used to. I think it might actually not segfault on a video all the time. So if you're brave, you can go and get the library there. You can get the KDN Live patches there and edit and report tons of bugs to me. So questions? Please hand up for the mic. I have two questions for you. The first one is, do you hams at providing a plug-in for just Dreamer because it's a very useful tool for making pipes of content? That's a plug-in for? Just Dreamer. Gstreamer. Not immediately. I would be very, very happy if someone wanted to add move support to Gstreamer. I don't unfortunately know Gstreamer very well, per se. One challenge here, though, is that for best possible move it performance, you really, really want to be able to chain things together. So if you have three successive move it filters, you want them with one Gstreamer component. You don't want them as three different ones. Because then you have to go all the way to the GPU, do something, go down to memory, go down to the CPU, and up again, which really, really hurts performance. A lot of the work that's gone into this is about keeping it on the GPU, chaining it efficiently. But if you want to do Gstreamer support and know Gstreamer much better than I do, then I'll be happy to talk to you. Another point is, as a user, I'm looking for a tool that uses GPU to do the transcoding of video format. Are you aware of some? Or maybe you're projecting some time, I don't know. So you're talking about the GPU to transcode, right? And there are two ways here. There's video decode, which is pretty well supported now. At least if you have NVIDIA, you can do NVIDIA Po. My KDNL and Pi patch doesn't support it right now. But it can certainly be done. It can play video, right? Now, encoding the other way is much, much more tricky. That's a very involved thing. You will be aware that Intel has something called Quick Sync, Quick Sync, which essentially does a live H.264 encode. There are some commercial GPU video encoders. I don't think there are any Linux ones. Basically, this is because this is a very, very complicated thing, right? I hear there's talk for H.264 to use OpenCL to try to accelerate. It already does it. OK, someone says that it already. It crashes a lot. It's very bad. It works very minimally. OK, just to repeat for the stream, there is the message here that it only does it for a look ahead, but it's not really done yet, right? It's given up, like. Oh, they've given up even. OK. It's not mentioned in the video. OK, so I think the answer for latter one is no. I don't think there's a lot of good things right now. The good thing, of course, is that once your GPU is doing all the pixel pushing, your CPU is free to handle the other work. Yes, on a question in the back. Yeah, how is the chaining working? Are you really chaining the effects in a program, like a shader, or do you chain them by rendering to target and texture from this? They are mainly chained as in the program. Most of it is run within one program. There are a few cases where you need to split the chain. One of them is when you do something like a blur, where you sample like 15 different pixels right there, then you want to sample from memory. You don't want every single computation to be run 15 times, right? There are some other few cases where you need to bounce to memory. And that is especially when you have a split in your chain. It needs to be used by two different things. So again, I'll just go back to one of the earlier slides here. It's not so early slides, actually. Here, right? If you look at the colors here, you see the three first ones are green. Those are part of the same shader. And that's the input, the gamma expansion, and actually some alpha handling, because we handle alpha correctly as well. So in general, we try to combine them into one shader as much as we can. And we really rely on the shader compiler to optimize and inline things away as much as we can. I would love for the Intel compiler to be smarter about things I hope it could be smarter for. But I assume there are people in the room here who can help me with that. There are more questions both in the front and the back. Good thing about not controlling the microphone is I don't have to make these decisions. I'll be coming. I'm sorry. I've got two questions. The first is, do you support 10 and 12-bit video? Yes. Let me elaborate a bit on this. I move it mostly uses 16-bit floating point support internally. This is not enough for doing 12-bit video with full accuracy. It is enough for doing 10-bit. We have accuracy tests in most of the crucial effects to check that we actually get within the right kicks level. So yes, it is supported. And this is clearly something I want to do for 12-bit. We would have to go to 32-bit floating point. This is not hard in itself. But I think open source things don't really support it all that much yet. But it's certainly something I would love to support. And it's very close to there. Because we're working in full floating point all the time, we don't have these 8-bit locks. The other question was, if you did video decode on the GPU, is it a zero copy operation from memory to, is there a trip to the CPU and back to use it? No. No. If you do video decoding on GPU, you can certainly put that into texture with whatever extension you have available. And then we can use it as an input. Again, KDN Live doesn't currently do this. But it would certainly be possible to do all the way without ever touching the CPU. There is a VD-Po extension for that. VD-Po.ngl compact. Yes. And there's also, of course, now a VD-Po emulator for VA APIs. You can run VD-Po things against your input card. I've never tested it. But you want something like this, right? Especially if you look at a demo here now, when it's doing a fade, it's CPU bound. It's my CPU not able to decode more than two streams of 40 something FPS. So first some comments on the previous question about GSTRIMA. So I think this would be really fun to implement. And maybe I'm going to look at that really soon. It was very fun in MLT. And well, then on Intel, you have with VA API the possibility to do the decoding on the GPU and then get textures from that. And in that case, you should also be able to have zero copy processing directly here. But now to my question, what's the input and output that you can have with your toolkit? Can you have GL textures as input and output? And can you also have system memory as input and output? I can tell you. So Movid is plug-in based in the center. You have a class that defines an input and an output. And you have effects, of course, in the middle. There is an input class that takes in a system memory. And then, of course, loads it to a texture and use it from there. I don't actually think I have a texture input, but this is trivial to port. This is like a pen line to support. It's not a problem at all. As for rendering, Movid renders to whatever FBO we give it. So you can either render to the screen. That's FBO 0. Or you can render to another FBO and then do a read pixels down into the CPU. So input and output there is sort of free. OK, thank you. Any more questions? We have like 40 minutes to fill or something, right? Yeah. I have to fill it, right? There's not a question down here. OK. What are your software requirements to make Movid working? You spoke about VDPo or? No. So my software requirements are, in a sense, very low. You need a C++ compiler. You need the eigenmatrix library. You need OpenGL. And by OpenGL, I mean any card that can do shaders, which means any card from eight or nine years ago. So there are no weird systems. It certainly works on both Intel API and Vidya, for instance. It works. I know people have it working on both Linux, Windows, OS X. I try to stick with, again, we're 2014. It should be allowed to use C++, right? But this is sort of the weirdest requirement, in a sense. What about other video editors, such as, say, OpenShot? Are they likely to include that? OpenShot uses MLT, I think. So it means that, what you need to do to use this MLT, you need OpenGL complex. Just you need to initialize OpenGL so I can play with it. Apart from that, it's completely transparent to you. MLT now handles all the upload and download things, transparently, and change things together. This was done by Dan Vanity, who was one of the MLT guys. I don't know if he might be here or not. I don't think so. But in general, it's easy to get to start with. If you want to play with it, you can go to Shotcut, which also has full native support, and has for a while. Thanks so much. Again, any more questions? OK, another one. Yes? You said you need OpenGL with shaders. What about OpenGL ES, or GLS2 or something? This is a good question. I don't support the GLS right now. I'm not sure if I actually will. It depends a bit on what the requirements are. What do you want to use it for? Because editing video on your mobile is maybe not a use case that I care that much about to begin with. I try very hard to do only one thing and do it well. And instead of trying to fragment myself, if you have a compelling use case for GLS, which is for those who don't know, like the mobile, newer, modern version of OpenGL, I might consider it. But I don't want to maintain a fork. But I don't really require any magic things. I only need to draw a quad and have a shader on it. I've been playing a lot with robots and had many issues by doing some video understanding, finding a shape, finding a color, or whatever. Do you think that move it can be used to, I mean, using some filter to maybe only select the red things on an image, and then being able to locate some shapes, or it could be an extension. Very interesting in the robot-y area. I think my answer here will have to be no. Just because this is, again, deviating from what I want to do, and there's only one of me. If you want to, you can most certainly use move it and write your own effects. But I'm not really sure what you would gain from it. Most likely, if you have that kind of specialized computer vision things, maybe you don't really want what move gives you. Maybe you just want to have your own shader. You don't want to blur. You want some sort of specialized image recognition thing. So then maybe move it. It's not for you. This is really about, for me at least, it's really about open source graphics. I think we are out of questions. Thank you, Ben. Thanks. Thank you. So the next presentation is going to be about the uploading of video on Wayland. Thank you. Thank you.
 The way back after the second lunch break.  My name is Stanna.  I will be talking about MoveIt, which is my newest hobby  project, more or less.  And as you can see, it has this sort of pun in it.  But it stands for the modern video toolkit.  And this is, of course, dangerous because anything  called modern usually isn't.  Anything called toolkit is usually hideous.  But it tries to be a high-performance, high-quality  open-source library for video filters.  And let me actually talk a bit first.  What do I mean by high-performance?  And to do that, we'll go back 25 years in time and see.  This is NCC.  I'm sorry to find this a bit small because there will be  more things.  But this is actually 25 years ago.  NCC has held up remarkably well.  But this is how you do a fade if you were programming this  in 1989.  I've stripped out all the complexity.  This is only gray scale.  There's no alpha.  There's no nothing.  But you take in an image A and image B. You want to fade  between them.  You multiply one by some fade constant F. And you multiply  one by the other constant.  And then you add them together.  It's plain and simple.  And then someone figures out, ah, maybe this is a bit slow.  So we rearrange it a bit.  We move things around.  So we only have one multiply.  And this is also a standard trick.  And then people come along and say, it's still too slow.  And the reason why it's so slow is we're using floating point.  And it's not really the floating point itself.  It's slow.  Floating point has been really, really fast for most of the  last at least the last 20 years.  The thing is, slow is moving things back and forth from  its two floats.  So we're doing fixed point.  We're just multiplying things a bit more tricky.  We need to round a bit.  We need to shift a bit.  But it's still like the same thing.  Now you come to MMX.  1997 around then Intel figured out that people want to do  the same things to many different values.  We have all these pixels.  We have all these pixels over here.  Let's do them four at a time.  The problem, of course, is it becomes very, very complicated.  I wrote this.  I used at least half an hour to get it to work at all.  It's probably not very efficient.  It's even worse if you want to write like pure assembler.  And before you talk about auto vectorization, I can tell  you no compiler in the world will auto vectorize the  previous things, this one, because we reduced position.  And in general, auto vectorization is not something  you can really rely on.  So you need to write something like this.  And of course, right at the bottom is still our old  friend, the scalar thing for fallback for the last few  pixels.  And then we invent not only MMX, but SSE, SC2, AVX, AVX2.  I hear there's now coming AVX512.  And you need to write code for every single one of these.  And then Intel comes along and says, hey, we have multiple  cores.  We cannot do more slow anymore.  We need you to split your work.  OK, so you need to do like P threads, whatever, fire up  threads.  And if one of them is slow, you will have a problem.  So you need to distribute it somehow more intelligent than  I have done.  And this is, of course, on top of all the SSE, SC2, AVX,  whatever, whatever, whatever.  OK, so we've figured this is 10 years ago, right?  We have multicore.  We've figured out how to do this.  Let me now show you how people actually do this.  This is 2006.  This is from a project called Frylick.  If you have ever edited video on your Linux machine, you are  most likely fading using this software or something very  much like it.  It is still, as you can see, completely scalar.  There's no AVX.  That's not support multicore.  So what the heck?  It turns out this model of doing things efficiently on the  CPU is very, very complicated.  So people don't do it.  At least we don't in the free software world.  Maybe you do so if you're on Windows, but this is not  reality.  So fast forwarded to 2014 in GPUs, you cannot really buy a  desktop machine anymore without a GPU.  And if you're lucky, you get a cat.  Actually, you cannot buy a laptop anymore without having a  GPU and maybe a cat.  You can't buy a cell phone anymore without a reasonably  powerful GPU.  And this is a bottle-deposited machine.  It has a GPU.  So they are no longer uncommon.  We should not anymore assume the user does not have a GPU.  So how does it program it?  This is GLSL.  This is the most common shading language that you would use  to program your GPU.  And this is the four lines you need to do everything the  CPU version did with full vectorization, with full  splitting, with full handling of all edge cases, yada, yada,  yada.  And Movid uses GLSL.  If you would write this in Movid, you need only a little  more glue around it.  This is basically because Movid is able to chain many of  these things together.  So it just needs some namespacing and things like this.  It is refreshingly simple.  And I think this is maybe the thing about using GPUs for  anything like this.  It's not necessarily that it's so much faster by default.  Well, it depends a lot on your GPU.  But I'm pretty sure that on the laptop, I could probably need  the GPU for most of these things if I really, really  want to.  But it's so simple to get high performance.  Refreshing.  So let's see what we can do about this.  And when I say high quality, this is, of course, now I'll  try not to diss a lot of other people too much.  This is one of the wrong things you can do.  I'm sure, I mean, this is the graphics  deck of the bedroom, right?  I'm sure everyone knows this, but I will say it nevertheless.  The pixel values that we store in our images are not  linear amounts of light.  They are a gamma compressed format, which means that  they have some sort of curve.  And if you do any sort of arithmetic on them, you need  to convert them into the real value first.  If you don't do this, you end up in like a situation here on  the right.  Or I've faded it like 50-50 together an image that is dark  and light.  And you can especially observe that sort of the wrong  things come through.  Like this right spot here is way too visible.  These edges here are just way too visible.  It's not right.  The left one is how it's supposed to look like.  This is how you'd look if you took physical grains of  film and put them on top of each other.  So this is one wrong thing you can do, not converting in  the right place.  And of course, the example code that I showed does not do  this correctly, of course, because it does the simple  thing.  You can use the wrong color spaces.  Now, I never trust beamers.  I sort of exaggerated the effect here.  I've done it three times.  But we talk about RGB.  We never really say what red, what green, what blue.  And it turns out there are a number of different  interpretations of this.  And if you don't use the right one, your image on the left  can end up like the image on the right with wrong colors in  some way.  So you really need to care about this sort of thing.  This is not that visible, so I've zoomed it in.  This is what happens if you put your chroma planes at the  wrong site.  There are a lot of different standards, or eight  standards.  If you put them wrong, you get these color fringes around  here.  So you need to really get this right.  And before you ask, a lot of current in this video software  does this wrong.  The effects are subtle, but you will see them if you know  where to look.  And finally, when I talk about quality, I mean more than  just image output quality.  Quality is not a feature in itself.  It really is a process.  It really is a commitment to how we want to do things.  Move it as a unit test.  Then ask the tests that you can actually say, if the test  runs in your machine, most likely everything would work.  And I really, really try to say this as, when you think  about you want to make something that works and  doesn't give people the wrong idea of how to use it, you  really want to convince yourself you're  doing the wrong things.  A lot of these things seem really, really simple.  They're like, don't do mistakes.  But they are mind-numbly hard.  They're so easy to get a one-pixel placement wrong.  Like, do I add one?  Do I subtract one?  Do I divide by two?  Do I multiply by two?  My education is in multimedia DSP, and I still get these  things wrong.  I try to have tests, and every time I write a test, I  feel like this is a way too stupid test.  This cannot possibly fail, and it fails.  You might be perfect programmers.  I'm not.  So again, 90% cold coverage.  So let's see how it is to use.  Move it.  Use a chain of effects.  I usually say that if your hello world is more than  five lines long, you've failed.  So in a sense, I've failed.  But I'll walk you through a bit what you do.  First, you define an input.  You say that it's in some color space.  You say it's in some gamma.  You define, well, OK, the order is VGA.  It's unsigned by all this.  You add it.  You add an effect.  In this case, a glow effect.  You add an output of some format.  You set how many output bits you want.  And you finalize it.  Now the chain is locked.  Now it gets compiled, uploaded to your GPU.  You can run it very, very fast, many times you want to run.  You just set, oh, here are some new pixels.  Render just green, this is how your main loop typically looks  like.  And doing this will create this chain here internally.  This is a debug dump from where move it works.  You have an input.  You have a glow effect.  And you have an output.  And then internally, the first thing move it does, it's  right is off to some big hideous monster.  This is, well, a glow effect.  That's sort of, I want to cut out the highlights.  I want to blur it horizontally.  I want to blur it vertically.  And I want to add it back to the first thing.  And then, this is, of course, way, way, way too small to  read.  I'm not expecting you to read this.  But after 19 passes of this mini compiler thing, it has  inserted all these gamma conversions that you need.  It's split things up the faces.  You can see the different colors here.  These are different faces, because sometimes it's  advantageous not to do everything change.  And it's eventually created programs for you.  And all of that will turn this image here to this image here.  More effects.  I don't have a lot of effects in it.  And the reason is I would rather have good effects than  many effects.  So each one of them, there are only like 10 or 12 currently.  There will be more.  But each of them are fast.  They work.  They don't give you any surprises.  They look good.  Vignette, fair and simple.  Just dark in the corners of it through some cosine squared  law.  This is sharpened.  This is not an unsharpened mask.  This is actually a deconvolution that tries to  invert the transfer function of the lens.  So you don't get this halo effect that you get.  I don't think you can see it to the well on the slide,  though, unfortunately.  But usually it's better than a sharp mask.  Saturation, desaturation.  Observe again that since we are operating in linear space,  which we should do, the image as a whole does not get more  darker or lighter.  It just becomes more or less desaturated.  And finally, this isn't actually in yet.  This is lens blur through FFT.  It actually does a full convolve with whatever you want.  And of course, when you have convolve, you can also go  crazy and convolve with a star or whatever you want to.  If you want a hexagonal blur, then fine by me.  So demo time.  And I really hope that my demo actually works in this  resolution.  It does.  This is some sort of, I mean, these widgets here are just  like something I threw together.  They're not meant to be a UI thing.  But they're like the standard color correction.  So we'll say you can find in almost anything except Linux.  So you can sort of drag your mid-tones around, your  highlights around, and you can make it as ugly as you want to.  But of course, the demo app is boring.  This is KDN Live.  This is, at least if you ask me, the only usable video editor  on Linux right now.  I hope there's no blender of people here to take offense.  And this is like my early alpha prehacked version that  actually uses MoveIt through a project called MLT.  Of course, I mean, I'm pulling it down a clip here.  It plays right now.  MoveIt isn't really doing a lot.  It's only doing color space conversion and  things like that.  But I can pull out an effect here.  Let's say I want to blur it.  And I can blur more or less.  And you can see this happens in 60 FPS.  This is 720p.  It's not, I mean, it's a small window, but it's not  cheating.  It's actually editing in 720p and just  sailing down for display.  If I'm bored with the blur, of course, I can do some sort of  fade, which will work properly.  I can, of course, also do white balance.  I just need to find some gray point in this clip here.  Let's just focus on here.  This is gray.  And now gray actually stays gray.  There are multiple Linux white balance things where gray  does not become gray.  Because no one ever wrote a test for it and it never worked.  I'm not kidding you.  This is frustrating.  You can change the color temperature.  It's super, super warm.  It still doesn't become warped in any way.  It doesn't clip.  Again, because we're working in linear space.  And you can now sort of play.  I think I forgot to sort of reverse lag on the clip, so  it's fading the wrong way.  This is a UISU, right?  So again, this is, I think, well, in the fade, I don't  think I can do 60 FPS.  I think this is 30 FPS, because this is a four-year-old  ultra-portable laptop.  This is not something high-end in any way.  And I think that's mostly because we're decoding the  video on the CPU.  So really, I mean, if I wanted to do this in the current  software world, this would be five FPS, maybe.  And now it's 60.  And this also means, of course, that if you have a much  better machine, like say, any mid-range and media card, you  can probably go 4K video.  Directly on the GPU, no tricks.  Just do things, move around, do whatever you want.  I think this was the demo.  Yes, future work is always the best part, of course.  There will be more filters, but they will be added again  slowly, because I don't want a lot of filters that don't  work.  There are some limitations in the YCBTR handling, because  it's actually surprisingly subtle to get all the scaling  correctly, all the interpolation.  I want better interpolation in today's contents.  How many people here like Interlaced?  No hands, right?  And unfortunately, of course, Interlaced's video is a  reality many of us still have to deal with.  And even though it's sort of the modern video toolkit, I  think I might actually have to sort of succumb a bit here  and support it eventually.  And then, of course, whatever clients need, because  Moved is not a program.  Moved is used by currently MLT, which is like this generic  video library thing that can do all sorts of things, which is  turned used by ShotCuts.  KDN Live, probably a few others, Flowblade.  So again, whatever sort of clients come to me and say,  oh, I need a shared library.  This is the kind of things that, obviously, I want to  support.  So you will find that this KDN Live patches have not been  released until now.  I just uploaded them like an hour ago.  So if you're really, really brave and happen to have a  machine that's sort of like mine, I can promise you it's  stopped hanging the Intel drivers in the kernel.  It's used to.  I think it might actually not segfault on a video all the  time.  So if you're brave, you can go and get the library there.  You can get the KDN Live patches there and edit and  report tons of bugs to me.  So questions?  Please hand up for the mic.  I have two questions for you.  The first one is, do you hams at providing a plug-in for  just Dreamer because it's a very useful tool for making  pipes of content?  That's a plug-in for?  Just Dreamer.  Gstreamer.  Not immediately.  I would be very, very happy if someone wanted to add  move support to Gstreamer.  I don't unfortunately know Gstreamer very well, per se.  One challenge here, though, is that for best possible  move it performance, you really, really want to be able  to chain things together.  So if you have three successive move it filters, you want them  with one Gstreamer component.  You don't want them as three different ones.  Because then you have to go all the way to the GPU, do  something, go down to memory, go down to the CPU, and up  again, which really, really hurts performance.  A lot of the work that's gone into this is about keeping it  on the GPU, chaining it efficiently.  But if you want to do Gstreamer support and know Gstreamer  much better than I do, then I'll be happy to talk to you.  Another point is, as a user, I'm looking for a tool that  uses GPU to do the transcoding of video format.  Are you aware of some?  Or maybe you're projecting some time, I don't know.  So you're talking about the GPU to transcode, right?  And there are two ways here.  There's video decode, which is pretty well supported now.  At least if you have NVIDIA, you can do NVIDIA Po.  My KDNL and Pi patch doesn't support it right now.  But it can certainly be done.  It can play video, right?  Now, encoding the other way is much, much more tricky.  That's a very involved thing.  You will be aware that Intel has something called Quick Sync,  Quick Sync, which essentially does a live H.264 encode.  There are some commercial GPU video encoders.  I don't think there are any Linux ones.  Basically, this is because this is a very, very complicated  thing, right?  I hear there's talk for H.264 to use OpenCL to try to  accelerate.  It already does it.  OK, someone says that it already.  It crashes a lot.  It's very bad.  It works very minimally.  OK, just to repeat for the stream, there is the message  here that it only does it for a look ahead, but it's not  really done yet, right?  It's given up, like.  Oh, they've given up even.  OK.  It's not mentioned in the video.  OK, so I think the answer for latter one is no.  I don't think there's a lot of good things right now.  The good thing, of course, is that once your GPU is doing  all the pixel pushing, your CPU is free to handle the other  work.  Yes, on a question in the back.  Yeah, how is the chaining working?  Are you really chaining the effects in a program, like  a shader, or do you chain them by rendering to target and  texture from this?  They are mainly chained as in the program.  Most of it is run within one program.  There are a few cases where you need to split the chain.  One of them is when you do something like a blur, where  you sample like 15 different pixels right there, then you  want to sample from memory.  You don't want every single computation to be  run 15 times, right?  There are some other few cases where you need to  bounce to memory.  And that is especially when you have a split in your chain.  It needs to be used by two different things.  So again, I'll just go back to one of the earlier slides  here.  It's not so early slides, actually.  Here, right?  If you look at the colors here, you see the three first  ones are green.  Those are part of the same shader.  And that's the input, the gamma expansion, and actually  some alpha handling, because we handle alpha  correctly as well.  So in general, we try to combine them into one shader  as much as we can.  And we really rely on the shader compiler to optimize and  inline things away as much as we can.  I would love for the Intel compiler to be smarter about  things I hope it could be smarter for.  But I assume there are people in the room here who can  help me with that.  There are more questions both in the front and the back.  Good thing about not controlling the microphone is I  don't have to make these decisions.  I'll be coming.  I'm sorry.  I've got two questions.  The first is, do you support 10 and 12-bit video?  Yes.  Let me elaborate a bit on this.  I move it mostly uses 16-bit floating point  support internally.  This is not enough for doing 12-bit video with full  accuracy.  It is enough for doing 10-bit.  We have accuracy tests in most of the crucial effects to  check that we actually get within the right kicks level.  So yes, it is supported.  And this is clearly something I want to do for 12-bit.  We would have to go to 32-bit floating point.  This is not hard in itself.  But I think open source things don't really support it all  that much yet.  But it's certainly something I would love to support.  And it's very close to there.  Because we're working in full floating point all the time,  we don't have these 8-bit locks.  The other question was, if you did video decode on the GPU,  is it a zero copy operation from memory to, is there a trip  to the CPU and back to use it?  No.  No.  If you do video decoding on GPU, you can certainly put that  into texture with whatever extension you have available.  And then we can use it as an input.  Again, KDN Live doesn't currently do this.  But it would certainly be possible to do all the way  without ever touching the CPU.  There is a VD-Po extension for that.  VD-Po.ngl compact.  Yes.  And there's also, of course, now a VD-Po emulator for VA APIs.  You can run VD-Po things against your input card.  I've never tested it.  But you want something like this, right?  Especially if you look at a demo here now,  when it's doing a fade, it's CPU bound.  It's my CPU not able to decode more than two streams of 40  something FPS.  So first some comments on the previous question about GSTRIMA.  So I think this would be really fun to implement.  And maybe I'm going to look at that really soon.  It was very fun in MLT.  And well, then on Intel, you have with VA API the possibility  to do the decoding on the GPU and then get textures from that.  And in that case, you should also be able to have zero copy  processing directly here.  But now to my question, what's the input and output that you  can have with your toolkit?  Can you have GL textures as input and output?  And can you also have system memory as input and output?  I can tell you.  So Movid is plug-in based in the center.  You have a class that defines an input and an output.  And you have effects, of course, in the middle.  There is an input class that takes in a system memory.  And then, of course, loads it to a texture  and use it from there.  I don't actually think I have a texture input,  but this is trivial to port.  This is like a pen line to support.  It's not a problem at all.  As for rendering, Movid renders to whatever FBO we give it.  So you can either render to the screen.  That's FBO 0.  Or you can render to another FBO and then do a read pixels  down into the CPU.  So input and output there is sort of free.  OK, thank you.  Any more questions?  We have like 40 minutes to fill or something, right?  Yeah.  I have to fill it, right?  There's not a question down here.  OK.  What are your software requirements  to make Movid working?  You spoke about VDPo or?  No.  So my software requirements are, in a sense, very low.  You need a C++ compiler.  You need the eigenmatrix library.  You need OpenGL.  And by OpenGL, I mean any card that can do shaders, which  means any card from eight or nine years ago.  So there are no weird systems.  It certainly works on both Intel API and Vidya, for  instance.  It works.  I know people have it working on both Linux, Windows,  OS X.  I try to stick with, again, we're 2014.  It should be allowed to use C++, right?  But this is sort of the weirdest requirement, in a sense.  What about other video editors, such as, say, OpenShot?  Are they likely to include that?  OpenShot uses MLT, I think.  So it means that, what you need to do to use this MLT, you  need OpenGL complex.  Just you need to initialize OpenGL so I can play with it.  Apart from that, it's completely transparent to you.  MLT now handles all the upload and download things,  transparently, and change things together.  This was done by Dan Vanity, who was one of the MLT guys.  I don't know if he might be here or not.  I don't think so.  But in general, it's easy to get to start with.  If you want to play with it, you can go to Shotcut, which  also has full native support, and has for a while.  Thanks so much.  Again, any more questions?  OK, another one.  Yes?  You said you need OpenGL with shaders.  What about OpenGL ES, or GLS2 or something?  This is a good question.  I don't support the GLS right now.  I'm not sure if I actually will.  It depends a bit on what the requirements are.  What do you want to use it for?  Because editing video on your mobile is maybe not a use  case that I care that much about to begin with.  I try very hard to do only one thing and do it well.  And instead of trying to fragment myself, if you have a  compelling use case for GLS, which is for those who don't  know, like the mobile, newer, modern version of OpenGL, I  might consider it.  But I don't want to maintain a fork.  But I don't really require any magic things.  I only need to draw a quad and have a shader on it.  I've been playing a lot with robots and had many issues  by doing some video understanding, finding a  shape, finding a color, or whatever.  Do you think that move it can be used to, I mean, using  some filter to maybe only select the red things on an  image, and then being able to locate some shapes, or it  could be an extension.  Very interesting in the robot-y area.  I think my answer here will have to be no.  Just because this is, again, deviating from what I want to  do, and there's only one of me.  If you want to, you can most certainly use move it and write  your own effects.  But I'm not really sure what you would gain from it.  Most likely, if you have that kind of specialized computer  vision things, maybe you don't really want what move gives  you.  Maybe you just want to have your own shader.  You don't want to blur.  You want some sort of specialized image  recognition thing.  So then maybe move it.  It's not for you.  This is really about, for me at least, it's really about  open source graphics.  I think we are out of questions.  Thank you, Ben.  Thanks.  Thank you.  So the next presentation is going to be about the  uploading of video on Wayland.  Thank you.  Thank you.
_FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_0_5711.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_5712_14135.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_14136_15791.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_15792_16607.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_16608_17099.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_17100_17183.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_17184_26579.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_26580_41963.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_41964_42011.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_42012_42755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_42756_43535.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_43536_45395.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_45396_45443.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_45444_45503.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_45504_45755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_45756_46895.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_46896_47255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_47256_47651.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_47652_48011.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_48012_48755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_48756_49139.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_49140_50255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_50256_50651.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_50652_51755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_51756_52139.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_52140_53255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_53256_54011.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_54012_54755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_54756_55139.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_55140_55883.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_55884_56255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_56256_56639.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_56640_57383.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_57384_57755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_57756_58139.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_58140_58883.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_58884_59255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_59256_59639.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_59640_60383.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_60384_60755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_60756_61139.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_61140_61883.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_61884_62255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_62256_63755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_63756_64139.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_64140_64523.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_64524_65255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_65256_66755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_66756_68255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_68256_69023.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_69024_69383.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_69384_69755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_69756_70523.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_70524_71255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_71256_71639.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_71640_72383.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_72384_72755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_72756_73139.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_73140_73511.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_73512_75011.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_75012_75755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_75756_75803.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_75804_76511.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_76512_77255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_77256_78011.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_78012_78755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_78756_78803.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_78804_79511.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_79512_80255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_80256_81011.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_81012_81755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_81756_81803.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_81804_82511.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_82512_83255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_83256_84383.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_84384_84755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_84756_85139.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_85140_87755.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_87756_89255.png _FOSDEM_2014__Movit__High-speed__high-quality_video_filters_on_the_GPU_IYWKyqyA8Rw_89256_89349.png
The way back after the second lunch break. My name is Stanna. I will be talking about MoveIt, which is my newest hobby project, more or less. And as you can see, it has this sort of pun in it. But it stands for the modern video toolkit. And this is, of course, dangerous because anything called modern usually isn't. Anything called toolkit is usually hideous. But it tries to be a high-performance, high-quality open-source library for video filters. And let me actually talk a bit first. What do I mean by high-performance? And to do that, we'll go back 25 years in time and see. This is NCC. I'm sorry to find this a bit small because there will be more things. But this is actually 25 years ago. NCC has held up remarkably well. But this is how you do a fade if you were programming this in 1989. I've stripped out all the complexity. This is only gray scale. There's no alpha. There's no nothing. But you take in an image A and image B. You want to fade between them. You multiply one by some fade constant F. And you multiply one by the other constant. And then you add them together. It's plain and simple. And then someone figures out, ah, maybe this is a bit slow. So we rearrange it a bit. We move things around. So we only have one multiply. And this is also a standard trick. And then people come along and say, it's still too slow. And the reason why it's so slow is we're using floating point. And it's not really the floating point itself. It's slow. Floating point has been really, really fast for most of the last at least the last 20 years. The thing is, slow is moving things back and forth from its two floats. So we're doing fixed point. We're just multiplying things a bit more tricky. We need to round a bit. We need to shift a bit. But it's still like the same thing. Now you come to MMX. 1997 around then Intel figured out that people want to do the same things to many different values. We have all these pixels. We have all these pixels over here. Let's do them four at a time. The problem, of course, is it becomes very, very complicated. I wrote this. I used at least half an hour to get it to work at all. It's probably not very efficient. It's even worse if you want to write like pure assembler. And before you talk about auto vectorization, I can tell you no compiler in the world will auto vectorize the previous things, this one, because we reduced position. And in general, auto vectorization is not something you can really rely on. So you need to write something like this. And of course, right at the bottom is still our old friend, the scalar thing for fallback for the last few pixels. And then we invent not only MMX, but SSE, SC2, AVX, AVX2. I hear there's now coming AVX512. And you need to write code for every single one of these. And then Intel comes along and says, hey, we have multiple cores. We cannot do more slow anymore. We need you to split your work. OK, so you need to do like P threads, whatever, fire up threads. And if one of them is slow, you will have a problem. So you need to distribute it somehow more intelligent than I have done. And this is, of course, on top of all the SSE, SC2, AVX, whatever, whatever, whatever. OK, so we've figured this is 10 years ago, right? We have multicore. We've figured out how to do this. Let me now show you how people actually do this. This is 2006. This is from a project called Frylick. If you have ever edited video on your Linux machine, you are most likely fading using this software or something very much like it. It is still, as you can see, completely scalar. There's no AVX. That's not support multicore. So what the heck? It turns out this model of doing things efficiently on the CPU is very, very complicated. So people don't do it. At least we don't in the free software world. Maybe you do so if you're on Windows, but this is not reality. So fast forwarded to 2014 in GPUs, you cannot really buy a desktop machine anymore without a GPU. And if you're lucky, you get a cat. Actually, you cannot buy a laptop anymore without having a GPU and maybe a cat. You can't buy a cell phone anymore without a reasonably powerful GPU. And this is a bottle-deposited machine. It has a GPU. So they are no longer uncommon. We should not anymore assume the user does not have a GPU. So how does it program it? This is GLSL. This is the most common shading language that you would use to program your GPU. And this is the four lines you need to do everything the CPU version did with full vectorization, with full splitting, with full handling of all edge cases, yada, yada, yada. And Movid uses GLSL. If you would write this in Movid, you need only a little more glue around it. This is basically because Movid is able to chain many of these things together. So it just needs some namespacing and things like this. It is refreshingly simple. And I think this is maybe the thing about using GPUs for anything like this. It's not necessarily that it's so much faster by default. Well, it depends a lot on your GPU. But I'm pretty sure that on the laptop, I could probably need the GPU for most of these things if I really, really want to. But it's so simple to get high performance. Refreshing. So let's see what we can do about this. And when I say high quality, this is, of course, now I'll try not to diss a lot of other people too much. This is one of the wrong things you can do. I'm sure, I mean, this is the graphics deck of the bedroom, right? I'm sure everyone knows this, but I will say it nevertheless. The pixel values that we store in our images are not linear amounts of light. They are a gamma compressed format, which means that they have some sort of curve. And if you do any sort of arithmetic on them, you need to convert them into the real value first. If you don't do this, you end up in like a situation here on the right. Or I've faded it like 50-50 together an image that is dark and light. And you can especially observe that sort of the wrong things come through. Like this right spot here is way too visible. These edges here are just way too visible. It's not right. The left one is how it's supposed to look like. This is how you'd look if you took physical grains of film and put them on top of each other. So this is one wrong thing you can do, not converting in the right place. And of course, the example code that I showed does not do this correctly, of course, because it does the simple thing. You can use the wrong color spaces. Now, I never trust beamers. I sort of exaggerated the effect here. I've done it three times. But we talk about RGB. We never really say what red, what green, what blue. And it turns out there are a number of different interpretations of this. And if you don't use the right one, your image on the left can end up like the image on the right with wrong colors in some way. So you really need to care about this sort of thing. This is not that visible, so I've zoomed it in. This is what happens if you put your chroma planes at the wrong site. There are a lot of different standards, or eight standards. If you put them wrong, you get these color fringes around here. So you need to really get this right. And before you ask, a lot of current in this video software does this wrong. The effects are subtle, but you will see them if you know where to look. And finally, when I talk about quality, I mean more than just image output quality. Quality is not a feature in itself. It really is a process. It really is a commitment to how we want to do things. Move it as a unit test. Then ask the tests that you can actually say, if the test runs in your machine, most likely everything would work. And I really, really try to say this as, when you think about you want to make something that works and doesn't give people the wrong idea of how to use it, you really want to convince yourself you're doing the wrong things. A lot of these things seem really, really simple. They're like, don't do mistakes. But they are mind-numbly hard. They're so easy to get a one-pixel placement wrong. Like, do I add one? Do I subtract one? Do I divide by two? Do I multiply by two? My education is in multimedia DSP, and I still get these things wrong. I try to have tests, and every time I write a test, I feel like this is a way too stupid test. This cannot possibly fail, and it fails. You might be perfect programmers. I'm not. So again, 90% cold coverage. So let's see how it is to use. Move it. Use a chain of effects. I usually say that if your hello world is more than five lines long, you've failed. So in a sense, I've failed. But I'll walk you through a bit what you do. First, you define an input. You say that it's in some color space. You say it's in some gamma. You define, well, OK, the order is VGA. It's unsigned by all this. You add it. You add an effect. In this case, a glow effect. You add an output of some format. You set how many output bits you want. And you finalize it. Now the chain is locked. Now it gets compiled, uploaded to your GPU. You can run it very, very fast, many times you want to run. You just set, oh, here are some new pixels. Render just green, this is how your main loop typically looks like. And doing this will create this chain here internally. This is a debug dump from where move it works. You have an input. You have a glow effect. And you have an output. And then internally, the first thing move it does, it's right is off to some big hideous monster. This is, well, a glow effect. That's sort of, I want to cut out the highlights. I want to blur it horizontally. I want to blur it vertically. And I want to add it back to the first thing. And then, this is, of course, way, way, way too small to read. I'm not expecting you to read this. But after 19 passes of this mini compiler thing, it has inserted all these gamma conversions that you need. It's split things up the faces. You can see the different colors here. These are different faces, because sometimes it's advantageous not to do everything change. And it's eventually created programs for you. And all of that will turn this image here to this image here. More effects. I don't have a lot of effects in it. And the reason is I would rather have good effects than many effects. So each one of them, there are only like 10 or 12 currently. There will be more. But each of them are fast. They work. They don't give you any surprises. They look good. Vignette, fair and simple. Just dark in the corners of it through some cosine squared law. This is sharpened. This is not an unsharpened mask. This is actually a deconvolution that tries to invert the transfer function of the lens. So you don't get this halo effect that you get. I don't think you can see it to the well on the slide, though, unfortunately. But usually it's better than a sharp mask. Saturation, desaturation. Observe again that since we are operating in linear space, which we should do, the image as a whole does not get more darker or lighter. It just becomes more or less desaturated. And finally, this isn't actually in yet. This is lens blur through FFT. It actually does a full convolve with whatever you want. And of course, when you have convolve, you can also go crazy and convolve with a star or whatever you want to. If you want a hexagonal blur, then fine by me. So demo time. And I really hope that my demo actually works in this resolution. It does. This is some sort of, I mean, these widgets here are just like something I threw together. They're not meant to be a UI thing. But they're like the standard color correction. So we'll say you can find in almost anything except Linux. So you can sort of drag your mid-tones around, your highlights around, and you can make it as ugly as you want to. But of course, the demo app is boring. This is KDN Live. This is, at least if you ask me, the only usable video editor on Linux right now. I hope there's no blender of people here to take offense. And this is like my early alpha prehacked version that actually uses MoveIt through a project called MLT. Of course, I mean, I'm pulling it down a clip here. It plays right now. MoveIt isn't really doing a lot. It's only doing color space conversion and things like that. But I can pull out an effect here. Let's say I want to blur it. And I can blur more or less. And you can see this happens in 60 FPS. This is 720p. It's not, I mean, it's a small window, but it's not cheating. It's actually editing in 720p and just sailing down for display. If I'm bored with the blur, of course, I can do some sort of fade, which will work properly. I can, of course, also do white balance. I just need to find some gray point in this clip here. Let's just focus on here. This is gray. And now gray actually stays gray. There are multiple Linux white balance things where gray does not become gray. Because no one ever wrote a test for it and it never worked. I'm not kidding you. This is frustrating. You can change the color temperature. It's super, super warm. It still doesn't become warped in any way. It doesn't clip. Again, because we're working in linear space. And you can now sort of play. I think I forgot to sort of reverse lag on the clip, so it's fading the wrong way. This is a UISU, right? So again, this is, I think, well, in the fade, I don't think I can do 60 FPS. I think this is 30 FPS, because this is a four-year-old ultra-portable laptop. This is not something high-end in any way. And I think that's mostly because we're decoding the video on the CPU. So really, I mean, if I wanted to do this in the current software world, this would be five FPS, maybe. And now it's 60. And this also means, of course, that if you have a much better machine, like say, any mid-range and media card, you can probably go 4K video. Directly on the GPU, no tricks. Just do things, move around, do whatever you want. I think this was the demo. Yes, future work is always the best part, of course. There will be more filters, but they will be added again slowly, because I don't want a lot of filters that don't work. There are some limitations in the YCBTR handling, because it's actually surprisingly subtle to get all the scaling correctly, all the interpolation. I want better interpolation in today's contents. How many people here like Interlaced? No hands, right? And unfortunately, of course, Interlaced's video is a reality many of us still have to deal with. And even though it's sort of the modern video toolkit, I think I might actually have to sort of succumb a bit here and support it eventually. And then, of course, whatever clients need, because Moved is not a program. Moved is used by currently MLT, which is like this generic video library thing that can do all sorts of things, which is turned used by ShotCuts. KDN Live, probably a few others, Flowblade. So again, whatever sort of clients come to me and say, oh, I need a shared library. This is the kind of things that, obviously, I want to support. So you will find that this KDN Live patches have not been released until now. I just uploaded them like an hour ago. So if you're really, really brave and happen to have a machine that's sort of like mine, I can promise you it's stopped hanging the Intel drivers in the kernel. It's used to. I think it might actually not segfault on a video all the time. So if you're brave, you can go and get the library there. You can get the KDN Live patches there and edit and report tons of bugs to me. So questions? Please hand up for the mic. I have two questions for you. The first one is, do you hams at providing a plug-in for just Dreamer because it's a very useful tool for making pipes of content? That's a plug-in for? Just Dreamer. Gstreamer. Not immediately. I would be very, very happy if someone wanted to add move support to Gstreamer. I don't unfortunately know Gstreamer very well, per se. One challenge here, though, is that for best possible move it performance, you really, really want to be able to chain things together. So if you have three successive move it filters, you want them with one Gstreamer component. You don't want them as three different ones. Because then you have to go all the way to the GPU, do something, go down to memory, go down to the CPU, and up again, which really, really hurts performance. A lot of the work that's gone into this is about keeping it on the GPU, chaining it efficiently. But if you want to do Gstreamer support and know Gstreamer much better than I do, then I'll be happy to talk to you. Another point is, as a user, I'm looking for a tool that uses GPU to do the transcoding of video format. Are you aware of some? Or maybe you're projecting some time, I don't know. So you're talking about the GPU to transcode, right? And there are two ways here. There's video decode, which is pretty well supported now. At least if you have NVIDIA, you can do NVIDIA Po. My KDNL and Pi patch doesn't support it right now. But it can certainly be done. It can play video, right? Now, encoding the other way is much, much more tricky. That's a very involved thing. You will be aware that Intel has something called Quick Sync, Quick Sync, which essentially does a live H.264 encode. There are some commercial GPU video encoders. I don't think there are any Linux ones. Basically, this is because this is a very, very complicated thing, right? I hear there's talk for H.264 to use OpenCL to try to accelerate. It already does it. OK, someone says that it already. It crashes a lot. It's very bad. It works very minimally. OK, just to repeat for the stream, there is the message here that it only does it for a look ahead, but it's not really done yet, right? It's given up, like. Oh, they've given up even. OK. It's not mentioned in the video. OK, so I think the answer for latter one is no. I don't think there's a lot of good things right now. The good thing, of course, is that once your GPU is doing all the pixel pushing, your CPU is free to handle the other work. Yes, on a question in the back. Yeah, how is the chaining working? Are you really chaining the effects in a program, like a shader, or do you chain them by rendering to target and texture from this? They are mainly chained as in the program. Most of it is run within one program. There are a few cases where you need to split the chain. One of them is when you do something like a blur, where you sample like 15 different pixels right there, then you want to sample from memory. You don't want every single computation to be run 15 times, right? There are some other few cases where you need to bounce to memory. And that is especially when you have a split in your chain. It needs to be used by two different things. So again, I'll just go back to one of the earlier slides here. It's not so early slides, actually. Here, right? If you look at the colors here, you see the three first ones are green. Those are part of the same shader. And that's the input, the gamma expansion, and actually some alpha handling, because we handle alpha correctly as well. So in general, we try to combine them into one shader as much as we can. And we really rely on the shader compiler to optimize and inline things away as much as we can. I would love for the Intel compiler to be smarter about things I hope it could be smarter for. But I assume there are people in the room here who can help me with that. There are more questions both in the front and the back. Good thing about not controlling the microphone is I don't have to make these decisions. I'll be coming. I'm sorry. I've got two questions. The first is, do you support 10 and 12-bit video? Yes. Let me elaborate a bit on this. I move it mostly uses 16-bit floating point support internally. This is not enough for doing 12-bit video with full accuracy. It is enough for doing 10-bit. We have accuracy tests in most of the crucial effects to check that we actually get within the right kicks level. So yes, it is supported. And this is clearly something I want to do for 12-bit. We would have to go to 32-bit floating point. This is not hard in itself. But I think open source things don't really support it all that much yet. But it's certainly something I would love to support. And it's very close to there. Because we're working in full floating point all the time, we don't have these 8-bit locks. The other question was, if you did video decode on the GPU, is it a zero copy operation from memory to, is there a trip to the CPU and back to use it? No. No. If you do video decoding on GPU, you can certainly put that into texture with whatever extension you have available. And then we can use it as an input. Again, KDN Live doesn't currently do this. But it would certainly be possible to do all the way without ever touching the CPU. There is a VD-Po extension for that. VD-Po.ngl compact. Yes. And there's also, of course, now a VD-Po emulator for VA APIs. You can run VD-Po things against your input card. I've never tested it. But you want something like this, right? Especially if you look at a demo here now, when it's doing a fade, it's CPU bound. It's my CPU not able to decode more than two streams of 40 something FPS. So first some comments on the previous question about GSTRIMA. So I think this would be really fun to implement. And maybe I'm going to look at that really soon. It was very fun in MLT. And well, then on Intel, you have with VA API the possibility to do the decoding on the GPU and then get textures from that. And in that case, you should also be able to have zero copy processing directly here. But now to my question, what's the input and output that you can have with your toolkit? Can you have GL textures as input and output? And can you also have system memory as input and output? I can tell you. So Movid is plug-in based in the center. You have a class that defines an input and an output. And you have effects, of course, in the middle. There is an input class that takes in a system memory. And then, of course, loads it to a texture and use it from there. I don't actually think I have a texture input, but this is trivial to port. This is like a pen line to support. It's not a problem at all. As for rendering, Movid renders to whatever FBO we give it. So you can either render to the screen. That's FBO 0. Or you can render to another FBO and then do a read pixels down into the CPU. So input and output there is sort of free. OK, thank you. Any more questions? We have like 40 minutes to fill or something, right? Yeah. I have to fill it, right? There's not a question down here. OK. What are your software requirements to make Movid working? You spoke about VDPo or? No. So my software requirements are, in a sense, very low. You need a C++ compiler. You need the eigenmatrix library. You need OpenGL. And by OpenGL, I mean any card that can do shaders, which means any card from eight or nine years ago. So there are no weird systems. It certainly works on both Intel API and Vidya, for instance. It works. I know people have it working on both Linux, Windows, OS X. I try to stick with, again, we're 2014. It should be allowed to use C++, right? But this is sort of the weirdest requirement, in a sense. What about other video editors, such as, say, OpenShot? Are they likely to include that? OpenShot uses MLT, I think. So it means that, what you need to do to use this MLT, you need OpenGL complex. Just you need to initialize OpenGL so I can play with it. Apart from that, it's completely transparent to you. MLT now handles all the upload and download things, transparently, and change things together. This was done by Dan Vanity, who was one of the MLT guys. I don't know if he might be here or not. I don't think so. But in general, it's easy to get to start with. If you want to play with it, you can go to Shotcut, which also has full native support, and has for a while. Thanks so much. Again, any more questions? OK, another one. Yes? You said you need OpenGL with shaders. What about OpenGL ES, or GLS2 or something? This is a good question. I don't support the GLS right now. I'm not sure if I actually will. It depends a bit on what the requirements are. What do you want to use it for? Because editing video on your mobile is maybe not a use case that I care that much about to begin with. I try very hard to do only one thing and do it well. And instead of trying to fragment myself, if you have a compelling use case for GLS, which is for those who don't know, like the mobile, newer, modern version of OpenGL, I might consider it. But I don't want to maintain a fork. But I don't really require any magic things. I only need to draw a quad and have a shader on it. I've been playing a lot with robots and had many issues by doing some video understanding, finding a shape, finding a color, or whatever. Do you think that move it can be used to, I mean, using some filter to maybe only select the red things on an image, and then being able to locate some shapes, or it could be an extension. Very interesting in the robot-y area. I think my answer here will have to be no. Just because this is, again, deviating from what I want to do, and there's only one of me. If you want to, you can most certainly use move it and write your own effects. But I'm not really sure what you would gain from it. Most likely, if you have that kind of specialized computer vision things, maybe you don't really want what move gives you. Maybe you just want to have your own shader. You don't want to blur. You want some sort of specialized image recognition thing. So then maybe move it. It's not for you. This is really about, for me at least, it's really about open source graphics. I think we are out of questions. Thank you, Ben. Thanks. Thank you. So the next presentation is going to be about the uploading of video on Wayland. Thank you. Thank you.
to be a high-performance, high-quality open-source library for video filters. And let me actually talk a bit first. What do I mean by let's see what we can do about this. And when I showed does not do this correctly, of course, to blur it horizontally. I want to blur it editing in 720p and just sailing down for into this is about keeping it on the GPU, this, right? Especially if you look at a demo what you need to do to use this MLT, you need can go to Shotcut, which also has full native what the requirements are. What do you want to gives you. Maybe you just want to have your own