Fearless Multimedia Programming
Ali, Zeeshan
Whether you are interested in multimedia programming in specific or curious about how Rust programming language can enable you to write almost-bug-free code without having to compromise on efficiency, this talk is for you. GStreamer is a popular framework of choice for multimedia programming in the Linux world, especially for embedded. Since efficiency is a typical core requirement for embedded solutions, traditionally C/C++ have been the languages of choice for writing GStreamer applications and plugins. Unfortunately, this efficiency comes at the price of safety. Even the most skilled C/C++ developers make mistakes with memory management and the results could potentially be catastrophic. Thread safety is another aspect that is central to multimedia solutions but is extremely difficult to achieve with C/C++. Rust language is designed to be both efficient and safe at the same time. In this talk, Zeeshan will present how GStreamer's Rust bindings not only make multimedia programming a lot safer, easier and fun but also allow developers to write even more efficient code than C/C++ in certain cases. While the talk will be focused on multimedia and GStreamer in specific, most of the concepts presented shall be very generic.
 Welcome to my talk about fearless multimedia. For the title of the talk, I'm pretty nervous, so not so fearless. Who am I? My name is Zishan Ali, and I work for a company called Collabra. Legally, I'm a freelancer, but I have only one client only, so it's kind of I'm working for them. And both my company and me, both my companies, the one man company and Collabra and me, we all work on FOSS. I have always been involved in FOSS, and mainly GNOME development has been my thing. I've worked on other things as well, mostly related to GNOME, but not always. And my hobbies include cats and flying, not the toys that people fly nowadays, but the real things. And what am I going to talk about is G-streamer and Rust, both of them combined. And I have to cover both of these topics, and they're pretty hard to understand, like, everything about it, all the essential things. So I'll try to be very fast. But if I'm not explaining something well, please just interrupt and tell me, and I can try to explain better before I move forward. So it's inspired by another talk by Sebastian Droeg. He's doing most of the G-streamer and Rust, and he has been a long-time G-streamer hacker, and he's really competent in both these technologies, so he does most of this work. You can think of me as a replacement of him in this talk. And it's inspired by his talks that he has been giving on this topic. First of all, let's talk about Rust. Before I move forward, how many of you know both Rust and G-streamer? Like just basics? Yeah, no off, like what it is. Okay, not many. Let's start with Rust. How many people know basics of Rust? Quite a lot, so I won't explain a lot. I'll just go very fast. So Rust is a systems programming language, and it's one of the first in its kind that it focuses on both safety and efficiency at the same time. There has been a lot of programming languages, as you all know. Many of them focus on safety, and they are pretty safe, and some are safer than the others, but most of them haven't been very efficient at system utilization. And then the programming languages are very efficient. They are never safe, like C and C++ mainly. They are extremely unsafe languages. Rust is the first one to combine them, or maybe there are other languages I don't know of them, but Rust is one of them. One of the main things about Rust is that it has a concept of zero cost abstraction. That is to say that it provides high-level APIs and concepts to easily achieve many things, many, for example, hash tables and lists and all the data types you're used to. It will provide those APIs for you, but the concept is that even though they're high-level and it's easy, they don't have any runtime costs because of that, which is associated with the same libraries, APIs, and other programming languages. And with regards to safety, one of the first things about it is that it has no null pointers, no dangling pointers allowed. Actually, you can't really call the pointers in Rust as pointers, but I'll talk about it a bit later as well. So a lot of problems that arise from pointer handling in other languages like C and C++, they don't exist in Rust. But sometimes you need to interact with unsafe code, like, for example, when you're interacting with some C code or C++ code, you have to do a foreign function interface, and that is by its nature unsafe because you cannot make the C code that is written safe because it's not written in Rust. So for that, Rust allows you to have an unsafe syntax. Like, it has a keyword and you have all the whichever code you want to write, which is not safe. You put it in that, in those, in the brackets, in the context. So you have all the unsafe code, very isolated. So when you have a problem in your Rust code, you would know which code to be suspicious of, first of all, and most likely that will be the code that is causing the problems if you have, especially, some memory issues. So it isolates it and that's already a big achievement. And also it has other concepts like non-mutable state by default. So by default, all your state is non-mutable. But if you want mutation, then you mark it as mutable parameter or a variable. And then when you have the same applies to that, when you have a problem, usually it's with mutable states, so you should be suspicious of all the variables and arguments that are marked mutable and then move on from there. So it makes it a bit easier to debug the code if you have ever problems. And it has very strict ownership semantics. It's not a concept in C and C++. Well, in C, it doesn't exist. There is no such thing as ownership. If you have a pointer to something, you can do whatever you want with it. If you, in C++, you have some APIs in modern C++, especially the smart pointers. But that's something you can use. It's not like you have to use it. You can still do the same thing as in C that you don't really have an owner and you can just mess things up in a big way. And in other high-level languages, this problem is solved by the garbage collector. So you don't have to deal with memory. There is a garbage collector that deals it for you. If all the code that is pointing, that needs a particular resource that goes out of scope, a garbage collector frees it at some point for you. But it comes at a runtime cost. So if you don't want to pay that cost, you don't want to use those languages then. But in Rust, it's solved by the ownership concept. The concept is that you have only and only one owner at a time of a resource. That means that you cannot, not just from two threads, but from any two points in the code, you cannot have ownership at the same time of the same resource. And this, for example, and this sample, if you assign something to S1, a resource, and then later on you assign the same thing to S2, you have moved the ownership now to S2. So if you try to build this code, it won't work. Rust will complain that you have already moved S1 resource to S2. And now S1 doesn't exist anymore because you have moved the resource because you can't have two owners at the same time. But you can't work like that. If you have just this restriction, you won't be able to achieve much in Rust. So there are then exceptions that are safer. And then you use those exceptions. And one of the first one is borrowing. It's used a lot for function arguments in the APIs. All the APIs, you will always see things being borrowed. And it's similar to reference, passing by reference in C++. And what you do is you give the ownership, you borrow the ownership to another variable or an argument to a function temporarily, and then they can use it and then they return it. For example, if you pass it to a function, when it returns, you have the ownership back. It's not borrowed anymore. So for example, in this sample, which is a modification of the previous one, instead of moving the S1, you are giving a reference. You're borrowing a reference to S1 and S2. And now you can use them both at the same time. And since you borrowed it read-only, you can have multiple reference, multiple borrows of the same resource, because it doesn't matter how many borrows you have at the same time. It's all read-only. But the problem with borrows is it's temporary. So a lot of times you want to keep the reference. Like, for example, you pass a resource to a function that wants to put it in a structure and that it wants to keep the structure around, then you can't use borrows. And for that reason, you have the other data types in Rust that makes it possible. Like, for example, RCT, it's reference counted. So it adds reference counting to the resource. So you actually don't keep the actual resource. You don't keep a pointer to that. For example, in here, S1 is the pointer to RC type. And that is keeping the actual resource in it. And each time you want to have multiple accesses to the same resource, you just clone the RC. And when all these RC instances go out of scope, the actual resource will also be freed automatically for you. So you don't have to care about that. It's a Rust. I didn't mention that. It's operates on scopes. So when things go out of scope, compiler frees those things for you. And it decided at build time, so there's no garbage collection involved at runtime. And then, but the RC that I mentioned just now, it's only from single thread. So you can't use it from multiple threads. Because Rust wants to make sure that you have it so that it's not like, because it's not atomic. So you can't atomically increase the reference count or decrease the reference count. So it would be a bit unsafe to use it from multiple threads. So for that, we have a concept in Rust called fearless concurrency. Because if you do concurrent programming in Rust, you are safe. You don't have to be scared of it. In other programming languages, people are really scared of threads. And they should be. But in Rust, you don't have to be. Because it has API and concepts that helps you make it safe. And ARC is one of those types that it provides for multi-threading. And it's just RC, but it's atomic reference counting. So you can use it from multiple threads at the same time. Although it only gives you read-only access. So you can't modify for multiple threads. You can't get a mutable state from it. And for that reason, there's another data I've called mutics that allows you to get mutable access to the actual resource. I would normally explain with samples here. But I have to cover a lot of things. So I won't get into details. But if you have some questions afterwards, please let me know. And I can explain each of these things in detail to you in person. That gets us to GStreamer. What is GStreamer? It's a multimedia framework. So if you want to write anything related to multimedia, especially audio and video, this is the framework you would want to use. It's the multimedia framework of choice for all Linux platforms. And it has some simple concepts, like elements and pipelines. So you create a pipeline for each multimedia application. And in that pipeline, you put elements together, depending on your needs. There's a concept of source and sync and filter. And you would connect those elements. It's a bit like LEGOs. So you have different elements, and you just put them together, and you achieve a goal. I have an example here so that it's easier to understand what I'm saying. For example, if you want to write a player for OGG files, video files, which has audio and video both multiplex into it, does everyone know what OGG is? And Warbis is? Good. So it's simple. You put a file source because you want to first read from a file. You connect it to a demuxer, which demultiplexes the audio and video parts. And then you connect both those parts to elements that decode both those formats. And once they are decoded, they need to be played. And for playing, you have syncs, different kinds of syncs. So I can't barely see it here. So we do sync, and here it's kept generic because we have multiple video syncs, real end and X and all those. And also, the audio syncs are, there's a lot of them. So there's no specific example here. But anyway, the concept is that in each element, you have something called pads. The blue things you see in here, they're called pads. And you don't actually connect elements directly, but you connect them through these pads. And each pad is of type, either source or sync. And that means that you can only connect source to a sync. You can't just connect source and source and sync and sync because that doesn't make sense. And all the real multimedia content, they flow only in one direction. There can be events and other things that flow upstream, but all the main multimedia content always flows downstream and through the source and sync. One example that I would show later, it would require this information about something called capabilities. It's called GStreamer caps in the GStreamer world. And those caps are on each pad. So when you connect two pads, how do you know they're compatible? The way to do that is through caps. Each pad, when you create your element, you declare what kind of capabilities the pad has. For example, in this case, the Warbis Decoder sync pad will declare that on this SRC pad, I can only take Warbis data and no other kind of data I can take. So if you try to connect the SRC02 pad to the sync of Decoder up there, it won't work because they are not compatible. And that's the way to check the compatibility. It's a plugin-based architecture which makes it very easy to write apps, generic apps. So you're in your application. You usually don't have to care which data formats to handle. You can do that if you really want to and you really need to. But usually, like, for example, a media player application like Totem or Known Video, however you call it, they don't usually need to do that. They don't deal with individual different kinds of data. And that makes it very powerful. And it also makes the core of Gstreamer very, very small. So it's really tiny. And the plugins do the actual work. It's written in C for efficiency reasons and for many other reasons. A lot of libraries that are used in GNOME, they are in C. So there is a cultural aspect to that, too. And it's heavily multithreaded. It makes use of multithreading so to make use of your multicore. So in the previous example, for example, there will be a separate CPU used for decoding the verbous audio and the Thera video parts. It's not very relevant to apps. So if you're an app developer, you wouldn't really need to care about multithreading that much. It really abstracts you, Gstreamer, from multithreading. But sometimes you might have to. And also, plugins, usually they don't have to. But in plugins, there's a lot of times when you have to do advanced stuff and then you need to care about multithreading. And then things get really difficult. It's object-oriented programming. It's using something called GeoObject. It's a G-Lib API to be able to do object-oriented programming in C. It's not easy to handle if you are writing plugins and stuff. But it's always very easy to use. It makes the APIs very nice and very easy to use. And why is Rust relevant in this Gstreamer and multimedia? First of all, parsing of media formats, that's what most of the plugins actually do. It's not just saved by design. Like, so many things can go wrong. There is random data coming from the internet. You can't trust it. And there has been like zero-day security advisories in the last some years with Gstreamer. Especially with the FLV decoder, there was a vulnerability where you can get access to the host through that. You provide a file that somehow corrupts the memory. I don't remember the exact details, but it was doing something with the memory. So it was able to exploit that. But if it was written in Rust in safe code, it wouldn't have been possible. And the other reason is multithreading. As I said, it's multithreaded. And plugins a lot of times have to deal with multithreading. And in C, it's extremely difficult to handle threading. In other programming languages, it's a bit easier. But in C, it's even more difficult to do multithreading, right? You will do it wrong, and that would have several consequences. And as I said, in Rust, you usually don't have to be concerned at all when you do multithreading. If you do it wrong, compiler will tell you, and you will satisfy the compiler, and then you won't have any crashes or anything like that, and simultaneous access to data and mutating at the same time and things like that. And mutability and ownership concepts of Rust, they map really well with GStreamer. And for example, in GStreamer, there is a concept of GST mini-objects. So as I said, about G object, GST mini-object is a variant of that that is much a lot lightweight than a proper G object. It has less API and stuff, but it's very lightweight. So if you want something in GStreamer, there is a lot of things that are created and destroyed very fast. And in one second, there will be thousands of these mini-objects created and destroyed, and you want it really, really fast, all this stuff. So that's why they invented something called GST mini-object. But the problem is they didn't want to have mutatable access to the same mini-object by different threads or multiple parts in the code. So they have a restriction there, which is that if it's read only unless it's read only if the reference count is more than one, if multiple, there's multiple references to same object, mini-object, you can't modify it. And there is no really guarantee there, but it tries, you know, and see you can't do much. But in Rust, you can do it much better. I'll show an example if time permits at the end, how exactly that works. And also, you avoid a lot of huge class of memory problems if you write in Rust. That's a generic advantage anyway. It's not just specific to GStreamer, but anything. Also sees an archaic language. It's not just unsafe. It's really old. It doesn't do a lot. Like nowadays, you want more. You can't be handling pointers and everything manually and doing everything manually. You need a higher level language. So Sebastian Drogue, as I mentioned, he wrote these bindings first of all for Rust, GStreamer Rust, RS. And with these, you can use already in your application. There's at least one or two applications out there. They're not very mainstream, but they exist and people are working on them. And they are in Rust, and they use GStreamer in there. So you can do that. But also, you can write plugins. And this GST plugins, RS, the second repository here, that's for that. There's all these plugins written in Rust there. Hopefully, this repository will get bigger and bigger, and you will have more and more plugins written in Rust rather than C, and you can rely on them. Most of the security problems that I mentioned that arise in GStreamer, they are in the plugins, not the GStreamer core. Because the core is so tiny, it's unlikely that it will have problems. Also, it's extremely well tested over the years. So it's pretty safe in that way. But plugins, they are not very safe. So it's better to write them in Rust. I'll have a simple example here where you can see the advantage of why do it in Rust. This very simple code, this will compile just fine in C. Nothing from compiler, no help whatsoever. That there's a lot of problems with this code. It looks very innocent. The caps, as I mentioned, caps as the GStreamer capabilities. It's a representation of one of the capabilities of a pad. Each capability is in the form of something called GST structure. So that's what's happening here. What we are saying is that give me the first capability in the form of the structure from this caps that we have. Then we are asking it to remove that structure. And then we are using, we are setting, we are mutating that structure after we have told it to remove it. So the API of getStructure, this gives you not your own reference on C level. So this thing will crash or cause memory problems in C. But the compiler is not telling you that. So let's see what the Rust compiler says. Can you read this at all? Somewhat? So the error you are getting here is about the set. But you can see the code, right? So the last statement here, Rust compiler is saying to you that the S in here does not have the set method. But I can assure you a structure, GST structure has a set method. And it should have it. But why is the compiler saying it doesn't exist? The reason is that you are not on the Rust level, you are not getting structure, actually. You are getting a thing called option. It's a wrapper type in Rust. In C, you have the equivalence with null pointers. So when you do the same thing in C, you will just get a null pointer, which you will know only at runtime. And if you use the null pointer, things will crash unless you are doing a null check. But Rust doesn't have that concept. You can't have a null pointer. So what you get is an option. And the option is an enum, which is either a none or some. And if you have some, then that means there's something in there, and then you can get a pointer to that. So I'll give you the example code there. So this one in that regard will work. So now you're not getting the structure, but you are unwrapping the, you're using this pattern matching in Rust to, so if you, from this function, call getStructure, if you get something, it gets assigned to the S, and then you can use it and be sure that it won't be null. It's a guarantee. You can also have an else there if you want to handle the case when it's null, but I didn't do that here. So we solved one problem now, but the compiler will give you even more errors now because you really can't see it, right? So when we remove the structure here and here, you're getting a mutable, you're borrowing caps as mutable because you're mutating it, because you're removing something from it. And that's not right because then you can't modify the structure, and you're using the S.set and you're modifying it. So that's not allowed. And that's why you get some errors about it being non-mutable. So we solved the mutability problem by using another function instead of getStructure. You use getMutStructure, and that gives you a mutable reference. And then you can hopefully call these functions, right? And also, the caps, you want to get the mutable reference to it. So you want to call getMut on it, and then call getMutStructure on that one instead. And all this would not be pointed out by a C compiler. There's so many problems with this code. And now you get into another error, which is about multiple mutabilities. So when you did this caps.getMut, you got a mutable reference to the caps. And then you do the C.removeStructure, and that again tries to get a mutable reference. And so you have multiple mutable reference in the same place for the same actual resource. And that's just calling for problems, right? And it's a compiler's right to not allow you to do that, because as I said, you remove the structure. Now the structure doesn't exist, and now you're writing to it. So that just won't work. So finally, we will have a code that actually would work. And that's you get a mutable structure. You get a mutable caps and then mutable structure. And then you set what you need to set, and then you can remove the structure if you want. Although it doesn't make any sense. You just wrote to a structure that you just removed, but this will work, actually. And this is safe code. That's why it works. Yeah, it works fine. And as I said, in C, it won't just build, but in runtime, you will have multiple problems to handle. But now that you built it, it will just work as you want it to work. There's no problem at runtime anymore. Am I out of time? Okay. But just wanted to say, please don't write new project in C and C++. At least consider doing it in Rust, because you don't want to create an unsafe world. You want a safer world, and that's what Rust achieves for you without any runtime costs. That's all from me. I think I'm out of time, but can I take one question? Okay. Anything? Okay. Any questions? Either nobody understood anything or everyone understood everything. I would ask a question. What is it that you do with that combination, like Rust and G-Streamer, in practice? So as I tried to show through the example, when you do it with Rust, you won't have all those runtime problems that some of them you will see immediately, right? Like the memory access problems. No, no, no. I don't mean the advantages that I got. I mean, like, do you know of any projects? Because I would propose one. As I said, there is these two media players that I know that uses it. And I think also now there's another desktop Rouginom app for, it's a metrics client. I forgot the name. And that is written in Rust. And now they're using multimedia for audio, video or something like that. Exactly, because I have seen, I've come across one use case that is a bit more out of the ordinary, which is time lens. Have you heard of the project? No. That is essentially taking any video input and compiling it into a timeline where the individual segments visualize a frame so that you can basically anticipate the mood and the, you know, parts, specific sections of the film just by looking at the timeline. It's quite neat. Timelands.io. Cool. And it's in Rust? Yes, it's Rust and G-streamer. Oh, cool. Now I know. The other questions? Good. Thanks, everyone.
 Welcome to my talk about fearless multimedia.  For the title of the talk, I'm pretty nervous, so not so fearless.  Who am I?  My name is Zishan Ali, and I work for a company called Collabra.  Legally, I'm a freelancer, but I have only one client only, so it's kind of I'm working  for them.  And both my company and me, both my companies, the one man company and Collabra and me, we  all work on FOSS.  I have always been involved in FOSS, and mainly GNOME development has been my thing.  I've worked on other things as well, mostly related to GNOME, but not always.  And my hobbies include cats and flying, not the toys that people fly nowadays, but the  real things.  And what am I going to talk about is G-streamer and Rust, both of them combined.  And I have to cover both of these topics, and they're pretty hard to understand, like,  everything about it, all the essential things.  So I'll try to be very fast.  But if I'm not explaining something well, please just interrupt and tell me, and I can  try to explain better before I move forward.  So it's inspired by another talk by Sebastian Droeg.  He's doing most of the G-streamer and Rust, and he has been a long-time G-streamer hacker,  and he's really competent in both these technologies, so he does most of this work.  You can think of me as a replacement of him in this talk.  And it's inspired by his talks that he has been giving on this topic.  First of all, let's talk about Rust.  Before I move forward, how many of you know both Rust and G-streamer?  Like just basics?  Yeah, no off, like what it is.  Okay, not many.  Let's start with Rust.  How many people know basics of Rust?  Quite a lot, so I won't explain a lot.  I'll just go very fast.  So Rust is a systems programming language, and it's one of the first in its kind that  it focuses on both safety and efficiency at the same time.  There has been a lot of programming languages, as you all know.  Many of them focus on safety, and they are pretty safe, and some are safer than the others,  but most of them haven't been very efficient at system utilization.  And then the programming languages are very efficient.  They are never safe, like C and C++ mainly.  They are extremely unsafe languages.  Rust is the first one to combine them, or maybe there are other languages I don't know of  them, but Rust is one of them.  One of the main things about Rust is that it has a concept of zero cost abstraction.  That is to say that it provides high-level APIs and concepts to easily achieve many  things, many, for example, hash tables and lists and all the data types you're used to.  It will provide those APIs for you, but the concept is that even though they're high-level  and it's easy, they don't have any runtime costs because of that, which is associated  with the same libraries, APIs, and other programming languages.  And with regards to safety, one of the first things about it is that it has no null pointers,  no dangling pointers allowed.  Actually, you can't really call the pointers in Rust as pointers, but I'll talk about  it a bit later as well.  So a lot of problems that arise from pointer handling in other languages like C and C++,  they don't exist in Rust.  But sometimes you need to interact with unsafe code, like, for example, when you're interacting  with some C code or C++ code, you have to do a foreign function interface, and that  is by its nature unsafe because you cannot make the C code that is written safe because  it's not written in Rust.  So for that, Rust allows you to have an unsafe syntax.  Like, it has a keyword and you have all the whichever code you want to write, which is  not safe.  You put it in that, in those, in the brackets, in the context.  So you have all the unsafe code, very isolated.  So when you have a problem in your Rust code, you would know which code to be suspicious  of, first of all, and most likely that will be the code that is causing the problems if  you have, especially, some memory issues.  So it isolates it and that's already a big achievement.  And also it has other concepts like non-mutable state by default.  So by default, all your state is non-mutable.  But if you want mutation, then you mark it as mutable parameter or a variable.  And then when you have the same applies to that, when you have a problem, usually it's  with mutable states, so you should be suspicious of all the variables and arguments that are  marked mutable and then move on from there.  So it makes it a bit easier to debug the code if you have ever problems.  And it has very strict ownership semantics.  It's not a concept in C and C++.  Well, in C, it doesn't exist.  There is no such thing as ownership.  If you have a pointer to something, you can do whatever you want with it.  If you, in C++, you have some APIs in modern C++, especially the smart pointers.  But that's something you can use.  It's not like you have to use it.  You can still do the same thing as in C that you don't really have an owner and you can  just mess things up in a big way.  And in other high-level languages, this problem is solved by the garbage collector.  So you don't have to deal with memory.  There is a garbage collector that deals it for you.  If all the code that is pointing, that needs a particular resource that goes out of scope,  a garbage collector frees it at some point for you.  But it comes at a runtime cost.  So if you don't want to pay that cost, you don't want to use those languages then.  But in Rust, it's solved by the ownership concept.  The concept is that you have only and only one owner at a time of a resource.  That means that you cannot, not just from two threads, but from any two points in the  code, you cannot have ownership at the same time of the same resource.  And this, for example, and this sample, if you assign something to S1, a resource, and  then later on you assign the same thing to S2, you have moved the ownership now to S2.  So if you try to build this code, it won't work.  Rust will complain that you have already moved S1 resource to S2.  And now S1 doesn't exist anymore because you have moved the resource because you can't  have two owners at the same time.  But you can't work like that.  If you have just this restriction, you won't be able to achieve much in Rust.  So there are then exceptions that are safer.  And then you use those exceptions.  And one of the first one is borrowing.  It's used a lot for function arguments in the APIs.  All the APIs, you will always see things being borrowed.  And it's similar to reference, passing by reference in C++.  And what you do is you give the ownership, you borrow the ownership to another variable  or an argument to a function temporarily, and then they can use it and then they return  it.  For example, if you pass it to a function, when it returns, you have the ownership back.  It's not borrowed anymore.  So for example, in this sample, which is a modification of the previous one, instead  of moving the S1, you are giving a reference.  You're borrowing a reference to S1 and S2.  And now you can use them both at the same time.  And since you borrowed it read-only, you can have multiple reference, multiple borrows  of the same resource, because it doesn't matter how many borrows you have at the same time.  It's all read-only.  But the problem with borrows is it's temporary.  So a lot of times you want to keep the reference.  Like, for example, you pass a resource to a function that wants to put it in a structure  and that it wants to keep the structure around, then you can't use borrows.  And for that reason, you have the other data types in Rust that makes it possible.  Like, for example, RCT, it's reference counted.  So it adds reference counting to the resource.  So you actually don't keep the actual resource.  You don't keep a pointer to that.  For example, in here, S1 is the pointer to RC type.  And that is keeping the actual resource in it.  And each time you want to have multiple accesses to the same resource, you just clone the RC.  And when all these RC instances go out of scope, the actual resource will also be freed  automatically for you.  So you don't have to care about that.  It's a Rust.  I didn't mention that.  It's operates on scopes.  So when things go out of scope, compiler frees those things for you.  And it decided at build time, so there's no garbage collection involved at runtime.  And then, but the RC that I mentioned just now, it's only from single thread.  So you can't use it from multiple threads.  Because Rust wants to make sure that you have it so that it's not like, because it's not  atomic.  So you can't atomically increase the reference count or decrease the reference count.  So it would be a bit unsafe to use it from multiple threads.  So for that, we have a concept in Rust called fearless concurrency.  Because if you do concurrent programming in Rust, you are safe.  You don't have to be scared of it.  In other programming languages, people are really scared of threads.  And they should be.  But in Rust, you don't have to be.  Because it has API and concepts that helps you make it safe.  And ARC is one of those types that it provides for multi-threading.  And it's just RC, but it's atomic reference counting.  So you can use it from multiple threads at the same time.  Although it only gives you read-only access.  So you can't modify for multiple threads.  You can't get a mutable state from it.  And for that reason, there's another data I've called mutics that allows you to get  mutable access to the actual resource.  I would normally explain with samples here.  But I have to cover a lot of things.  So I won't get into details.  But if you have some questions afterwards, please let me know.  And I can explain each of these things in detail to you in person.  That gets us to GStreamer.  What is GStreamer?  It's a multimedia framework.  So if you want to write anything related to multimedia, especially audio and video, this  is the framework you would want to use.  It's the multimedia framework of choice for all Linux platforms.  And it has some simple concepts, like elements and pipelines.  So you create a pipeline for each multimedia application.  And in that pipeline, you put elements together, depending on your needs.  There's a concept of source and sync and filter.  And you would connect those elements.  It's a bit like LEGOs.  So you have different elements, and you just put them together, and you achieve a goal.  I have an example here so that it's easier to understand what I'm saying.  For example, if you want to write a player for OGG files, video files, which has audio  and video both multiplex into it, does everyone know what OGG is?  And Warbis is?  Good.  So it's simple.  You put a file source because you want to first read from a file.  You connect it to a demuxer, which demultiplexes the audio and video parts.  And then you connect both those parts to elements that decode both those formats.  And once they are decoded, they need to be played.  And for playing, you have syncs, different kinds of syncs.  So I can't barely see it here.  So we do sync, and here it's kept generic because we have multiple video syncs, real  end and X and all those.  And also, the audio syncs are, there's a lot of them.  So there's no specific example here.  But anyway, the concept is that in each element, you have something called pads.  The blue things you see in here, they're called pads.  And you don't actually connect elements directly, but you connect them through these pads.  And each pad is of type, either source or sync.  And that means that you can only connect source to a sync.  You can't just connect source and source and sync and sync because that doesn't make sense.  And all the real multimedia content, they flow only in one direction.  There can be events and other things that flow upstream, but all the main multimedia  content always flows downstream and through the source and sync.  One example that I would show later, it would require this information about something called  capabilities.  It's called GStreamer caps in the GStreamer world.  And those caps are on each pad.  So when you connect two pads, how do you know they're compatible?  The way to do that is through caps.  Each pad, when you create your element, you declare what kind of capabilities the pad  has.  For example, in this case, the Warbis Decoder sync pad will declare that on this SRC pad,  I can only take Warbis data and no other kind of data I can take.  So if you try to connect the SRC02 pad to the sync of Decoder up there, it won't work  because they are not compatible.  And that's the way to check the compatibility.  It's a plugin-based architecture which makes it very easy to write apps, generic apps.  So you're in your application.  You usually don't have to care which data formats to handle.  You can do that if you really want to and you really need to.  But usually, like, for example, a media player application like Totem or Known Video, however  you call it, they don't usually need to do that.  They don't deal with individual different kinds of data.  And that makes it very powerful.  And it also makes the core of Gstreamer very, very small.  So it's really tiny.  And the plugins do the actual work.  It's written in C for efficiency reasons and for many other reasons.  A lot of libraries that are used in GNOME, they are in C. So there is a cultural aspect  to that, too.  And it's heavily multithreaded.  It makes use of multithreading so to make use of your multicore.  So in the previous example, for example, there will be a separate CPU used for decoding the  verbous audio and the Thera video parts.  It's not very relevant to apps.  So if you're an app developer, you wouldn't really need to care about multithreading that  much.  It really abstracts you, Gstreamer, from multithreading.  But sometimes you might have to.  And also, plugins, usually they don't have to.  But in plugins, there's a lot of times when you have to do advanced stuff and then you  need to care about multithreading.  And then things get really difficult.  It's object-oriented programming.  It's using something called GeoObject.  It's a G-Lib API to be able to do object-oriented programming in C. It's not easy to handle  if you are writing plugins and stuff.  But it's always very easy to use.  It makes the APIs very nice and very easy to use.  And why is Rust relevant in this Gstreamer and multimedia?  First of all, parsing of media formats, that's what most of the plugins actually do.  It's not just saved by design.  Like, so many things can go wrong.  There is random data coming from the internet.  You can't trust it.  And there has been like zero-day security advisories in the last some years with Gstreamer.  Especially with the FLV decoder, there was a vulnerability where you can get access to  the host through that.  You provide a file that somehow corrupts the memory.  I don't remember the exact details, but it was doing something with the memory.  So it was able to exploit that.  But if it was written in Rust in safe code, it wouldn't have been possible.  And the other reason is multithreading.  As I said, it's multithreaded.  And plugins a lot of times have to deal with multithreading.  And in C, it's extremely difficult to handle threading.  In other programming languages, it's a bit easier.  But in C, it's even more difficult to do multithreading, right?  You will do it wrong, and that would have several consequences.  And as I said, in Rust, you usually don't have to be concerned at all when you do multithreading.  If you do it wrong, compiler will tell you, and you will satisfy the compiler, and then  you won't have any crashes or anything like that, and simultaneous access to data and mutating  at the same time and things like that.  And mutability and ownership concepts of Rust, they map really well with GStreamer.  And for example, in GStreamer, there is a concept of GST mini-objects.  So as I said, about G object, GST mini-object is a variant of that that is much a lot lightweight  than a proper G object.  It has less API and stuff, but it's very lightweight.  So if you want something in GStreamer, there is a lot of things that are created and destroyed  very fast.  And in one second, there will be thousands of these mini-objects created and destroyed,  and you want it really, really fast, all this stuff.  So that's why they invented something called GST mini-object.  But the problem is they didn't want to have mutatable access to the same mini-object by  different threads or multiple parts in the code.  So they have a restriction there, which is that if it's read only unless it's read only  if the reference count is more than one, if multiple, there's multiple references to  same object, mini-object, you can't modify it.  And there is no really guarantee there, but it tries, you know, and see you can't do much.  But in Rust, you can do it much better.  I'll show an example if time permits at the end, how exactly that works.  And also, you avoid a lot of huge class of memory problems if you write in Rust.  That's a generic advantage anyway.  It's not just specific to GStreamer, but anything.  Also sees an archaic language.  It's not just unsafe.  It's really old.  It doesn't do a lot.  Like nowadays, you want more.  You can't be handling pointers and everything manually and doing everything manually.  You need a higher level language.  So Sebastian Drogue, as I mentioned, he wrote these bindings first of all for Rust, GStreamer  Rust, RS.  And with these, you can use already in your application.  There's at least one or two applications out there.  They're not very mainstream, but they exist and people are working on them.  And they are in Rust, and they use GStreamer in there.  So you can do that.  But also, you can write plugins.  And this GST plugins, RS, the second repository here, that's for that.  There's all these plugins written in Rust there.  Hopefully, this repository will get bigger and bigger, and you will have more and more  plugins written in Rust rather than C, and you can rely on them.  Most of the security problems that I mentioned that arise in GStreamer, they are in the  plugins, not the GStreamer core.  Because the core is so tiny, it's unlikely that it will have problems.  Also, it's extremely well tested over the years.  So it's pretty safe in that way.  But plugins, they are not very safe.  So it's better to write them in Rust.  I'll have a simple example here where you can see the advantage of why do it in Rust.  This very simple code, this will compile just fine in C. Nothing from compiler, no help whatsoever.  That there's a lot of problems with this code.  It looks very innocent.  The caps, as I mentioned, caps as the GStreamer capabilities.  It's a representation of one of the capabilities of a pad.  Each capability is in the form of something called GST structure.  So that's what's happening here.  What we are saying is that give me the first capability in the form of the structure from  this caps that we have.  Then we are asking it to remove that structure.  And then we are using, we are setting, we are mutating that structure after we have told  it to remove it.  So the API of getStructure, this gives you not your own reference on C level.  So this thing will crash or cause memory problems in C. But the compiler is not telling you  that.  So let's see what the Rust compiler says.  Can you read this at all?  Somewhat?  So the error you are getting here is about the set.  But you can see the code, right?  So the last statement here, Rust compiler is saying to you that the S in here does not  have the set method.  But I can assure you a structure, GST structure has a set method.  And it should have it.  But why is the compiler saying it doesn't exist?  The reason is that you are not on the Rust level, you are not getting structure, actually.  You are getting a thing called option.  It's a wrapper type in Rust.  In C, you have the equivalence with null pointers.  So when you do the same thing in C, you will just get a null pointer, which you will know  only at runtime.  And if you use the null pointer, things will crash unless you are doing a null check.  But Rust doesn't have that concept.  You can't have a null pointer.  So what you get is an option.  And the option is an enum, which is either a none or some.  And if you have some, then that means there's something in there, and then you can get a  pointer to that.  So I'll give you the example code there.  So this one in that regard will work.  So now you're not getting the structure, but you are unwrapping the, you're using this  pattern matching in Rust to, so if you, from this function, call getStructure, if you get  something, it gets assigned to the S, and then you can use it and be sure that it won't  be null.  It's a guarantee.  You can also have an else there if you want to handle the case when it's null, but I didn't  do that here.  So we solved one problem now, but the compiler will give you even more errors now because  you really can't see it, right?  So when we remove the structure here and here, you're getting a mutable, you're borrowing  caps as mutable because you're mutating it, because you're removing something from it.  And that's not right because then you can't modify the structure, and you're using the  S.set and you're modifying it.  So that's not allowed.  And that's why you get some errors about it being non-mutable.  So we solved the mutability problem by using another function instead of getStructure.  You use getMutStructure, and that gives you a mutable reference.  And then you can hopefully call these functions, right?  And also, the caps, you want to get the mutable reference to it.  So you want to call getMut on it, and then call getMutStructure on that one instead.  And all this would not be pointed out by a C compiler.  There's so many problems with this code.  And now you get into another error, which is about multiple mutabilities.  So when you did this caps.getMut, you got a mutable reference to the caps.  And then you do the C.removeStructure, and that again tries to get a mutable reference.  And so you have multiple mutable reference in the same place for the same actual resource.  And that's just calling for problems, right?  And it's a compiler's right to not allow you to do that, because as I said, you remove  the structure.  Now the structure doesn't exist, and now you're writing to it.  So that just won't work.  So finally, we will have a code that actually would work.  And that's you get a mutable structure.  You get a mutable caps and then mutable structure.  And then you set what you need to set, and then you can remove the structure if you want.  Although it doesn't make any sense.  You just wrote to a structure that you just removed, but this will work, actually.  And this is safe code.  That's why it works.  Yeah, it works fine.  And as I said, in C, it won't just build, but in runtime, you will have multiple problems  to handle.  But now that you built it, it will just work as you want it to work.  There's no problem at runtime anymore.  Am I out of time?  Okay.  But just wanted to say, please don't write new project in C and C++.  At least consider doing it in Rust, because you don't want to create an unsafe world.  You want a safer world, and that's what Rust achieves for you without any runtime costs.  That's all from me.  I think I'm out of time, but can I take one question?  Okay.  Anything?  Okay.  Any questions?  Either nobody understood anything or everyone understood everything.  I would ask a question.  What is it that you do with that combination, like Rust and G-Streamer, in practice?  So as I tried to show through the example, when you do it with Rust, you won't have all  those runtime problems that some of them you will see immediately, right?  Like the memory access problems.  No, no, no.  I don't mean the advantages that I got.  I mean, like, do you know of any projects?  Because I would propose one.  As I said, there is these two media players that I know that uses it.  And I think also now there's another desktop Rouginom app for, it's a metrics client.  I forgot the name.  And that is written in Rust.  And now they're using multimedia for audio, video or something like that.  Exactly, because I have seen, I've come across one use case that is a bit more out of the  ordinary, which is time lens.  Have you heard of the project?  No.  That is essentially taking any video input and compiling it into a timeline where the  individual segments visualize a frame so that you can basically anticipate the mood and  the, you know, parts, specific sections of the film just by looking at the timeline.  It's quite neat.  Timelands.io.  Cool.  And it's in Rust?  Yes, it's Rust and G-streamer.  Oh, cool.  Now I know.  The other questions?  Good.  Thanks, everyone.
Fearless_Multimedia_Programming_CAQIxBquF3w_0_155.png Fearless_Multimedia_Programming_CAQIxBquF3w_156_263.png Fearless_Multimedia_Programming_CAQIxBquF3w_264_395.png Fearless_Multimedia_Programming_CAQIxBquF3w_396_575.png Fearless_Multimedia_Programming_CAQIxBquF3w_576_659.png Fearless_Multimedia_Programming_CAQIxBquF3w_660_851.png Fearless_Multimedia_Programming_CAQIxBquF3w_852_1151.png Fearless_Multimedia_Programming_CAQIxBquF3w_1152_1199.png Fearless_Multimedia_Programming_CAQIxBquF3w_1200_1271.png Fearless_Multimedia_Programming_CAQIxBquF3w_1272_1343.png Fearless_Multimedia_Programming_CAQIxBquF3w_1344_1691.png Fearless_Multimedia_Programming_CAQIxBquF3w_1692_3539.png Fearless_Multimedia_Programming_CAQIxBquF3w_3540_6287.png Fearless_Multimedia_Programming_CAQIxBquF3w_6288_6395.png Fearless_Multimedia_Programming_CAQIxBquF3w_6396_7931.png Fearless_Multimedia_Programming_CAQIxBquF3w_7932_10439.png Fearless_Multimedia_Programming_CAQIxBquF3w_10440_13607.png Fearless_Multimedia_Programming_CAQIxBquF3w_13608_19403.png Fearless_Multimedia_Programming_CAQIxBquF3w_19404_23411.png Fearless_Multimedia_Programming_CAQIxBquF3w_23412_24791.png Fearless_Multimedia_Programming_CAQIxBquF3w_24792_25475.png Fearless_Multimedia_Programming_CAQIxBquF3w_25476_26303.png Fearless_Multimedia_Programming_CAQIxBquF3w_26304_26519.png Fearless_Multimedia_Programming_CAQIxBquF3w_26520_26627.png Fearless_Multimedia_Programming_CAQIxBquF3w_26628_28199.png Fearless_Multimedia_Programming_CAQIxBquF3w_28200_29651.png Fearless_Multimedia_Programming_CAQIxBquF3w_29652_29927.png Fearless_Multimedia_Programming_CAQIxBquF3w_29928_32135.png Fearless_Multimedia_Programming_CAQIxBquF3w_32136_32879.png Fearless_Multimedia_Programming_CAQIxBquF3w_32880_36479.png Fearless_Multimedia_Programming_CAQIxBquF3w_36480_36551.png Fearless_Multimedia_Programming_CAQIxBquF3w_36552_36743.png Fearless_Multimedia_Programming_CAQIxBquF3w_36744_36803.png Fearless_Multimedia_Programming_CAQIxBquF3w_36804_39467.png Fearless_Multimedia_Programming_CAQIxBquF3w_39468_39851.png Fearless_Multimedia_Programming_CAQIxBquF3w_39852_39983.png Fearless_Multimedia_Programming_CAQIxBquF3w_39984_40715.png Fearless_Multimedia_Programming_CAQIxBquF3w_40716_41315.png Fearless_Multimedia_Programming_CAQIxBquF3w_41316_41819.png Fearless_Multimedia_Programming_CAQIxBquF3w_41820_42023.png Fearless_Multimedia_Programming_CAQIxBquF3w_42024_43127.png Fearless_Multimedia_Programming_CAQIxBquF3w_43128_43199.png Fearless_Multimedia_Programming_CAQIxBquF3w_43200_44519.png Fearless_Multimedia_Programming_CAQIxBquF3w_44520_44723.png Fearless_Multimedia_Programming_CAQIxBquF3w_44724_44819.png Fearless_Multimedia_Programming_CAQIxBquF3w_44820_45695.png Fearless_Multimedia_Programming_CAQIxBquF3w_45696_46943.png Fearless_Multimedia_Programming_CAQIxBquF3w_46944_47088.png
Welcome to my talk about fearless multimedia. For the title of the talk, I'm pretty nervous, so not so fearless. Who am I? My name is Zishan Ali, and I work for a company called Collabra. Legally, I'm a freelancer, but I have only one client only, so it's kind of I'm working for them. And both my company and me, both my companies, the one man company and Collabra and me, we all work on FOSS. I have always been involved in FOSS, and mainly GNOME development has been my thing. I've worked on other things as well, mostly related to GNOME, but not always. And my hobbies include cats and flying, not the toys that people fly nowadays, but the real things. And what am I going to talk about is G-streamer and Rust, both of them combined. And I have to cover both of these topics, and they're pretty hard to understand, like, everything about it, all the essential things. So I'll try to be very fast. But if I'm not explaining something well, please just interrupt and tell me, and I can try to explain better before I move forward. So it's inspired by another talk by Sebastian Droeg. He's doing most of the G-streamer and Rust, and he has been a long-time G-streamer hacker, and he's really competent in both these technologies, so he does most of this work. You can think of me as a replacement of him in this talk. And it's inspired by his talks that he has been giving on this topic. First of all, let's talk about Rust. Before I move forward, how many of you know both Rust and G-streamer? Like just basics? Yeah, no off, like what it is. Okay, not many. Let's start with Rust. How many people know basics of Rust? Quite a lot, so I won't explain a lot. I'll just go very fast. So Rust is a systems programming language, and it's one of the first in its kind that it focuses on both safety and efficiency at the same time. There has been a lot of programming languages, as you all know. Many of them focus on safety, and they are pretty safe, and some are safer than the others, but most of them haven't been very efficient at system utilization. And then the programming languages are very efficient. They are never safe, like C and C++ mainly. They are extremely unsafe languages. Rust is the first one to combine them, or maybe there are other languages I don't know of them, but Rust is one of them. One of the main things about Rust is that it has a concept of zero cost abstraction. That is to say that it provides high-level APIs and concepts to easily achieve many things, many, for example, hash tables and lists and all the data types you're used to. It will provide those APIs for you, but the concept is that even though they're high-level and it's easy, they don't have any runtime costs because of that, which is associated with the same libraries, APIs, and other programming languages. And with regards to safety, one of the first things about it is that it has no null pointers, no dangling pointers allowed. Actually, you can't really call the pointers in Rust as pointers, but I'll talk about it a bit later as well. So a lot of problems that arise from pointer handling in other languages like C and C++, they don't exist in Rust. But sometimes you need to interact with unsafe code, like, for example, when you're interacting with some C code or C++ code, you have to do a foreign function interface, and that is by its nature unsafe because you cannot make the C code that is written safe because it's not written in Rust. So for that, Rust allows you to have an unsafe syntax. Like, it has a keyword and you have all the whichever code you want to write, which is not safe. You put it in that, in those, in the brackets, in the context. So you have all the unsafe code, very isolated. So when you have a problem in your Rust code, you would know which code to be suspicious of, first of all, and most likely that will be the code that is causing the problems if you have, especially, some memory issues. So it isolates it and that's already a big achievement. And also it has other concepts like non-mutable state by default. So by default, all your state is non-mutable. But if you want mutation, then you mark it as mutable parameter or a variable. And then when you have the same applies to that, when you have a problem, usually it's with mutable states, so you should be suspicious of all the variables and arguments that are marked mutable and then move on from there. So it makes it a bit easier to debug the code if you have ever problems. And it has very strict ownership semantics. It's not a concept in C and C++. Well, in C, it doesn't exist. There is no such thing as ownership. If you have a pointer to something, you can do whatever you want with it. If you, in C++, you have some APIs in modern C++, especially the smart pointers. But that's something you can use. It's not like you have to use it. You can still do the same thing as in C that you don't really have an owner and you can just mess things up in a big way. And in other high-level languages, this problem is solved by the garbage collector. So you don't have to deal with memory. There is a garbage collector that deals it for you. If all the code that is pointing, that needs a particular resource that goes out of scope, a garbage collector frees it at some point for you. But it comes at a runtime cost. So if you don't want to pay that cost, you don't want to use those languages then. But in Rust, it's solved by the ownership concept. The concept is that you have only and only one owner at a time of a resource. That means that you cannot, not just from two threads, but from any two points in the code, you cannot have ownership at the same time of the same resource. And this, for example, and this sample, if you assign something to S1, a resource, and then later on you assign the same thing to S2, you have moved the ownership now to S2. So if you try to build this code, it won't work. Rust will complain that you have already moved S1 resource to S2. And now S1 doesn't exist anymore because you have moved the resource because you can't have two owners at the same time. But you can't work like that. If you have just this restriction, you won't be able to achieve much in Rust. So there are then exceptions that are safer. And then you use those exceptions. And one of the first one is borrowing. It's used a lot for function arguments in the APIs. All the APIs, you will always see things being borrowed. And it's similar to reference, passing by reference in C++. And what you do is you give the ownership, you borrow the ownership to another variable or an argument to a function temporarily, and then they can use it and then they return it. For example, if you pass it to a function, when it returns, you have the ownership back. It's not borrowed anymore. So for example, in this sample, which is a modification of the previous one, instead of moving the S1, you are giving a reference. You're borrowing a reference to S1 and S2. And now you can use them both at the same time. And since you borrowed it read-only, you can have multiple reference, multiple borrows of the same resource, because it doesn't matter how many borrows you have at the same time. It's all read-only. But the problem with borrows is it's temporary. So a lot of times you want to keep the reference. Like, for example, you pass a resource to a function that wants to put it in a structure and that it wants to keep the structure around, then you can't use borrows. And for that reason, you have the other data types in Rust that makes it possible. Like, for example, RCT, it's reference counted. So it adds reference counting to the resource. So you actually don't keep the actual resource. You don't keep a pointer to that. For example, in here, S1 is the pointer to RC type. And that is keeping the actual resource in it. And each time you want to have multiple accesses to the same resource, you just clone the RC. And when all these RC instances go out of scope, the actual resource will also be freed automatically for you. So you don't have to care about that. It's a Rust. I didn't mention that. It's operates on scopes. So when things go out of scope, compiler frees those things for you. And it decided at build time, so there's no garbage collection involved at runtime. And then, but the RC that I mentioned just now, it's only from single thread. So you can't use it from multiple threads. Because Rust wants to make sure that you have it so that it's not like, because it's not atomic. So you can't atomically increase the reference count or decrease the reference count. So it would be a bit unsafe to use it from multiple threads. So for that, we have a concept in Rust called fearless concurrency. Because if you do concurrent programming in Rust, you are safe. You don't have to be scared of it. In other programming languages, people are really scared of threads. And they should be. But in Rust, you don't have to be. Because it has API and concepts that helps you make it safe. And ARC is one of those types that it provides for multi-threading. And it's just RC, but it's atomic reference counting. So you can use it from multiple threads at the same time. Although it only gives you read-only access. So you can't modify for multiple threads. You can't get a mutable state from it. And for that reason, there's another data I've called mutics that allows you to get mutable access to the actual resource. I would normally explain with samples here. But I have to cover a lot of things. So I won't get into details. But if you have some questions afterwards, please let me know. And I can explain each of these things in detail to you in person. That gets us to GStreamer. What is GStreamer? It's a multimedia framework. So if you want to write anything related to multimedia, especially audio and video, this is the framework you would want to use. It's the multimedia framework of choice for all Linux platforms. And it has some simple concepts, like elements and pipelines. So you create a pipeline for each multimedia application. And in that pipeline, you put elements together, depending on your needs. There's a concept of source and sync and filter. And you would connect those elements. It's a bit like LEGOs. So you have different elements, and you just put them together, and you achieve a goal. I have an example here so that it's easier to understand what I'm saying. For example, if you want to write a player for OGG files, video files, which has audio and video both multiplex into it, does everyone know what OGG is? And Warbis is? Good. So it's simple. You put a file source because you want to first read from a file. You connect it to a demuxer, which demultiplexes the audio and video parts. And then you connect both those parts to elements that decode both those formats. And once they are decoded, they need to be played. And for playing, you have syncs, different kinds of syncs. So I can't barely see it here. So we do sync, and here it's kept generic because we have multiple video syncs, real end and X and all those. And also, the audio syncs are, there's a lot of them. So there's no specific example here. But anyway, the concept is that in each element, you have something called pads. The blue things you see in here, they're called pads. And you don't actually connect elements directly, but you connect them through these pads. And each pad is of type, either source or sync. And that means that you can only connect source to a sync. You can't just connect source and source and sync and sync because that doesn't make sense. And all the real multimedia content, they flow only in one direction. There can be events and other things that flow upstream, but all the main multimedia content always flows downstream and through the source and sync. One example that I would show later, it would require this information about something called capabilities. It's called GStreamer caps in the GStreamer world. And those caps are on each pad. So when you connect two pads, how do you know they're compatible? The way to do that is through caps. Each pad, when you create your element, you declare what kind of capabilities the pad has. For example, in this case, the Warbis Decoder sync pad will declare that on this SRC pad, I can only take Warbis data and no other kind of data I can take. So if you try to connect the SRC02 pad to the sync of Decoder up there, it won't work because they are not compatible. And that's the way to check the compatibility. It's a plugin-based architecture which makes it very easy to write apps, generic apps. So you're in your application. You usually don't have to care which data formats to handle. You can do that if you really want to and you really need to. But usually, like, for example, a media player application like Totem or Known Video, however you call it, they don't usually need to do that. They don't deal with individual different kinds of data. And that makes it very powerful. And it also makes the core of Gstreamer very, very small. So it's really tiny. And the plugins do the actual work. It's written in C for efficiency reasons and for many other reasons. A lot of libraries that are used in GNOME, they are in C. So there is a cultural aspect to that, too. And it's heavily multithreaded. It makes use of multithreading so to make use of your multicore. So in the previous example, for example, there will be a separate CPU used for decoding the verbous audio and the Thera video parts. It's not very relevant to apps. So if you're an app developer, you wouldn't really need to care about multithreading that much. It really abstracts you, Gstreamer, from multithreading. But sometimes you might have to. And also, plugins, usually they don't have to. But in plugins, there's a lot of times when you have to do advanced stuff and then you need to care about multithreading. And then things get really difficult. It's object-oriented programming. It's using something called GeoObject. It's a G-Lib API to be able to do object-oriented programming in C. It's not easy to handle if you are writing plugins and stuff. But it's always very easy to use. It makes the APIs very nice and very easy to use. And why is Rust relevant in this Gstreamer and multimedia? First of all, parsing of media formats, that's what most of the plugins actually do. It's not just saved by design. Like, so many things can go wrong. There is random data coming from the internet. You can't trust it. And there has been like zero-day security advisories in the last some years with Gstreamer. Especially with the FLV decoder, there was a vulnerability where you can get access to the host through that. You provide a file that somehow corrupts the memory. I don't remember the exact details, but it was doing something with the memory. So it was able to exploit that. But if it was written in Rust in safe code, it wouldn't have been possible. And the other reason is multithreading. As I said, it's multithreaded. And plugins a lot of times have to deal with multithreading. And in C, it's extremely difficult to handle threading. In other programming languages, it's a bit easier. But in C, it's even more difficult to do multithreading, right? You will do it wrong, and that would have several consequences. And as I said, in Rust, you usually don't have to be concerned at all when you do multithreading. If you do it wrong, compiler will tell you, and you will satisfy the compiler, and then you won't have any crashes or anything like that, and simultaneous access to data and mutating at the same time and things like that. And mutability and ownership concepts of Rust, they map really well with GStreamer. And for example, in GStreamer, there is a concept of GST mini-objects. So as I said, about G object, GST mini-object is a variant of that that is much a lot lightweight than a proper G object. It has less API and stuff, but it's very lightweight. So if you want something in GStreamer, there is a lot of things that are created and destroyed very fast. And in one second, there will be thousands of these mini-objects created and destroyed, and you want it really, really fast, all this stuff. So that's why they invented something called GST mini-object. But the problem is they didn't want to have mutatable access to the same mini-object by different threads or multiple parts in the code. So they have a restriction there, which is that if it's read only unless it's read only if the reference count is more than one, if multiple, there's multiple references to same object, mini-object, you can't modify it. And there is no really guarantee there, but it tries, you know, and see you can't do much. But in Rust, you can do it much better. I'll show an example if time permits at the end, how exactly that works. And also, you avoid a lot of huge class of memory problems if you write in Rust. That's a generic advantage anyway. It's not just specific to GStreamer, but anything. Also sees an archaic language. It's not just unsafe. It's really old. It doesn't do a lot. Like nowadays, you want more. You can't be handling pointers and everything manually and doing everything manually. You need a higher level language. So Sebastian Drogue, as I mentioned, he wrote these bindings first of all for Rust, GStreamer Rust, RS. And with these, you can use already in your application. There's at least one or two applications out there. They're not very mainstream, but they exist and people are working on them. And they are in Rust, and they use GStreamer in there. So you can do that. But also, you can write plugins. And this GST plugins, RS, the second repository here, that's for that. There's all these plugins written in Rust there. Hopefully, this repository will get bigger and bigger, and you will have more and more plugins written in Rust rather than C, and you can rely on them. Most of the security problems that I mentioned that arise in GStreamer, they are in the plugins, not the GStreamer core. Because the core is so tiny, it's unlikely that it will have problems. Also, it's extremely well tested over the years. So it's pretty safe in that way. But plugins, they are not very safe. So it's better to write them in Rust. I'll have a simple example here where you can see the advantage of why do it in Rust. This very simple code, this will compile just fine in C. Nothing from compiler, no help whatsoever. That there's a lot of problems with this code. It looks very innocent. The caps, as I mentioned, caps as the GStreamer capabilities. It's a representation of one of the capabilities of a pad. Each capability is in the form of something called GST structure. So that's what's happening here. What we are saying is that give me the first capability in the form of the structure from this caps that we have. Then we are asking it to remove that structure. And then we are using, we are setting, we are mutating that structure after we have told it to remove it. So the API of getStructure, this gives you not your own reference on C level. So this thing will crash or cause memory problems in C. But the compiler is not telling you that. So let's see what the Rust compiler says. Can you read this at all? Somewhat? So the error you are getting here is about the set. But you can see the code, right? So the last statement here, Rust compiler is saying to you that the S in here does not have the set method. But I can assure you a structure, GST structure has a set method. And it should have it. But why is the compiler saying it doesn't exist? The reason is that you are not on the Rust level, you are not getting structure, actually. You are getting a thing called option. It's a wrapper type in Rust. In C, you have the equivalence with null pointers. So when you do the same thing in C, you will just get a null pointer, which you will know only at runtime. And if you use the null pointer, things will crash unless you are doing a null check. But Rust doesn't have that concept. You can't have a null pointer. So what you get is an option. And the option is an enum, which is either a none or some. And if you have some, then that means there's something in there, and then you can get a pointer to that. So I'll give you the example code there. So this one in that regard will work. So now you're not getting the structure, but you are unwrapping the, you're using this pattern matching in Rust to, so if you, from this function, call getStructure, if you get something, it gets assigned to the S, and then you can use it and be sure that it won't be null. It's a guarantee. You can also have an else there if you want to handle the case when it's null, but I didn't do that here. So we solved one problem now, but the compiler will give you even more errors now because you really can't see it, right? So when we remove the structure here and here, you're getting a mutable, you're borrowing caps as mutable because you're mutating it, because you're removing something from it. And that's not right because then you can't modify the structure, and you're using the S.set and you're modifying it. So that's not allowed. And that's why you get some errors about it being non-mutable. So we solved the mutability problem by using another function instead of getStructure. You use getMutStructure, and that gives you a mutable reference. And then you can hopefully call these functions, right? And also, the caps, you want to get the mutable reference to it. So you want to call getMut on it, and then call getMutStructure on that one instead. And all this would not be pointed out by a C compiler. There's so many problems with this code. And now you get into another error, which is about multiple mutabilities. So when you did this caps.getMut, you got a mutable reference to the caps. And then you do the C.removeStructure, and that again tries to get a mutable reference. And so you have multiple mutable reference in the same place for the same actual resource. And that's just calling for problems, right? And it's a compiler's right to not allow you to do that, because as I said, you remove the structure. Now the structure doesn't exist, and now you're writing to it. So that just won't work. So finally, we will have a code that actually would work. And that's you get a mutable structure. You get a mutable caps and then mutable structure. And then you set what you need to set, and then you can remove the structure if you want. Although it doesn't make any sense. You just wrote to a structure that you just removed, but this will work, actually. And this is safe code. That's why it works. Yeah, it works fine. And as I said, in C, it won't just build, but in runtime, you will have multiple problems to handle. But now that you built it, it will just work as you want it to work. There's no problem at runtime anymore. Am I out of time? Okay. But just wanted to say, please don't write new project in C and C++. At least consider doing it in Rust, because you don't want to create an unsafe world. You want a safer world, and that's what Rust achieves for you without any runtime costs. That's all from me. I think I'm out of time, but can I take one question? Okay. Anything? Okay. Any questions? Either nobody understood anything or everyone understood everything. I would ask a question. What is it that you do with that combination, like Rust and G-Streamer, in practice? So as I tried to show through the example, when you do it with Rust, you won't have all those runtime problems that some of them you will see immediately, right? Like the memory access problems. No, no, no. I don't mean the advantages that I got. I mean, like, do you know of any projects? Because I would propose one. As I said, there is these two media players that I know that uses it. And I think also now there's another desktop Rouginom app for, it's a metrics client. I forgot the name. And that is written in Rust. And now they're using multimedia for audio, video or something like that. Exactly, because I have seen, I've come across one use case that is a bit more out of the ordinary, which is time lens. Have you heard of the project? No. That is essentially taking any video input and compiling it into a timeline where the individual segments visualize a frame so that you can basically anticipate the mood and the, you know, parts, specific sections of the film just by looking at the timeline. It's quite neat. Timelands.io. Cool. And it's in Rust? Yes, it's Rust and G-streamer. Oh, cool. Now I know. The other questions? Good. Thanks, everyone.
a replacement of him in this talk. And it's that it focuses on both safety and efficiency at the same time. There has been a lot of and some are safer than the others, but most of some C code or C++ code, you have to do a foreign likely that will be the code that is causing the with memory. There is a garbage collector that at some point for you. But it comes at a runtime at the same time of the same resource. And this, to achieve much in Rust. So there are then you. So you don't have to care about that. It's a concurrent programming in Rust, you are safe. You don't have to be. Because it has API and concepts write anything related to multimedia, especially want to use. It's the multimedia framework of choice for all Linux platforms. And it has some GStreamer caps in the GStreamer world. And those extremely difficult to handle threading. In other programming languages, it's a bit easier. But in C, it's even more difficult to do multithreading, generic advantage anyway. It's not just specific bindings first of all for Rust, GStreamer Rust, applications out there. They're not very GStreamer in there. So you can do that. But also, the plugins, not the GStreamer core. Because the a compiler's right to not allow you to do that,