Fearless Multimedia Programming
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 50 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43097 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
All Systems Go! 201841 / 50
1
4
11
12
13
14
16
17
19
23
24
25
29
30
32
34
35
39
40
41
43
44
50
00:00
SpacetimeSystem programmingComputer programmingAbstractionPointer (computer programming)Default (computer science)Semantics (computer science)Inclusion mapMultimediaCodeSpeicherbereinigungState of matterPointer (computer programming)Poisson-KlammerContext awarenessProgramming languageAbstractionPhysical systemMultiplication signParameter (computer programming)Software developerSystem softwareVariable (mathematics)Run time (program lifecycle phase)Table (information)Hash functionPoint (geometry)Boolean algebraBitExistenceFunctional programmingSemantics (computer science)Computer programmingDefault (computer science)Exception handlingData type1 (number)Computer-assisted translationPlastikkarteLevel (video gaming)Hacker (term)Semiconductor memorySoftware testingLibrary (computing)Interface (computing)Utility softwareFormal languageSampling (statistics)Client (computing)High-level programming languageMeeting/InterviewComputer animation
08:01
System programmingCloningConcurrency (computer science)MultimediaElement (mathematics)Instance (computer science)Source codeFunctional programmingType theoryKey (cryptography)Cartesian coordinate systemMobile appDataflowDirected graphParameter (computer programming)Pointer (computer programming)Computing platformElement (mathematics)Thread (computing)Atomic numberCompilerComputer fileFreezingWritingBitDirection (geometry)Data typeSpeicherbereinigungState of matterDifferent (Kate Ryan album)MehrfachzugriffConcurrency (computer science)SynchronizationFile formatComputer architectureRow (database)Programming languageVideoconferencingSampling (statistics)Message passingGeneric programmingGraphics tabletSoftware frameworkMultiplication signCountingMultimediaDemo (music)CASE <Informatik>Connected spaceCodierung <Programmierung>Axiom of choiceMereologyComputer iconData structureBuildingInformationComputer programmingEvent horizonMultiplicationRun time (program lifecycle phase)Spherical capWechselseitiger AusschlussLatent heatGoodness of fitComputer animation
15:47
WritingMobile appPlug-in (computing)Object-oriented programmingSource codeMultiplicationError messageRead-only memoryFormal languageKeyboard shortcutTape driveFile formatPlug-in (computing)CountingMereologyMultimediaRead-only memoryMultiplication signOnline helpObject-oriented programmingCompilerData structureThread (computing)MultiplicationCodeInformation securityAbstractionRight angleProgramming languageSemiconductor memoryFormal languageCore dumpInternetworkingSpherical capKeyboard shortcutDifferent (Kate Ryan album)ParsingCrash (computing)VideoconferencingCartesian coordinate systemHypermediaComputer programmingBefehlsprozessorComputer fileSocial classLibrary (computing)Form (programming)Pointer (computer programming)Vulnerability (computing)Graphics tabletRepresentation (politics)Proper mapRepository (publishing)Mobile appSoftware developerBitEstimationTable (information)RandomizationGeneric programmingComputer animation
23:32
System programmingComputer configurationSet (mathematics)Data structureSheaf (mathematics)CASE <Informatik>MereologyRun time (program lifecycle phase)Latent heatFunctional programmingSet (mathematics)Pattern languageProjective planeoutputError messageMultiplication signPointer (computer programming)Combinational logicData structureTelephone number mappingSpherical capCompilerCodeGoodness of fitMatrix (mathematics)Semiconductor memoryLevel (video gaming)Mobile appHypermediaStatement (computer science)Right angleMultimediaEquivalence relationType theoryComputer configurationWrapper (data mining)VideoconferencingClient (computing)Crash (computing)Compilation albumCausalityBitWritingExistenceMetric system
31:18
System programming
Transcript: English(auto-generated)
00:07
Welcome to my talk about Fearless Multimedia. For the title of the talk, I'm pretty nervous, so not so fearless. Who am I? My name is Zishan Ali,
00:20
and I work for a company called Collabra. Legally, I'm a freelancer, but I have only one client, so it's kind of I'm working for them. And both my company and me, both my companies, the one-man company and Collabra, and me, we all work on FOSS.
00:42
I have always been involved in FOSS, and mainly GNOME development has been my thing. I've worked on other things as well, mostly related to GNOME, but not always. And my hobbies include cats and flying, not the toys that people fly nowadays, but the real things.
01:05
And what I'm going to talk about is Gstreamer and Rust, both of them combined. And I have to cover both of these topics, and they're pretty hard to understand, like everything about it, all the essential things, so I'll try to be very fast.
01:20
But if I'm not explaining something well, please just interrupt and tell me, and I can try to explain better before I move forward. So it's inspired by another talk by Sebastian Drogue. He's doing most of the Gstreamer and Rust, and he has been a long-time Gstreamer hacker, and he's really competent in both these technologies,
01:43
so he does most of this work. You can think of me as a replacement of him in this talk. And it's inspired by his talks that he has been giving on this topic. First of all, let's talk about Rust. Before I move forward, how many of you know both Rust and Gstreamer?
02:03
Like, just basics. Yeah, no off, like what it is. OK, not many. Let's start with Rust. How many people know basics of Rust? Quite a lot, so I won't explain a lot.
02:20
I'll just go very fast. So Rust is a systems programming language, and it's one of the first in its kind that it focuses on both safety and efficiency at the same time. There has been a lot of programming languages, as you all know. Many of them focus on safety, and they are pretty safe, and some are safer than the others,
02:42
but most of them haven't been very efficient at system utilization. And then the programming languages are very efficient. They are never safe, like C and C++, mainly. They are extremely unsafe languages. Rust is the first one to combine them all.
03:02
Maybe there are other languages, I don't know of them, but Rust is one of them. One of the main things about Rust is that it has a concept of zero-cost abstraction. That is to say that it provides high-level APIs and concepts to easily achieve many things,
03:21
many, for example, hash tables and lists and all the data types you're used to. It will provide those APIs for you. But the concept is that even though they are high-level and it's easy, they don't have any runtime costs because of that, which is associated with the same libraries, APIs and other programming languages.
03:43
And with regards to safety, one of the first things about it is that it has no null pointers, no dangling pointers allowed. Actually, you can't really call the pointers in Rust as pointers, but I'll talk about it a bit later as well.
04:02
So a lot of problems that arise from pointer handling in other languages like C and C++, most of them don't exist in Rust. But sometimes you need to interact with unsafe code, like, for example, when you're interacting with some C code or C++ code.
04:21
You have to do foreign function interface, and that is by its nature unsafe because you cannot make the C code that is written safe because it's not written in Rust. So for that, Rust allows you to have an unsafe syntax. Like, it has a keyword and you have whichever code you want to write,
04:45
which is not safe. You put it in the brackets in the context. So you have all the unsafe code very isolated. So when you have a problem in your Rust code,
05:01
you would know which code to be suspicious of first of all, and most likely that will be the code that is causing the problems if you have especially some memory issues. So it isolates it, and that's already a big achievement. And also it has other concepts like non-mutable state by default.
05:22
So by default, all your state is non-mutable. But if you want mutation, then you mark it as mutable parameter or a variable. And then when you have... The same applies to that. When you have a problem, usually it's with mutable state, so you should be first suspicious of all the variables
05:41
and arguments that are marked mutable, and then move on from there. So it makes it a bit easier to debug the code if you have problems. And it has very strict ownership semantics. It's not a concept in C and C++. Well, in C it doesn't exist. There is no such thing as ownership.
06:01
If you have a pointer to something, you can do whatever you want with it. In C++, you have some APIs. In modern C++ especially, there are smart pointers. But that's something you can use. It's not like you have to use it. You can still do the same thing as in C that you don't really have an owner
06:20
and you can just mess things up in a big way. And in other high-level languages, this problem is solved for you by the garbage collector. So you don't have to deal with memory. There's a garbage collector that deals it for you. Like, if all the code that needs a particular resource,
06:44
it goes out of scope. A garbage collector frees it at some point for you. But it comes at a runtime cost. So if you don't want to pay that cost, you don't want to use those languages then. But in Rust, it's solved by the ownership concept.
07:04
And the concept is that you have only and only one owner at a time of a resource. That means that you cannot, not just from two threads, but from any two points in the code, you cannot have ownership at the same time
07:20
of the same resource. And, sorry, and this, for example, in this sample, if you assign something to S1, a resource, and then later on you assign the same thing to S2, you have moved the ownership now to S2. So if you try to build this code, it won't work. Rust will complain that you have already moved S1 resource to S2,
07:42
and now S1 doesn't exist anymore because you have moved the resource, because you can't have two owners at the same time. But you can't work like that. If you have just this restriction, you won't be able to achieve much in Rust. So there are then exceptions that are safer, and then you use those exceptions.
08:01
And one of the first ones is borrowing. It's used a lot for function arguments in the APIs. All the APIs, you will always see things being borrowed. And it's similar to passing by reference in C++. And what you do is you give the ownership, you borrow the ownership to another variable
08:23
or an argument to a function temporarily, and then they can use it, and then they return it. For example, if you pass it to a function, when it returns, you have the ownership back. It's not borrowed anymore. So for example, in this sample, which is a modification of the previous one,
08:43
instead of moving the S1, you are giving a reference. You're borrowing a reference to S1 and S2, and now you can use them both at the same time. And since you borrowed it read-only, you can have multiple borrows of the same resource, because it doesn't matter how many borrows you have at the same time.
09:01
It's all read-only. But the problem with borrows is it's temporary. So a lot of times, you want to keep the reference. Like, for example, you pass a resource to a function that wants to put it in a structure, and that it wants to keep the structure around.
09:21
Then you can't use borrows. And for that reason, you have the other data types in Rust that makes it possible. Like, for example, RCT, it's reference-counted, so it adds reference-counting to the resource. So you actually don't keep the actual resource. You don't keep a pointer to that.
09:40
For example, in here, S1 is the pointer to RC type. So that is keeping the actual resource in it. And each time you want to have multiple accesses to the same resource, you just clone the RC. And when all these RC instances go out of scope,
10:05
the actual resource will also be freed automatically for you. So you don't have to care about that. Rust, I didn't mention that, it operates on scopes. So when things go out of scope, Rust compiler frees those things for you. And it decided at build time,
10:21
so there's no garbage collection involved at runtime. And then, but the RC that I mentioned just now, it's only from single thread, so you can't use it from multiple threads. Because Rust wants to make sure that you have it
10:41
so that it's not like... Because it's not atomic, so you can't atomically increase the reference count or decrease the reference count, so it would be a bit unsafe to use it from multiple threads. So for that, we have a concept in Rust called fearless concurrency,
11:01
because if you do concurrent programming in Rust, you are safe, you don't have to be scared of it. In other programming languages, people are really scared of threads, and they should be. But in Rust, you don't have to be, because it has API and concepts that helps you make it safe. And arc is one of those types that it provides for multithreading.
11:24
And it's just RC, but it's atomic reference counting, so you can use it from multiple threads at the same time. Although it only gives you read-only access, so you can't modify from multiple threads, you can't get a mutable state from it.
11:44
And for that reason, there is another data type called mutex that allows you to get mutable access to the actual resource. I would normally explain with samples here, but I have to cover a lot of things, so I won't get into details. But if you have some questions afterwards,
12:01
please let me know, and I can explain each of these things in detail to you in person. That gets us to Gstreamer. What is Gstreamer? It's a multimedia framework. So if you want to write anything related to multimedia, especially audio and video, this is the framework you would want to use.
12:21
It's the multimedia framework of choice for all the Linux platforms. And it has some simple concepts like elements and pipelines. So you create a pipeline for each multimedia application, and in that pipeline, you put elements together,
12:41
depending on your needs. There's a concept of source and sync and filter, and you would connect those elements. It's a bit like Legos. So you have different elements, and you just put them together, and you achieve a goal. I have an example here, so that it's easier to understand what I'm saying. For example, if you want to write a player for OGG files, video files,
13:07
which has audio and video both multiplexed into it. Does everyone know what OGG is? And Vorbis is? Good. So it's simple. You put a file source,
13:20
because you want to first read from a file. You connect it to a demuxer, which demultiplexes the audio and video parts. And then you connect both those parts to elements that decode both those formats. And once they are decoded, they need to be played. And for playing, you have syncs, different kinds of syncs.
13:40
So I can't really see it here. So video sync in here, it's kept generic, because we have multiple video syncs, VLN and X and all those. And also, the audio syncs, there's a lot of them. So there's no specific example here. But anyway, the concept is that in each element,
14:01
you have something called pads. The blue things you see in here, they're called pads. And you don't actually connect elements directly, but you connect them through these pads. And each pad is of type either source or sync. And that means that you can only connect source to a sync. You can't just connect source and source and sync and sync, because that doesn't make sense.
14:20
And all the real multimedia content, they flow only in one direction. There can be events and other things that flow upstream, but all other, the main multimedia content, it always flows downstream through these source and syncs. One example that I would show later,
14:40
it will require this information about something called capabilities. It's called Gstreamer caps in the Gstreamer world. And those caps are on each pad. So when you connect two pads, how do you know they're compatible? The way to do that is through caps. Each pad, when you create your element,
15:02
you declare what kind of capabilities the pad has. For example, in this case, the Vorbis decoder's sync pad will declare that on this SRC pad, I can only take Vorbis data and no other kind of data I can take.
15:21
So if you try to connect the SRC02 pad to the sync of decoder up there, it won't work, because they are not compatible. And that's the way to check the compatibility. It's a plugin-based architecture, which makes it very easy to write apps, generic apps.
15:45
So you're in your application, you usually don't have to care which data formats to handle. You can do that if you really want to and you really need to, but usually, for example, a media player application like Totem or GNOME Video, however you call it,
16:01
they don't usually need to do that. They don't deal with individual different kinds of data. And that makes it very powerful. And it also makes the core of Gstreamer very, very small. So it's really tiny. And the plugins do the actual work. It's written in C for efficiency reasons
16:22
and for many other reasons. A lot of libraries that are used in GNOME, they are in C. So there is a cultural aspect to that too. And it's heavily multithreaded. It makes use of multithreading to make use of your multicore. So in the previous example, for example,
16:41
there will be a separate CPU used for decoding the Vorbis audio and the Theora video parts. It's not very relevant to apps. So if you're an app developer, you wouldn't really need to care about multithreading that much. It really abstracts you, Gstreamer, from multithreading. But sometimes you might have to.
17:02
And also, plugins, usually they don't have to. But in plugins, there's a lot of times when you have to do advanced stuff and then you need to care about multithreading. And then things get really difficult. It's object-oriented programming. It's using something called GeoObject.
17:21
It's a glib API to be able to do GeoObject-oriented programming in C. It's not easy to handle if you are writing plugins and stuff, but it's always very easy to use. It makes the APIs very nice and very easy to use. And why is Rust relevant in Gstreamer and multimedia?
17:41
First of all, parsing of media formats, that's what most of the plugins actually do. It's not just saved by design. So many things can go wrong. There's random data coming from the internet. You can't trust it. And there have been zero-day security advisories in the last some years with Gstreamer.
18:02
Especially with the FLV decoder, there was a vulnerability where you can get access to the host through that. You provide a file that somehow corrupts the memory. I don't remember the exact details, but it was doing something with the memory.
18:20
So it was able to exploit that. But if it was written in Rust, in safe code, it wouldn't have been possible. And the other reason is multithreading. As I said, it's multithreaded, and plugins a lot of times have to deal with multithreading. And in C, it's extremely difficult to handle threading.
18:46
In other programming languages, it's a bit easier, but in C, it's even more difficult to do multithreading right. You will do it wrong, and that would have severe consequences. And as I said, in Rust, you usually don't have to be concerned at all when you do multithreading.
19:04
If you do it wrong, compiler will tell you, and you will satisfy the compiler, and then you won't have any crashes or anything like that, and simultaneous access to data and mutating at the same time and things like that.
19:21
And mutability and ownership concepts of Rust, they map really well with Gstreamer. And for example, in Gstreamer, there's a concept of GST mini-objects. So as I said about G-object, GST mini-object is a variant of that that is a lot light-weight
19:44
than a proper G-object. It has less API and stuff, but it's very light-weight. So if you want something in Gstreamer, there is a lot of things that are created and destroyed very fast. In one second, there will be thousands of these mini-objects
20:01
created and destroyed, and you want it really, really fast, all this stuff. So that's why they invented something called GST mini-object. But the problem is they didn't want to have mutable access to the same mini-object by different threads or multiple parts in the code.
20:20
So they have a restriction there, which is that if it's read-only, unless it's read-only if the reference count is more than one, if there's multiple references to the same object, mini-object, you can't modify it. And there is no real guarantee there,
20:42
but it tries, you know, in C, you can't do much. But in Rust, you can do it much better. I'll show an example if time permits at the end, how exactly that works. And also, you avoid a huge class of memory problems if you write in Rust. That's a generic advantage anyway.
21:02
It's not just specific to Gstreamer, but anything. It also sees an archaic language. It's not just unsafe, it's really old, it doesn't do a lot. Like nowadays, you want more. You can't be handling pointers and everything manually
21:21
and doing everything manually. You need a higher-level language. So Sebastian Drogue, as I mentioned, he wrote these bindings, first of all, for Rust, Gstreamer Rust. And with these, you can use already in your application.
21:40
There's at least one or two applications out there. They're not very mainstream, but they exist and people are working on them. And they are in Rust, and they use Gstreamer in there. So you can do that, but also you can write plugins. And this GST plugins RS, the second repository here,
22:01
that's for that. There's all these plugins written in Rust there. Hopefully, this repository will get bigger and bigger, and you will have more and more plugins written in Rust rather than C, and you can rely on them. Most of the security problems that I mentioned that arise in Gstreamer, they are in the plugins,
22:22
not the Gstreamer core. Because the core is so tiny, it's unlikely that it will have problems. Also, it's extremely well tested over the years, so it's pretty safe in that way. But plugins, they are not very safe, so it's better to write them in Rust.
22:41
Now, I'll have a simple example here, where you can see the advantage of why do it in Rust. This very simple code, this will compile just fine in C, nothing from compiler, no help whatsoever, that there's a lot of problems with this code. It looks very innocent.
23:02
The caps, as I mentioned, caps has these Gstreamer capabilities. It's a representation of capabilities of a pad, and each capability is in the form of something called GSD structure. So that's what's happening here. What we are saying is that, give me the first capability in the form of the structure
23:24
from these caps that we have. And then we are asking it to remove that structure, and then we are using, we are setting, we are mutating that structure after we have told it to remove it.
23:44
So the API of getStructure, this gives you not your own reference on C level. So this thing will crash or cause memory problems in C, but the compiler is not telling you that. So let's see what the Rust compiler says.
24:01
Can you read this at all? Somewhat? So the error you are getting here is about the set. You can see the code, right? So the last statement here,
24:20
Rust compiler is saying to you that the S in here does not have the set method, but I can assure you a structure, GSD structure has a set method, and it should have it, but why is the compiler saying it doesn't exist? The reason is that you are not, on the Rust level, you are not getting structure actually.
24:42
You are getting a thing called option. It's a wrapper type in Rust. In C, you have the equivalence with null pointers. So when you do the same thing in C, you will just get a null pointer, which you will know only at compile runtime, and if you use the null pointer, things will crash unless you are doing a null check.
25:05
But Rust doesn't have that concept. You can't have a null pointer. So what you get is an option, and the option is an enum, which is either a none or some, and if you have some, then that means there is something in there, and then you can get a pointer to that.
25:20
So I'll give you the example code there. So this one, in that regard, will work. So now, you're not getting the structure, but you are unwrapping the... You're using this pattern matching in Rust to... So if you, from this function called get structure, if you get something, it gets assigned to the S,
25:41
and then you can use it and be sure that it won't be null. It's a guarantee. You can also have an else there if you want to handle the case when it's null. But I didn't do that here. So we solved one problem now, but the compiler will give you even more errors now,
26:01
because you really can't see it, right? So when we removed the structure here, in here, you're getting a mutable... You're borrowing caps as mutable because you're mutating it,
26:20
because you're removing something from it. And that's not right, because then you can't modify the structure, and you're using the S.set and you're modifying it. So that's not allowed. And that's why you get some errors about it being non-mutable.
26:41
So we solved the mutability problem by using another function. Instead of get structure, you use get moot structure, and that gives you a mutable reference. And then you can hopefully call these functions, right? And also, the caps, you want to get the mutable reference to it.
27:01
So you want to call get mute on it, and then call get mute structure on that one instead. And all this would not be pointed out by C compiler. There's so many problems with this code. And now you get into another error, which is about multiple mutabilities.
27:21
So when you did this caps.getmute, you got a mutable reference to the caps. And then you do the C.remove structure, and that again tries to get a mutable reference. And so you have multiple mutable reference in the same place
27:43
for the same actual resource. And that's just calling for problems, right? And the compiler is right to not allow you to do that, because as I said, you remove the structure, now the structure doesn't exist. And now you're writing to it, so that just won't work.
28:03
So finally, we will have a code that actually would work. And that's, you get a mutable structure, you get a, sorry, mutable caps and then mutable structure. And then you set what you need to set, and then you can remove the structure if you want. Although it doesn't make any sense,
28:21
you just wrote to a structure that you just removed, but this will work, actually. And this is safe code. That's why it works. Yeah, it works fine. And as I said in C, it will just build, but in runtime you will have multiple problems to handle. But now that you built it,
28:40
it will just work as you want it to work. There's no problem at runtime anymore. I have said everything I've said. Am I out of time? Okay. But just wanted to say, please don't write new project in C and C++. At least consider doing it in Rust,
29:00
because you don't want to create an unsafe world. You want a safer world, and that's what Rust achieves for you, without any runtime costs. That's all from me. I think I'm out of time, but can I take one more question? Okay. Anything?
29:26
Any questions? Either nobody understood anything, or have you understood everything? I would ask a question. Okay. Like, what is it that you do with that combination, like Rust and Gstreamer, in practice?
29:45
So, as I tried to show through the example, when you do it with Rust, you won't have all those runtime problems that, some of them you will see immediately, right? Like the memory access problems. No, no, no. I don't mean the advantages that I got. No, I mean, like, do you know of any projects?
30:03
Because I would propose one. As I said, there is these two media players that I know that uses it, and I think also now there is another desktop GNOME app for... It's a metrics client. I forgot the name.
30:20
And that is written in Rust, and now they are using multimedia for audio, video, or something like that. Exactly, because I have seen... I've come across one use case that is a bit more out of the ordinary, which is Timelands. Have you heard of the project? No. That is essentially taking any video input and compiling it into a timeline where the individual segments visualize a frame,
30:46
so that you can basically anticipate the mood and the parts, specific sections of the film, just by looking at the timeline. It's quite neat, Timelands.io. Cool. And it's in Rust?
31:01
Yes, it's Rust and Gstreamer. Oh, cool. Now I know. No other questions? Good. Thanks, everyone.
Recommendations
Series of 14 media