Fearless Multimedia Programming
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 561 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44264 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Computer programmingMultimediaPresentation of a groupMultimediaMultiplication signComputer animationLecture/Conference
00:31
MultimediaComputer programmingTape driveSystem programmingPointer (computer programming)Default (computer science)Semantics (computer science)Function (mathematics)Type theoryGoodness of fitOpen sourceSemiconductor memoryBlock (periodic table)System programmingData typeFocus (optics)SpeicherbereinigungCodeReal numberPoint cloudFunctional programmingFreewareProgramming languageSemantics (computer science)BitCombinational logicMereologyMemory managementDefault (computer science)Pointer (computer programming)Variable (mathematics)Interface (computing)Different (Kate Ryan album)CompilerRun time (program lifecycle phase)Normal (geometry)State of matterSource codeProgramming paradigmRaw image formatException handlingFormal languageComputer-assisted translationSoftwareCASE <Informatik>Multiplication signParticle systemSinc functionStandard deviationComputer animation
07:34
Concurrency (computer science)MultimediaMultiplicationPlug-in (computing)Object-oriented programmingSource codeHypermediaFile formatSynchronizationCountingData typeMereologyDifferent (Kate Ryan album)Thread (computing)Mobile appVideoconferencingObject-oriented programmingGoodness of fitComputer programmingMultiplication signElement (mathematics)CASE <Informatik>CodeComputer fileSemiconductor memoryGraphics tabletCategory of beingHypermediaShared memoryLine (geometry)Error messageParsingRun time (program lifecycle phase)Physical systemProgramming languageCore dumpType theoryBitPoint (geometry)Software developerPlug-in (computing)Memory managementMultimediaData structureCartesian coordinate systemDirected graphCodierung <Programmierung>Wechselseitiger AusschlussContent (media)Source codeKeyboard shortcutArc (geometry)Atomic numberConcurrency (computer science)Filter <Stochastik>Bit rateLevel (video gaming)State of matterSoftware frameworkCompilation albumCompilerComputer animation
14:20
Source codeFile formatHypermediaMultiplicationError messageRead-only memoryFormal languageKeyboard shortcutData structureInformationComputer configurationSet (mathematics)Content (media)Spherical capFormal languageBlock (periodic table)CodeSource codeObject-oriented programmingCASE <Informatik>Graphics tabletSemiconductor memoryError messageData structureSet (mathematics)Computer configurationElement (mathematics)Maxima and minimaTelephone number mappingMultiplication signState of matterRight anglePointer (computer programming)Dependent and independent variablesRun time (program lifecycle phase)Link (knot theory)Plug-in (computing)InternetworkingWritingBitLine (geometry)Sinc functionThread (computing)Keyboard shortcutCategory of beingConstructor (object-oriented programming)MultiplicationCompilerAxiom of choiceGoodness of fitSummierbarkeitComputer animation
21:07
Axiom of choiceRight angleMereologyVariety (linguistics)Revision controlInteractive televisionScripting languageCartesian coordinate systemFormal languageType theoryProgramming languageCodeTwitterStability theoryPoint (geometry)Musical ensemblePointer (computer programming)Wrapper (data mining)Computer fileProgrammer (hardware)TypinferenzVariable (mathematics)Goodness of fitLecture/ConferenceComputer animation
26:49
Computer animation
Transcript: English(auto-generated)
00:21
So, we start our next presentation. This time on fieldless multimedia programming using Gstreamer and Rust. Please welcome Zeeshan. Thank you for the warm welcome. Can you hear me at the back still? Yeah? Good.
00:40
So, start. First of all, who am I? My name is Zeeshan Ali, as you know. I work for Red Hat, at least nowadays. And most of my background is in FOSS, different kinds of free software, open source stuff. I've been doing mostly on GNOME stuff, but some other things as well. And nowadays I work on cloud stuff,
01:01
or at least I'm expected to. And I love flying, and I love cats, so that's the thing about me. And what am I talking about? I'll talk about two different technologies when they're put together. They make a really good combination. Tim, in the previous talk, I heard that he talked a bit about that already,
01:23
so I don't have to explain a lot, I hope. But these are two, like Sebastian, he's sitting over there. He's the real hero here, I'm not. I'm just talking about the subject. He did all the work and made it possible. So, if you want to thank someone, thank him,
01:42
and buy him beers for this work. And my talk is inspired by his own talk. He has presented about the same subject before. I have a lot to cover, so I'll go a bit fast. But if you don't understand anything, please catch me after the talk, and I'll happily explain things to you if I don't explain well in the talk.
02:06
Firstly, Rust. It's a system programming language, and it's designed from the beginning to be both efficient and very safe at the same time. There are many programming languages out there that focus on efficiency,
02:22
but they are not very safe. And then the others are very safe, but they're not efficient. But Rust is one of the few programming languages out there that focuses on both. And one of the ways it does that is through NLP. There's no pointer arithmetic in normal Rust, in safe Rust,
02:48
which means you can't deref dangling pointers. You can't deref null pointers. And that's one of the biggest source of memory problems in C and C++ and those languages.
03:00
And even in many higher languages, you still have null pointers, raw null pointers that you can easily deref, and you have null pointer exception and stuff. But Rust doesn't have that. But of course, you can't do everything within the safe paradigm of Rust. So Rust allows you to make an exception, which is called unsafe,
03:22
and you just mark your code unsafe. And that part of your code usually deals with foreign function interface with C and other programming languages to that world. But since Rust cannot assume about safety of what the memory handled by those programming languages from FFI,
03:41
it cannot guarantee safety for you. And that's why you have to mark it unsafe to Rust. And it also allows you to do some pointer arithmetic, which you need to do for interfacing with C. But the good news is that since you mark it as unsafe, that part of the code, if you have a memory problem, you know where the problem lies, and it's much easier to debug if it's marked as unsafe, that part of the code.
04:04
Another thing is that the concept is non-mutable state by default. So you cannot, once you assign a resource to a variable, for example, you cannot modify it unless it's marked as mutable. That also does the similar thing, because if you have a problem, any kind of problem,
04:21
you know that it can only be mutable state that's problematic. It cannot be unmutable state that gives you any problems. So it's a similar concept. Another thing Rust has is strict ownership semantics. This is not a concept in C and C++. In some modern C++, yeah, it is, but not really.
04:43
It's not at least enforced by the compiler. So you can still do, even with the most modern C++ standards, you can still do a lot of harm through memory management and stuff if you want to. And it's really easy to do without even wanting to do it.
05:02
And in other languages, it's the same concept, the same thing is achieved with Garbage Collector, but Garbage Collector is always on runtime. So you are using your user's resources to manage the memory for you. But Rust doesn't do that. Rust is at least as efficient as C and C++.
05:22
So it doesn't make use of Garbage Collector. And because of that, it has these concepts like ownership of resources. The concept is very simple. Each resource can only and only have one owner. So once you assign to a variable, for example here, a resource, it's owned by that variable.
05:47
So in this example, this code won't work because once you assign S1 to S2, you have passed the ownership over to S2 and you cannot use S1 anymore because it doesn't have any resources assigned to it anymore.
06:02
Similarly with functions, if you pass by value any resource to a function, you have passed the ownership to that function and you don't own it anymore unless that function returns it to you and you assign it to a variable. If this was S1 equal to some function S1, then it would work
06:21
because you got the ownership back to you. But the normal, the small data types that are efficient to copy, they implement a trait in Rust called copy. And because of that, when you pass it to a function by value or to another variable, they just get copied.
06:41
So you can have, in a way, a multiple ownership, but it's just copies. And if you have a very small data type which is easy to transfer and stuff, or memory efficient doesn't matter to you in your case, then you can implement the same trait and you can just copy things around.
07:04
And you can borrow because if you just have one owner and your ownership gets transferred, it's just not going to work. You have so many problems that you need something like borrowing. So you can, it's a bit like passed by reference in C++.
07:25
You give a reference and you can have multiple non-mutable references to the same resource in the same code block. But borrows are temporary. So we have other data types in Rust.
07:40
Something like, for example, RC, which stands for reference counting. So you can have multiple objects actually pointing to the same resource with this one. And they are reference counted for you. So each time you need another one, you just clone the container of the actual resource. And it's lightweight, so you don't have to worry about that.
08:03
You're copying a lot of the RC structures. But the actual resource might be a bit more heavy, so you use RC for that. And instead of copying or references. So this code will work, for example, because you clone and you're using it separately.
08:24
It's, now we have come to the topic which is kind of like inspiring the title of this talk. In Rust, there is a concept of fearless concurrency, which means that in other programming languages, as soon as you involve threads, things go wrong.
08:43
And it's really, really hard to manage threads in other programming languages. But Rust makes it safe. So you will get a lot of errors from compilers, from compiler, when you first do threading in Rust. But once you get your code compiled, usually it just works.
09:01
And you can rest assured that it's working fine at runtime. And for achieving that, we have data types like ARC. One thing I didn't mention is that you don't have shared, you can't share a state between threads, unless they implement some particular traits, the data type that you want to share.
09:25
And most of them don't, so you need some kind of container that's efficient and at the same time safe to use from multiple threads at the same time. And that container is called ARC, which is just atomic reference counting. So if you need to share resources between the same thread,
09:43
then you should use RC, which is a reference count thread, but it's not atomic, so it's not safe to use from two different threads at the same time. But ARC is atomics, the reference counting is atomic, so you can use it at the same time from different threads.
10:01
Similarly, Mutex, like ARC, is only giving you read-only access. So if you want to modify something from different threads, then you have to put it in another container called Mutex, and then you can achieve the same. That was my really, really quick introduction to Rust.
10:20
I'm sure that you have a lot of questions, but you can ask me later. Now we come to the second part of the topic. It's Gstreamer. How many of you were in the previous talk? Quite a lot, but I'll explain very quickly then. Gstreamer is a multimedia framework. It's very, very popular, especially in the embedded systems out there.
10:43
There's a lot of embedded systems using it. And it uses very simple concepts like elements and pipelines. You have a pipeline for each multimedia application you want to create. You can have multiple pipelines in the same application as well.
11:01
And then you just, it's like Legos. You have multiple elements doing different things. You just connect them to each other, and then you have source, you have filters, different kinds of filters, and then you have a sync to actually play the media on. A quick example is if you want to play AUG video
11:23
with Vorbis audio and Theora video in it, so you will create a pipeline like that. You have something to read from the source, which is in this case a file if you want to read from files, so file source. And then a demultiplexer, which demultiplexes the video and audio parts
11:42
and gives it to these two different decoders, and then they go to the sync to be played, audio and video sync, whichever it is in this case. And the elements, they connect through something called pads, and the way you connect pads is that each pad has a bunch of capabilities on them,
12:05
which means what can they do. For example, in Vorbis decoder case, the sync pad will have capabilities saying that, oh, I can handle Vorbis video of different kinds with different properties. But it can't, for example, you can't connect it to the pad from demuxer,
12:23
which gives you the Theora audio, for example, because it doesn't tell you that I can handle it, so you won't ever connect it. It's heavily plug-in based, so the core of Gstreamer is actually pretty small, so most of the things happen through plug-ins,
12:40
like all the actual capabilities come from there. It's written in C. It's multi-threaded. Well, for most app developers, it doesn't really matter in case of Gstreamer, but when you're writing plug-ins, if you are writing some complex plug-ins, I can be corrected if I'm wrong, but in those cases, you might need to,
13:02
you don't actually have to handle threads, but you have to take care of that, because this code is running from multiple threads, and you have to take care of locking and stuff. Am I right? Yeah, good. So, in a way, you have to care of threading at some point or another,
13:22
and also, I've never seen a Gstreamer plug-in developer who has not touched the core, like if they keep doing plug-in development, at some point, the line gets blurred very soon. And it's a very object-oriented API using the GeoObject type system from glib,
13:43
so you can do object-oriented programming on C-level, which means that on higher level, if you have bindings, then they get exposed in an object-oriented way as well. So why is Rust relevant here? First of all, the major thing is
14:01
Gstreamer has to be used for media parsing quite a lot, and it basically is about media parsing, all it's about. And you cannot really just trust media content, like just start parsing it. You can have many kind of memory problems, and you can have kind of memory problem-based attacks and stuff,
14:26
especially if you're getting it from untrusted sources, which you usually do from Internet and stuff. So a safe language would be a really good choice for this, because it makes sure that you don't do most of the memory.
14:40
You don't make most of the memory handling errors in it. And another thing is multithreading is hard. It's hard in every language, especially in C and C++. It's even much harder. So as I said, Rust makes multithreading really easy to handle.
15:02
Easy in the sense that it makes sure that you don't do most of the errors you will do when sharing, for example, state between threads. So since it doesn't allow you to do those things, it makes it easier, and that's why, in case of Gstreamer, since it's multithreaded-based,
15:21
so it's very relevant to you. And the concepts of mutability and ownership in Rust, they map really well to the Gstreamer concepts. I had an example here, but I'm skipping it, because we don't have enough time. But I'll have other examples later.
15:40
And to avoid many memory errors in general, as I said, in C and C++, it's really easy to do those, so Rust lets you avoid those. So also, C is a very archaic language. It's not just actually unsafe. It's from the 70s. What, 70s? 60s? Something like that. 70s, yeah. Thanks.
16:01
So it's really old. We need some modern language which has modern constructs and stuff like that. So Sebastian's sitting over there still. He made these first Rust bindings, which is the first link here on GitHub. I think it's now on GitLab, right?
16:21
On Gstreamer, it's been merged. I forgot to update these links. Anyway, you can find it easily. And those bindings cover a lot of API. I think most of it, all of it? I can say all of it, yeah. And then he wrote a bunch of plugins based on, in Rust, based on these bindings.
16:46
So you have really good examples already of how to write plugins in Rust if you want to. And there's a place for them to go to upstream. If you want to write a new plugin in Gstreamer, in Rust, you can do that. A simple example how it helps, Rust helps Gstreamer.
17:03
So I mentioned in my introduction to the pipelines and elements that you have paths and you have capabilities on them. Those capabilities are represented in Gstreamer API by something called caps, GST caps. And each capability, because each pad can have multiple capabilities, right?
17:22
So each capability is represented by something called structure. And so caps can have multiple structures, right? And this code, what it's trying to do is, it's a very silly code, actually. You won't never actually use that. So it's taking off the first structure in the caps. There is the first capability.
17:41
And then it tells the caps that I want to remove this capability now. And then the capability it got, it's trying to set a G object property on that. In C, if you write the same code in C, it will compile or it will be happy. It will let you do what you want. But you will have multiple memory problems at run time.
18:03
But Rust would not allow you to do that, as you can see here. This is the first error you will get from compiling this code. And it seems a bit weird at first, but I'll go back. If you see the first line, you see get structure.
18:21
But there could be that caps doesn't have any structure. So how does Rust handle it? Because as I said, it doesn't do null pointers. So instead, what it has is an enum. Sorry. Instead, it has something called, it has an enum called option.
18:43
And that enum has two values, sum or none. In case of none, there is no data associated with it. But if it's sum, that means there is something to be given, like it's not null. And then you can get that object or resource, whatever it's in the option, in sum.
19:03
And so that's why you're getting this error, which just says that what you got was option. And option doesn't have this set method you're calling on it. So it's giving you an error. So let's handle that. Now we parse that option.
19:21
Rust gives you a nice construct to parse these return values. So you just say, I want to assign this to, if it's sum, then I want to assign it to S, the actual structure. And then you use it in that block. And then you just ignore the case of none here. But you can handle it. If you have an else block here, that will handle the none case.
19:43
So now it should work, right? But no, you have other problems now. You have a problem that you're trying to modify something that is not marked mutable. So the return value of this get struct, like when you get something, it's not mutable.
20:00
It's not marked mutable, so you cannot mutate it anymore. And you are trying to do it when you say caps dot remove structure, and you are doing S dot set. Both things are modifying something, so they need a mutable reference to work with. So that's why you're getting all these errors.
20:20
So that's all the mutability issue. So instead of that, we just tell caps that give us a mutable reference first. Then you get a mutable reference to caps. Then from a mutable reference of caps, you can get a mutable reference structure. And then you can do what you want, right? So you do, it's the same thing, remove struct and then set.
20:42
Am I out of time already? Okay. There's also then simultaneous mutability issue, but we don't have time. Anyway, the main thing is that C compiler would have let you do all these horrible things, but Rust stopped you from doing all these things.
21:03
And then in the end, you come up with really, really safe code. It was still useless, but it was safe. So anyway, I'll skip this. Thanks. We have a few minutes for questions, if you want.
21:42
Hi. How hard is it to have ABI compatibility, basically, with Rust? Because in C, you can just define something in the C file, and then you don't expose it. Or GObject, you also have the private kind of offset struct thing. How do we do that in Rust?
22:00
As far as I know, you can't really. There's no ABI compatibility right now. The long answer is a bit long, so I can't explain everything. But in short, it's not easily possible at least.
22:21
Other questions? Yeah. So you mean that you have a C wrapper, and then you... What I was just saying is you can always export a C ABI, and that's stable.
22:44
It's just that Rust itself doesn't have a stable ABI at this point. But you can export a C ABI. You can tell Rust that I'm on PC. Other questions?
23:04
All right, then. Well, it was crystal clear, I guess. Everybody understood everything. It's cool. Don't hesitate. We still have two minutes, if you want. Yeah. Does the Rust version generate basically exactly the same code as the safe C version would?
23:36
Or is it better, or is it worse? It's not the same, but it's safer.
23:42
You have to do something to make it safe, right? Can't be the same. You don't have null pointers even at C level, as far as I know. Last question, maybe? I have two questions.
24:01
Good. I was just wondering, in your examples from Rust, you were defining a variable with...
24:20
Sorry? You were defining a variable with let? Yeah. But it was immutable? Yeah. And then you were still defining in your example again as let, but then you got back a mutable variable. Yeah. Isn't that kind of... Well, you got a reference, like when you do the let, so you don't always have to give it the type.
24:41
So Rust can do type inference. Okay. So it was getting a mutable reference, so it just got... But how would I know if it's mutable? Because of the return value. Okay. I've never done anything with Rust, so I was just wondering, too.
25:04
At least in the method, it said mut, so it was obvious from the method, at least. Yeah, yeah. We sometimes talk about glue languages, as in higher-level languages, like script languages. And is there a trend, or can you see a trend, where Rust could be used as a subsystem for higher performance,
25:26
and a higher-level language, or like script, or maybe Go, because I'm more of a Go programmer, where they both could collaborate in an application? Does that make sense? Are Rust applications meant to be entirely written in Rust?
25:43
Well, you have a way to do FFI through unsafe code. So you can interact with any kind of language through that, not just C. But it's not like... I don't think there's a helper to make it easy for you, specifically for Go, I mean, or any particular language.
26:03
I think there is something for Python that makes it easy to do, like, interaction between Python. The guys sitting next to you would know about that. Now the other side, yeah. Yeah, you. But no, I don't know if I need Go, specifically for Go. I mean, as in Go coding Rust.
26:23
Or either way around, yeah. Okay, thanks. But as Sebastian said, you can expose a C ABI, an ABI, through Rust. And from Go it's really easy to do C, so you can use it that way.
26:43
Thank you, Isham. Thank you, Aaron.
Recommendations
Series of 14 media