GStreamer embedded state of the union 2019
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 561 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44307 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Embedded systemCollaborationismMultiplication signBitState of matterCASE <Informatik>Beta functionComputer animation
00:38
Open setCollaborationismStreaming mediaSystem callComputing platformTape driveVideoconferencingComputer animationJSONXML
01:05
Open setStreaming mediaCollaborationismPoint cloudPhysical systemProduct (business)VideoconferencingComputer animation
01:40
PlastikkarteCollaborationismSpacetimeWorkstation <Musikinstrument>SpacetimeMultiplication signStress (mechanics)Bookmark (World Wide Web)Computer hardwareField programmable gate arrayCuboidGreatest elementProduct (business)Topological vector spacePlanningSet-top boxSoftwareSlide rule
03:10
CollaborationismRevision control1 (number)Electric generatorPlanningComputer animation
03:42
Workstation <Musikinstrument>SpacetimeCollaborationismPlastikkarteTopological vector spacePlastikkarteData compressionSoftware frameworkVideoconferencingProduct (business)OvalOperator (mathematics)Computer animation
04:58
CodecElement (mathematics)Kernel (computing)Open setInformation securityProcess (computing)Computer hardwareComputer networkData bufferSystem programmingFile formatParsingVideoconferencingAsynchronous Transfer ModeParameter (computer programming)AerodynamicsDirected setMemory managementSynchronizationShader <Informatik>Asynchronous Transfer ModeMultiplicationProcess (computing)Multiplication signComputing platformGoodness of fitSpacetimeKernel (computing)Element (mathematics)SoftwareFlow separationVideoconferencingDirection (geometry)Electronic visual displaySoftware bugDynamical systemField (computer science)AlgorithmSynchronizationCodecPattern recognitionBitCASE <Informatik>Graph coloringParsingFrame problemFile formatSet (mathematics)Streaming mediaAnalogyComputer hardwareMedical imagingBefehlsprozessorDifferent (Kate Ryan album)Buffer solutionSoftware frameworkLine (geometry)Data conversionTwitterLevel (video gaming)Stability theoryElectronic mailing listLatent heatProxy serverCodierung <Programmierung>Cartesian coordinate systemMereologyShader <Informatik>Raw image formatInternetworking2 (number)Cache (computing)Semiconductor memoryValidity (statistics)AreaMaxima and minimaInformation securityData compressionScalar fieldTransformation (genetics)Data streamRight anglePhysical systemComputer fileOpen setBinary fileResource allocationDigital rights managementState of matter
14:25
CollaborationismOpen setComputer hardwareBranch (computer science)CodeBuildingComputing platformPrototypeEmbedded systemBitLoginSoftware frameworkJava appletFigurate numberWindowProcess (computing)Tracing (software)Right angleAndroid (robot)Artificial neural networkBranch (computer science)INTEGRALPrototypeNeuroinformatikPlanningSoftware testingMultiplication signTape driveInformationProjective planeComputing platformVideoconferencingStreaming mediaFrame problemOpen sourceComputer hardwareWorkloadRemote procedure callPoint cloudCartesian coordinate systemFinite-state machinePrice indexElement (mathematics)Software bugMobile appGoodness of fitRow (database)DemosceneOperator (mathematics)Graphics processing unitSoftwareNear-ringComputer animation
19:48
Embedded systemCartesian coordinate systemProcess (computing)Link (knot theory)CASE <Informatik>Network socketPrinciple of maximum entropySemiconductor memoryMereologyPoint (geometry)Codierung <Programmierung>1 (number)Buffer solutionMessage passingPatch (Unix)Population densityIndependence (probability theory)Entire functionServer (computing)Streaming mediaComputer hardwareOpen sourceSynchronizationPlanningState of matterInterprozesskommunikationBitKernel (computing)ImplementationComputer fileGame controllerMixed realityVideoconferencingDirected graphSoftware development kitData bufferNetwork topologyPattern languageFlow separationPlug-in (computing)DebuggerIntelBefehlsprozessorComputer animationEngineering drawingLecture/Conference
24:47
Canonical ensembleComputer animation
Transcript: English(auto-generated)
00:07
So it's time. Let's start again with, again, just from a talk about this time embedded state of the union. Please welcome Olivier.
00:21
Hello, everyone. So this is a bit of a follow-up to Tim's talk for those who were there an hour ago. But this time, I'm going to really focus on what's new for embedded users' use cases. So first, who am I? Quick introduction. My name is Olivier Caet.
00:41
I've been a Gstreamer contributor, developer, for 12 years now at Collabra since 2007. First I started doing video calls using telepathy for the MeeGo Maimo platform with Nokia. And since then, I respect everything that you can do with Gstreamer, from video editing
01:07
to embedded systems to transcoding cloud systems. But a lot of the work that I've been doing in the last couple years has been on embedded systems. So this is why I'm going to talk about them today.
01:24
Just a quick introduction. What kind of embedded systems use Gstreamer? And the answer is a lot. Everything that does video, audio, and embedded systems, you will find Gstreamer in many, many products there. Sometimes we're really surprised to find them.
01:41
So for example, in the TV space, it's pretty big. On my slide, you have some LG TVs, some Samsung TVs. A lot of these actually have Gstreamer on the inside, on the smart side. On the top left, you have the Xfinity box from Comcast. For those who are not from America, Comcast is the biggest cable company in the world.
02:02
And every set-top box that they ship is Linux box with Gstreamer for all the playback. And then there's a bunch of others. On the bottom left, it's Youview, which is a British company. More set-top box. A lot of the TV space has it on the end point.
02:21
But now actually, a lot of the recently, it's been growing in the production side as they've been transitioning from hardware and FPGAs to software-based workflows. And Gstreamer has seen a lot of traction there. In-flight entertainment is another big one that people interact with all the time. So in a lot of planes, almost all the planes, modern planes these days, by modern
02:44
I mean from the last 10 to 15 years, you go and in-flight entertainment, every time you play a movie, that's Gstreamer. Even in the space station, this is a really cool one. It's one of my favorites. I show it every time. It's a little camera that floats inside the space station made by the Japanese space agency.
03:07
And that has Gstreamer inside. So I spoke about these, but there's also a bunch of other devices like security cameras. A lot of the high-end ones have Gstreamer on the inside.
03:21
Yes? Could you elaborate on what version of Gstreamer is open or one could do it? Where? By this? So it depends. All the new generations are 1.0 generally. The ones that fly in the sky, in the planes, these are probably from 10 years ago. So that's probably whatever version they deployed in the very past.
03:41
But for example, this is 1.10 something, it's pretty recent. Smart TVs, some of them are actually quite like the Comcast guys are on 1.14 as far as I know. So a lot of these, actually, they keep up because they want to deploy new features, right? They need the newer dash and the newer HLS features.
04:03
So they have an incentive to keep up. And they have like rapid deployment and all these things in this industry now. It's not like, especially in the TVs, in the cable operator, they really want to deploy new features very quickly. So I was saying there's all of these, but there are also a bunch of others, like industrial
04:23
equipment also, that you don't think about, but they process video in there. And they use Gstreamer very often because it's on many embedded chips. When you buy them, Gstreamer is the framework that comes already working. If you buy Xilinx FPGA that have a video encoder there, well, the framework that Xilinx
04:44
enables is Gstreamer. So it's the quickest way to get a working product. So I'll give a little summary of the things that we have done that Tim hasn't covered that are really specific to embedded. First, a lot of work has happened again this year around Video for Linux codec support.
05:05
So Video for Linux is the Linux kernel API that is used for things that have a queue, so video encoders, video decoders, amongst other things, displaying things, capturing from a camera, some display devices, some other video processing devices,
05:22
scalers, and things like that, that are not in the display path, that are used in memory-to-memory mode. So there's a lot of these, but a big part is video encoders and decoders. In the last year, we've merged the HEVC encoder and decoder support. We have a JPEG encoder, a plug-in, and a VI codec, which is kind of interesting,
05:47
because it's not a useful codec at all. It's only to be able to test the kernel infrastructure. So it's a fake codec that is implemented in software in the kernel to be able to test the whole codec infrastructure inside the kernel without having to deal with the actual hardware.
06:06
Another nice feature, especially on the camera side, some cameras took seconds to probe, because we enumerated a bunch of things that we didn't really need to enumerate. So Nikola has done a lot of work, and now the device probing is instant on almost all hardware.
06:22
So you can get the list of all the devices and the relevant capabilities. Last year, I talked about stable element names for encoder and decoders. So originally, for video for Linux elements in Gstreamer, we were generating a new element name every time a device would pop up.
06:41
So we'd connect a new camera, and a new element, a new encoder, and the element name would change. It would appear with a new name, which means that on some system, every time you reboot it, the element would have different names, which is good, which is OK if you're using Play Bin, and it's an auto-generated pipeline, because you don't care about the name.
07:01
But if you're doing pipelines by hand, it was a bit annoying. So now we have these still, but we also have a set of elements that have static names, and then you give it the device so you can control it more manually. So we had this for encoders and decoders last year. This year, we've also added the same thing but for transformational elements.
07:21
So transformational elements are things like scalers or color converters or different elements that normally convert raw images into different formats. One thing that is being discussed that's not there yet, because the kernel is not there, is having good support for stateless codecs.
07:44
So most codecs that people traditionally use on these embedded hardware, you would just give it the H.264 bitstream, and it would give you a decoded stream. It did the whole parsing of the bitstream and everything in the hardware side, which was often just firmware running on a different chip.
08:01
But now the new trend is to make the hardware cheaper by moving all of this parsing onto the CPU side and by doing it in user space and software. So you need slightly different APIs in the kernel. There's a lot of work happening with the request APIs in the kernel to do exactly that, and this human will support it once it's there.
08:22
This is something actually we've been actually working on. Completely separate subject now. We've merged a plug-in called IPC pipeline, which I like to talk about because I think it's really cool, and it allows you to split a pipeline into multiple processes but the master pipeline controls all the others.
08:45
So for example, the typical case is that you have one process that talks to a network that downloads the dash stream from the internet that is exposed, and then it passes the data to another process which actually talks to the hardware decoder.
09:00
So you can separate the hardware decoder, the part that, and these are often not as secure as you would like. They do a lot of bitstream parsing and things like that. So you want to make sure that this part that might be compromised by an invalid stream is not connected to the network and vice versa. So we can actually split the pipeline in multiple levels.
09:22
I have one stage that talks to the network, one stage that does the parsing, and then a separate stage that talks to the hardware. And this is also useful to implement DRM, sadly. So it allows you to have multiple levels of security, and right now, you have to create everything manually,
09:46
so it's not automatic at all. You have to actually create your pipeline manually, but for these kinds of devices, you actually want to control exactly what happens and what process and everything. Another completely separate thing, we've implemented a new mode of interlacing
10:02
that actually exposes how some of the hardware works. So traditionally, interlacing is that you have one frame in traditional analog TV. You have one frame that contains every outline, and then the next frame, which is the next field, contains every event line. So instead of having 30 frames per second,
10:20
you have 60 fields, and these fields are really half the frame, one odd, one even, odd, even. And when these come to digital, mostly what you did is that you put both of these fields in the same file, in the same frame, in the same buffer. So you have the odd lines, the even lines are taken at slightly different times,
10:43
and this is why when you look at it without properly interlacing it, you'll see a jagged image. This is a traditional way to do it. Some hardware, NH265 actually, do it in a different way, is that you actually have the fields separately in separate buffers at separate times.
11:00
This allows for a bit lower latency, so you don't have to wait for both fields that have been captured to start processing the first one. And we've implemented this for JSTOMX and actually throughout the Jestremer framework. We've done also a bunch of work to reduce the latency in RTP pipelines.
11:23
When we actually tried to measure the latency of the actual data, we realized that the latency that Jestremer claimed to have was not really true, that there are a bunch of little bugs all around the different elements that actually made the data stay in there longer than it should have. So we did a bunch of little bug fixes,
11:42
and now we can actually push buffers with the latency that it claims to have, which can be almost zero if you're not doing anything that, any queuing. We also done a bunch of work in the Jestremer OpenMAX elements.
12:01
A lot of this is for the Xilinx Zinc MP platform. We fixed a lot of bugs. We've added support for 10-bit video formats for HDR. We have a lot more DME buff and zero copy modes so that you can connect them in different ways and have one side or the other do the allocation,
12:22
depending on what's better for your specific use case. We've had the region of interest. This is a really cool one, I think. Basically, it allows the application to say, in my image, this bit is where the really important thing is, so put more bits there when you compress.
12:41
For example, you can have a face recognition algorithm that says there's a face there, so in my video, everything else is not that important, but the face has to be recognizable, or the car plate, or things like that, or the subtitles. If you have subtitles burning through the video, you might want to put more detail there so that the text is readable, even though you really compress the rest of the video a lot.
13:03
Another little thing is the dynamic frame rate in the encoder. Now the encoder, if you change the frame rate, it doesn't reinitialize everything. It just changes the frame rate. So you can change the frame rate really quickly. Another nice thing, we've done a bunch of little improvements
13:24
on the DMA-Buff support. One of them is that now we do explicit DMA-Buff synchronization. It turns out that the Linux kernel, when you use DMA-Buff buffers, when you access them, you actually need to tell it, I'm going to access it, and now I'm stopping accessing it so that it does the appropriate cache invalidations.
13:43
Otherwise, you get corruption, especially on Intel hardware, which is one that actually lots of people use. And we have a direct GL DMA-Buff uploader. So traditionally, when you would import a YUV image,
14:01
you would have to use a shader, convert it to RGB to be able to be put on a display. But some hardware, in particular Vivante and the iMX6, they can actually take the YUV and display it directly. Entirely in the hardware, so there's a direct uploader now
14:21
that bypasses this shader that does the conversion. So these are the things, a bit of a summary of the things that I found by looking at the Git log and trying to remember what happened in the last year. These are some things that are actually being worked on more or less actively. But these, I think, are kind of interesting.
14:44
There's been a lot of activity around neural networks in distro. Neural networks are all the rage this year. And a lot of it is to process video, to recognize things in video. And distro is really good with video. So we have, for example, NVIDIA has released something called
15:00
the Deep Stream SDK that works both on embedded Tegra and on the cloud side. And that uses CUDA to do the actual neural network processing. But all of the video elements are actually using Gstreamer. This is largely proprietary, sadly.
15:22
But the Gstreamer, actual Gstreamer is not. But all of the interesting bits that NVIDIA has done is using their proprietary things. But there's also a couple of open source projects that have been released. There's one called NN Streamer. There's something called JST in France that's meant to be released next month. They promised to do neural networks with Gstreamer.
15:43
So I expect that in the next year or two, we'll probably have something upstream that everyone can collaborate on to integrate neural network frameworks with Gstreamer. Another thing that's coming up now is different companies are coming up with specialized hardware.
16:01
So instead of using GPUs, yes. We will use TensorFlow. A lot of these frameworks actually integrate TensorFlow or TensorFlow RT with Gstreamer. That's the main one. But I know some people are actually using other frameworks, too.
16:22
Yes. So yes, also specialized hardware is coming. So a lot of these are specialized accelerators that are not like GPUs, but have been really designed for AI workloads. And these will require some integration. And Gstreamer is probably going to be a key thing there.
16:41
Another branch that we have not merged in a long time, but I have promised to review, is Android Camera 2 API. So this is a new camera API that Android has had for a couple years now. And that is completely more modern than what we're using right now. This new API, it's exposing the NDK.
17:01
So you can use a native C API to access it, instead of going through Java and going back to the C++. It does things where you can record a video and take a picture at the same time. It allows you to capture multiple streams at the same time, like the front camera and the back camera at the same time on some phones,
17:22
et cetera. And to have all the modern features that your camera application has in your phone, they're all exposed to that API. So there is a plug-in in the GitLab that is meant to be reviewed, and hopefully we'll merge it in the near future.
17:42
There's also been a bunch of work about remote tracing. So Gstreamer has a tracer framework that allows you to write tracers to trace things. These are actually nice to find performance, brought on the next, figure out what exactly is going on in the pipeline when it's running. And now there's some interest to do it remotely,
18:00
so to have some infrastructure to forward this information to a separate computer. So you can have the tracer running in your embedded device, but have a nice UI on your computer to know what's going on in my pipeline and why are my frames being dropped, even though many indicators say they shouldn't be. So where is the bottleneck? What is the bug? What have I been doing wrong?
18:26
Another next step that we're going to do, that we've just switched to GitLab, where we gain amazing new technologies, such as doing a build before you merge the branch and not after, and running tests before you merge instead of after.
18:42
And in this move, right now we build for Linux x86-64, we build for Android, and we build for Windows. And we would really like next to build for an embedded platform to reflect where Gstreamer is most used. And the next step after that will
19:01
be to actually run tests on an embedded platform. We've built a prototype using a Raspberry Pi. And the prototype used Jenkins and Lava. So Jenkins to do the build, and Lava to actually run on the device. Our goal would be to replace Jenkins with GitLab CI here, so that we can actually integrate it nicely in our CI.
19:25
We have a vague plan on how to integrate GitLab CI with Lava, but I don't know if anyone else has already done it, so we actually have to actually see if it works, to use the GitLab CI APIs to drive Lava, and have all the details together.
19:44
So this is basically what I had to say. Do you have any questions? Yes. Yes, you.
20:02
I'm curious when all these new things will come to NVIDIA SDK for Jetson. So the NVIDIA part is already there. It's called Deep Stream SDK. It works on Jetson and on whatever the server ones are called. On this point of interest, too?
20:22
No, no. A point of interest is that, as far as I know, is only available on Intel and Glide Links. So it needs support in the encoder. I don't know what the NVENC API can do. So but the density will be in all platforms, hopefully. In a few years, it will be everywhere.
20:42
Well, yeah. Especially this region of interest thing, it really depends on the actual encoder implementation. So I don't know what they have in their implementation. And I have another question, but maybe later. Yeah, go for it. Other questions? Yes.
21:10
I have two questions. You mentioned before this IPC pipelines, too. Great. There's another plug-in called Interpipe, which allows to make a separate pipeline
21:23
and link them together. What would be advantages of this one or that one? So the main thing that IPC pipeline does differently from Interpipe is that IPC pipeline actually controls the pipeline entirely. So when you put the master one in the pause state,
21:41
it puts all of the other ones in the other processes also in the pause state. So the control is unified. Well, Interpipe line, kind of the point of it is that it's the other way around. They're really separate. So you have a sender and a receiver, and you can stop, let's say, stop the sender and change it and stop the receiver independently.
22:02
Well, IPC pipeline is really meant to have things that, through the application, appears as a single pipeline, even though other bits might actually be separate in the process. If I would want to build like a very dynamic application where I create the parts of the pipeline at the runtime,
22:22
which one would you recommend to use? So if you want to stop and start different pipes, the inter-video sync source and the inter-pipeline, whatever it's called, inter- Interpipe. Interpipe, yeah, is probably more suitable. IPC pipeline is more meant,
22:41
it's actually meant for like playback cases where you want to separate things, but it's not because you want them to run separately. It's because you want them to actually appear as one thing to the application. And the second question is more on the embedded side of it, but the same IPC. If I'm using like very low powered IMX6 CPU,
23:05
and I have some hardware source of video, if I would split my pipeline in parts, how would it pass the buffers between the pipelines? Would it be like a mem copy and it would be big performance hit? Yes, so right now IPC pipeline, it's a socket.
23:22
So actually copy everything through the kernel. For the use case that we're looking at, it was fast enough. But I had the plan to actually pass memory buffers, either passing the file descriptor or shared memory, but we never implemented it because it was not required for our use case.
23:44
Last question. Question regarding the debugging tools, are you actively working on it
24:01
or is something already available? So there is something available from the guys from Samsung. I can't remember the name of it. It's open source. It's on this GitHub. Does someone remember the name of the thing that Martin worked on? The Oak Tracer?
24:25
Oak Tracer? Oak Tracer, yeah. So that's one of the efforts. I would like to see it more integrated. Right now you need to patch things to actually use it live.
24:41
Okay, thank you Olivier. Thank you.