Upipe video pipelines for multimedia transcoders, streamers and players
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 199 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/32648 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
DataflowSource codeBuildingSoftware frameworkHand fanElectronic data processingProjective planeComputer animation
00:29
Software frameworkCartesian coordinate systemComputer fileStandard deviationSpectrum (functional analysis)Module (mathematics)Genetic programmingStreaming mediaModel theoryCodePoint (geometry)Execution unitComplex (psychology)Limit (category theory)Instance (computer science)Multiplication signComputer programmingLevel (video gaming)Goodness of fitDifferent (Kate Ryan album)Moment (mathematics)Web 2.0CASE <Informatik>ProgrammschleifeEvent horizonData structureServer (computing)Internet service providerSoftware developerTwitterSystem callElectronic visual displayFunction (mathematics)Civil engineeringDomain nameDecision theoryView (database)Derivation (linguistics)Task (computing)Element (mathematics)Right angleFamilyProjective planeFormal grammarMereologyAdditionTunisNatural numberTableauPhysical systemMusical ensembleComputing platformAreaSource codeTranscodierungResultantTDMADistribution (mathematics)Particle systemError messageProcess (computing)Electronic data processingModule (mathematics)SuperscalarParallel computingPresentation of a groupConfiguration spaceVideoconferencingCore dumpMultimediaDepictionFreewareLeakSemiconductor memoryEvent-driven programmingAutonomic computingOpen sourceFlow separationBefehlsprozessorBroadcasting (networking)Validity (statistics)HypermediaPhotographic mosaicSoftware maintenanceElectronic mailing listAttribute grammarBuffer solutionWritingReading (process)Lecture/ConferenceComputer animation
08:34
1 (number)Network topologyWebsiteInformationMultiplication signSubject indexingAtomic numberLibrary (computing)String (computer science)Touch typingSoftware frameworkCartesian coordinate systemSystem callPoint (geometry)Video gameMoment (mathematics)Level (video gaming)Streaming mediaThumbnailHash functionComputer fileBlock (periodic table)Right angleSoftware testingCodierung <Programmierung>PlanningNumberProduct (business)Thread (computing)Model theoryMereologyBitComputer programmingQuicksortInstance (computer science)Stress (mechanics)Function (mathematics)Elementary arithmeticTwitterStreamlines, streaklines, and pathlinesCausalityEvent horizonSource codeEuler anglesBranch (computer science)Semantics (computer science)Attribute grammarBuffer solutionDifferent (Kate Ryan album)ImplementationMultimediaHypermediaCopula (linguistics)Module (mathematics)Social classData structureData exchangeTerm (mathematics)WeightCASE <Informatik>Run time (program lifecycle phase)Type theorySoftware developerWrapper (data mining)CodeComponent-based software engineeringFilter <Informatik>Queue (abstract data type)Standard deviationDigitizingModule (mathematics)Reading (process)WritingCountingUnit testingSuperscalarCore dumpLoop (music)Presentation of a groupEmailDirected graphFingerprintHD DVDComputer hardwareProjective planeSingle-precision floating-point formatGraphical user interfaceComputer animation
16:36
Thread (computing)Software frameworkForcing (mathematics)ArmError messagePoint (geometry)Unit testingInstance (computer science)Source codeDigital rights managementTheoryLine (geometry)Module (mathematics)Function (mathematics)Software testingVector spaceMultiplication signQueue (abstract data type)LeakBuffer solutionSemiconductor memoryGame controllerEmailFitness functionCodeModule (mathematics)Computer animationLecture/Conference
Transcript: English(auto-generated)
00:00
I'd like to thank my fan club first. I'm not talking about VLC today, but I will be talking about Chupipe, which is a brand new project, what we call a flexible data flow framework. So what is it? It's a framework that allows you to build pipelines of data processing from data sources to data sinks, one
00:26
or several sources to one or several sinks. When I'm speaking about data, I mean essentially pictures and sounds, because we're in the multimedia business. But it can be suitable for any kind of data that has a
00:42
notion of frames or packets. So the pipeline is composed of modules that we call pipes. And Upipe is a framework that defines APIs that allow first to configure the pipes. So typical configuration commands would be, send your
01:01
output to this other pipe. So that is the Upipe set output command. Or open this file for reading or writing. There is an API to feed data into the pipe. This is a packet, process it. Upipe also defines an API to allow modules to throw events.
01:22
What I call event is, for instance, well, the end of file is reached. What do we do? Or there has been a fatal error. What do we do? Should we terminate the application now? So that's the kind of event a module can throw. And so we have an API to catch that. We also have an API that defines what a buffer is in
01:44
an efficient manner. I'll come back on that later. And it allows us also to give arbitrary attributes to each data buffer. And finally, there's also an API that allows us to interact with an event loop.
02:00
I'll come back on that later as well. Upipe is also a distribution of standard basic pipes, and we'll see a list at the end of the presentation. So what could it be used for? Well, when the multimedia business and the company I work for, Penheadens, develops Upipe, it's selling
02:22
broadcast equipment. So that would be typically transcoders, multiplexers, playout systems, mosaic systems, where you can have several video sources inside one output. And also, we've worked on embedded platforms to make embedded distribution platforms or
02:42
embedded media players. The framework is very lightweight, so it's quite adapted, quite tootable for that. The question you're probably asking right now is, why yet another multimedia framework now in 2014? Isn't there already something that exists for that? Well, yes, there is.
03:01
There is VLC that I know pretty well. There is Gstreamer, of course, MLT framework, and probably other very valid open source projects. However, most of these projects are at least 15 years old, and there have been new trends that emerged since the times they've been designed.
03:21
In particular, there is a generalization of superscalar architectures. Now, all CPUs have several cores of execution. Even your cell phone has two or four cores. So the framework has to provide ways for applications to take advantage of several cores if needed.
03:42
Also, a new trend in the development industry is what we call event-driven loops, or asynchronous programming. That's something that's been introduced with the event, or the BBV, for web servers initially, something like Nginx. And it's a technology that allows you to register events.
04:01
I want to be notified when there's something to read on that socket, or I want to be notified when that timer expires, and you will get called back by the framework. And so this is kind of a new way of programming applications. Actually, it's been generalized in the past 10 years, and the multi-media frameworks need to take that
04:22
into account. Also, in existing multi-media frameworks, maintenance is made more difficult because of a lack of modularity and the complexity of the modules. For instance, a transport stream, DMUX, in most frameworks, is just one module. So you have just one entry point that you configure.
04:43
In your pipe, if you look at the source code, we have like 10 modules to implement a TS-DMUX, one for every DBB table, and so on. And so you can arrange and fine-tune your TS-DMUX as you like. And also maintenance is more difficult because of the confusion in some of those frameworks between what we
05:02
call data processing and decision. What I call decision is deciding which elementary stream I want to decode. I want to trench audio. I want subtitles, not. And which path, which module is really instantiant. Most of the time, this is done either by the modules themselves or the framework itself.
05:21
And we think it should be the task of the application. So we've designed the API to allow the application to get events from the modules and do the decision part itself. Also, I'm in the professional broadcast world. And the problem we have with most multi-media frameworks
05:41
is that they've been designed for one kind of application, which is please play this file, and use a display output, GLX or whatever. And sometimes we do want to access lower level APIs, like to do transcoding with fine-tuning, and generate
06:03
more like a hack into the framework. So U-pipe has been designed for that kind of use case. So we started writing U-pipe only two years ago with new policies. So we've decided to specify it bottom up.
06:23
Usually, you specify the API top down. This is the API I want to expose to the application. And then I go down and try to find the simplest data structures. We did the contrary. We asked ourselves, what is the simplest structure that could
06:41
represent a module or pipe? What is the simplest structure that can represent a buffer? And then we built on top of that layers of APIs. And so different level of APIs are possible with U-pipe. You can talk to the lower level API, which we currently mostly use. But you can imagine a higher level of APIs as well.
07:01
We tried to keep modules as simple and autonomous as possible, that's the Unix philosophy. And that allows us to do very intensive unit tests, including check the memory leaks with Valgrind. All our code goes through Valgrind. We also have an emphasis on documentation.
07:21
Probably if you've seen our website, you can see we have tutorials, and not only Docsigen, but also comprehensive documentation on the frameworks. And one last thing that is very appreciated by professionals is that we try to follow standards as much as possible. We do not aim at playing every single poorly encoded file.
07:40
If a file is encoded and does not follow the spec, well, we may not play it. And we're happy with that. So for the licensing, we tried a very aggressive approach. The core is under the MIT license. That's as free as can be. You can use it even in proprietary applications.
08:02
Some modules are under GPL or LGPL license. So how do we compare against existing frameworks, older frameworks? At first glance, we are lower levels than most of them, because the application can really decide how to
08:21
design the pipeline. The application decides where to put threads, where to put buffering cues, and so on. We have a unified API for all kinds of modules. We don't have an API for sources, an API for syncs, an API for filters. We have the same API for everybody, and that's thanks
08:41
to a component that's called U-Pump that allows to, in fact, it acts as a wrapper around a library like libev, which is the one we use currently, but it could be adapted to glib or any other kind of event loop library as well. And it may sound funny at first to hear that there's
09:01
nothing in U-Pump to spawn threads or to handle multi-threading. But we manage the superscalar use case by having a lockless or weightless data structures. All our structures are lockless. We have lockless refills for buffer pools. We have lockless refills for cues.
09:22
So the framework is optimized for superscalar architecture. So from the start. Also, we make a big use of atomic intrinsic that allows us to do atomic reference counting. Almost all our structures are reference counted, so you don't have to care about when to release a structure.
09:42
It's done automatically when the ref count goes down to zero, the structure will be deallocated and de-initialized. That also allows us to implement copy and write buffer semantics. So we can have the same buffer that is in different stages of the pipelines, different branches, for instance, of the pipelines, without having to copy the data.
10:01
The data will only be copied if it must be modified. So that is something we have had from the beginning as well. We also have zero copy semantics. That is, you can build buffers from smaller pieces of buffer, saying this picture is made of 100 TS packets. And the API allows to build this buffer out of 100 TS
10:24
packets without copying anything until the end, until you need to have it in a contiguous buffer. So I talked about attributes. What we call attributes is a triplet name, type, and value, so the name can be totally arbitrary.
10:42
We have standard attribute types like dates, time stamps, presentation time stamps, decoding time stamps, durations. But you can define your own attributes and attach them to buffer at any time, and it will be propagated throughout the pipeline. So that's an interesting feature also.
11:01
And lastly, a very interesting feature is that you can dynamically build the pipeline when you catch events from modules. For instance, you can catch an event that says, hey, from now on, I have a new elementary stream. I have a subtitle stream, for instance. And the application can catch that event and allocate a part of the pipeline that will deal with it, or even modify
11:24
another part of the pipeline to do something else, because there is something that happens. As long as you're in the same thread, you can modify the pipeline any time you want, so it's quite dynamic. So that was on the upside of the project. On the downside, you must understand that this is a very
11:41
young project, so that means that we have fewer modules than existing frameworks, by far. In particular, we have no support for hardware decoding or hardware encoding, and we also have fewer users, so fewer people to test. I have to be completely honest with that. So about the status, we've just released what we call
12:04
UPI preview release number two. We're quite happy with the API, and we have no plans to change at least the module level and the application level API for the moment. We may want to improve the API by adding new calls that will make application developers' lives easier.
12:24
But it's quite a good starting point if you want to have a look at it and start developing for it. There are already many modules that are available. From external libraries, we have support for fmpeg-slash-libav. We have support for x264.
12:42
And we have a few native modules, in particular, TS-Dmux and TS-Mux. This is our core business. And everything that you need to read a file or to read streams from UDP or HTTP, and so on. And we have utility modules, such as DUP, which allows you to put a branch in your pipelines, a trick play or
13:03
digitering of data, and queues, of course. I would like to end the talk with a few examples. So this one is an example of the GLX play example. It has three threads.
13:20
The main thread that is running the GLX module. And it has one source thread that reads data from the network and from the file. And one thread, decoding thread, that takes data from the source and decodes it to output it finally.
13:41
So you can see the use of queues here. One queue after the source thread and one queue after the AV codec thread. You can design it the way you want. We have no problems with frameworks who require you to do all your outputs in one thread. For instance, the PPAPI of Chrome requires you to have
14:03
all your outputs in one thread. While it's not a problem, you can have queues that will bridge all buffers to one single thread and output them from this thread. You can really build the pipeline as you want. Finally, this is an example that is a bit more complicated. That's a real application that runs in our products, but we
14:21
call it URecordHash. It's a program that does four things at once. It receives a UDP stream that's on top of it. And first, it writes the TS buffers to the disk. So that's a data sink on the left. At the same time, it writes the reception time of each
14:42
buffer to an index file, the OX sink on the right. This file allows you to replay the stream exactly as we received it for time shifting purposes or debugging purposes or whatever. And the big blob in the middle instantiates TSGmux and the decoder that allows us to do two things.
15:04
First, we have a JPEG sink, so we write thumbnails of the stream on the fly. And also, we have a hash sink which writes fingerprints that allow us to do quickly frame detection on the stream.
15:23
So all of this runs in three threads that are not symbolized here in the schematic. But it's something that would have been very difficult to do in many other frameworks. So to summarize, we would ask you to keep in touch with
15:42
us, where you have more information on our website. We're also 24-7 on IRC on Freenode. We have a mailing list. And I have booked tomorrow a bathroom at 2 o'clock. It's a meet-up, so if you're interested, you can ask your questions here. I'm not sure we will have many time for questions.
16:01
But if you have questions, you can drop by tomorrow at 2 o'clock, and we'll be around, and we'll be able to answer your questions tomorrow. Yes, we have the time right now for one or two questions. OK. There is one person. You go in front.
16:23
So this might be actually a quite vague question, and you might want to go for more specific ones. Though you mentioned unit tests for, well, a multimedia framework. And I just would like to know more about what approach did you take for this? Basically, every time we write a new module, by module
16:41
I have PIPE, every time we write a PIPE, we have a unit test associated with that, that will send control commands to it. We try to fit data into it, fix data or readily generated data. And we check if the output is the same as what is supposed
17:00
to be defined. And if it changes, it's considered to be an error. Or if there is a memory leak detected by Valgrind, it's also supposed to be an error. So that's for PIPEs. Now we also have a lot of source files that implement well, buffer management or UPump. And this, for UPump, for instance, it's tested by having a unit test that forks several threads and tries to
17:26
send packets to different queues that will be read by different threads. And we check that we receive all buffers in the correct order, and that we do not leak anything, and so on. So the interest of the framework is that every module
17:47
is very autonomous. So you don't have to import thousands of lines of support code to run them. Just a few headers are needed to be able to test the modules you're writing.
18:02
Thank you very much. You're welcome.
Recommendations
Series of 14 media