Okay, we start again, please. If you can take another time. Thank you. We start again. I remind you that you can ask questions online using onlinequestions.org code 2147, the name of our room.

Our next talk is about GPAC, GPAC 1.0 overview. Please welcome Jean. Okay, thank you. So, I'm Jean. I'm an associate professor at Telecom Paris. That's actually our name since one year.

And I'm also the main developer of GPAC, and I'm here to talk about what happened in GPAC in a very long time. So, what's GPAC? Well, okay, sorry about that. GPAC is old, like me, actually. It's been a long time that we haven't done a lot of work on the core platform.

We are basically a C multimedia framework. We run on, let's say, most platforms. So, Linux, Windows, OS X, iOS, and Android. It's mainly our labs research platform. So, that's why we put a lot of funky features in it. We are known mostly for MP4 box, which is a packager, dasher, and it was originally a beefs encoder.

I don't know if you guys ever heard about beefs. And we also have a client that we use for a lot of our research projects, which mainly features a quite large piece of code, which we call the interactive renderer doing audio, video, text, and synthetic graphics.

So, what is exactly the problem that we have now? So, a bit of history. First, we started almost 20 years ago in 2001. We did the first release of GPAC in 2003. And our catchphrase at that time was that we were the only open source MP4 systems player.

And that was actually the core topic of the project. So, we worked mostly on MP4 systems and on the player. We went LGPL in 2005, time at which I joined Telecom Paris. And since then, we've been involved in a bunch of European and French funded projects on a huge amount of topics.

So, we had broadcast, we had OTT, we had encryption, we had VR, we had whatever you want listing them. So, we ended up with a bunch of tools that were quite nice, quite powerful. And we were really happy with that, except that after, well, basically 15 years, that's how it looks like.

So, we basically ended up in an architectural nightmare because we didn't have time to stop developing because we had so many projects. And so, this is a high-level view of what you have on master right now in GPAC. You end up having applications that are mostly front-ends to our core library, which we call the libGPAC. And in that core library, we have one slot of mostly authoring related functions, such as dashes, servers for RTP, TS-Muxers,

a lot of code for media importing. And on the other side, all the player architecture, which has the 2D, 3D renderer, text rasterizer, dmuxers,

and then a big API that is dedicated A for decoders and B for dmuxers and network plugins. And those actually were really bad because we had originally a lot of code here that we started porting back into MP4 box, but we couldn't get a unified framework, so we had a lot of code duplication.

So, basically after 15 years, it was time to tear walls down, right? We had no choice. So, the first step was getting rid of all code duplications, obviously. We had a huge amount of options in MP4 box, of configuration options in the player.

None of them were unified, so that was a mess for the user. So, we decided to find a way to, let's say, homogenize this. And we didn't want to modify your scripts, so that means we don't want to touch the command line. The syntax should stay the same. We don't want to change the APIs as much as possible.

Obviously, you're not making an omelet without breaking a few eggs, right? But mostly for the players, because changing the API of the player would mean redoing all the ports we've been doing on iOS, on Android, and so on. That would be a killer, so we tried to avoid that. And most importantly, we wanted to have exact binary results for the packagers.

We didn't want to break anything, so what you had before would be produced in the same way with the new version. That was kind of a challenge, but that was made possible because our test architecture is based on hashing of files, so that helped us a lot, basically. And that's not the only thing, obviously. What we want to do is make all those parts of the library communicate with each other,

be able to have authoring tools communicate with player tools, and don't have a distinction between the two. Which means we want to have user-defined media pipelines, where you can basically say, I want to connect this decoder with this particular mixer, or things like that. And with that, we thought, and we hoped, that we could be able to finally plug in all the tools that we couldn't add,

just because of this architectural problem. So, it took us two years before our first public preview, almost three, because the first commit on that new architecture was in February 2017, actually. We are now 3,000 commits ahead of master. That's a lot.

And the public preview has been available since June. It's called 090-DEV, and it's actually the filters branch. So, quick overview, without getting into too many details. My students usually do slides like that, and I shout at them because there's too much text.

But that's okay. So, we are basically inserting a new pump inside JPEG, based on the filter architecture. We have four main components here. The first one that we call the filter session, which is something in charge of scheduling everything. So, it's a task-based, real-time and unreal-time handling buffer management, memory collections.

We can work in threaded and non-threaded modes with log-free queues, these kind of things. We added a dynamic link resolution inside the session, so that you don't need to exactly know how to queue your filters. The framework will take that for you, will do that for you.

We do have event management also for communication in between filters. The filters themselves, that was the tricky part to do. We designed the entire set of filters on a single API. So, we no longer differentiate mixers, inputs, outputs, decoders.

Everything is one single API, which simplifies a lot the testing, the integration, the embedding through scripting and things like that. Filters describe, obviously, their own IO capabilities, so that you can resolve dynamically a graph, right? They're inherently single-threaded. You could, obviously, have your own threads inside, but whatever gets into the filter is always coming from a single thread.

We wanted to have dynamic stream reconfiguration. What do I mean by that? Well, you're familiar in HTTP streaming with switching resolutions, but you also might want to switch codecs. So, switch from HEVC to AVC or things like that.

You don't want to have to relink completely the pipeline, or at least you don't want the user to know about that. And we also wanted to have the possibility to move from compressed to uncompressed domain in the same way. So, that was quite tough to do, but that was fun. We define our own documentation. So, if you're familiar with FFmpeg, this is very similar on that part.

Filters communicate through links that we call PIDs. They basically handle the packets. I'm not going into too many details. The only interesting thing here is that every properties of a filter is dynamic. So, we no longer have a class where you list everything that you can do. You just have a dynamic list, and you just push properties, and then you define how the behaviors of those PIDs apply, depending on the properties.

And those properties can obviously be user assignable. So, if you want to add your own property, you can, without touching anything in the code. And packets, so that was a big problem we had in the architecture. We had two specialized packets. Now, data packets are completely non-specialized. They can be files, they can be part of a frame, they can be pretty much whatever you want.

And the associated data that you get is either data that you allocate from the session, so it's managed, data that you share from your filter, so it's something that you need to care about when you release and discard pictures. You reference count everything, obviously. And we also have this concept of interface objects.

So, those of you familiar with media pipelines and video decoding know that sometimes you just get textures. That's the only thing you can get from your decoder. So, that's basically the goal of these objects. And again, properties that you can tag, such as all your timing. And dynamic properties, such as command encryption information, for example, things like that.

User expandable as well. That's it for the tough part. So, the high-level impact on GPAC is, A, we have a new app that we call GPAC. It's a generic pipeline assembler in command line. So, most of our tools are command line. And it allows you to build whatever you want as a pipeline.

As far as MP4 box, we moved all importers, exporters, encryptors, and so on into filters so they can be reused. And we removed all the streaming codes so that it's now dealt with in the GPAC app. For the player side, same thing. All demuxers, all decoder plugins move to filters. And we basically now have the player as running a filter session with the composition engine, which has been extracted as a filter.

All export, AVI, and image has been removed as well. It was not well-tested enough, and we now have the same functionality in the GPAC app. And so that's a big warning for those of you who might have used them. We completely removed MP4, 2TS, and DashCast.

DashCast was more of a student project, and it never got really maintained, so we completely dropped it. And we now have all the support we want inside GPAC to do exactly the same and much more. So, time to look at what's inside. In terms of IO, before we were mostly file-based, I have to say. Now we have pipe support, so we have Linux, OS X, and Windows in and output pipes.

We have sockets, UDP, TCP, and UNIX domain. Obviously, files. We had HTTP input. We now have HTTP output. We started to plug in a server inside, so we also have an HTTP server. The goal here is to play a bit with push and low latency in HTTP streaming scenarios.

We had RTP input and output, and we added an RTP server because we had the code, but we didn't have the API for you guys to use it. So that's part of the history here. We have support for raw data. It's been a long time we had that as a requirement. So most while UV, RGB, PCM formats out there.

Like I said, we extracted that compositor engine so that you can now, in your media pipeline, produce an SVG and get your RGB or while UV or whatever results you want, format you want from that file. So that's actually quite fun to do, and we've been testing it a lot, and users have already tried to have fun with that.

What's more? So we had FFmpeg support since 2005, I think, for the demuxers and the decoders. We like it a lot, so we completed that support. We added mux format. We added encoders. So for the first time now, GPAC finally can encode stuff in a nice way, actually.

We added support for LibAV device so that we have grabbers as well. And we also, that's the most recent work we did on the filters, add support for LibAV filter. So you can plug in also a LibAV filter inside GPAC. And what was really nice to do is trying to understand how the FFmpeg option systems were working

to remap it directly into our filter so that we can build filters that exactly expose the options that you have in an AV class in FFmpeg. We wanted to have distributed processing for a while. So you want to have two instances running on either the same machine or distributed machines. But we have a problem here that is inherent to the kind of mux you're using.

So for example, going from MP4 to TS, you might lose some metadata that you get in MP4, and you don't know how to write them on MP2 TS. Since all the metadata is already exposed in GPAC, we figured out the simpler way to do that was to define our own multiplexing format, which we call the GSF, it's the serialization format,

that you can write to file, write to pipe, write to UDP circuits, TCP, even HTTP. And you can even use AES, CBC on that. So if you want to protect the stuff while you exchange it, you can have encryption here. What's not completely done here is bidirectional communication.

So in case we have events requiring to send, sorry, filters requiring to send events down the pipe, we still don't have that communication in between processes. We're working on that. More features. I told you we like FFmpeg. Actually, I love Fabrice Belard, the creator of FFmpeg. He released some months ago this quick.js engine, which is an ECMAScript 2020,

almost complete support of 2020 ECMAScript. So we dropped our old spider monkey, which was 15 years old, basically. We write complete bindings of all this new API into JavaScript.

We added all the bindings that we were missing on our core rendering functionality. So we had all this 2D software rasterizer that you could only use through BIF or SVG. We just added JavaScript wrappers around that. And now you can draw in JavaScript without knowing anything about scene graphs in general. We plugged, we added XHR support.

We plugged that back into this API for filters. And we also added WebGL 1.0 core. So that means you can instantiate the WebGL context inside your JavaScript and play with that. So we have more to come. These are two examples. So the first one is basically using our logo as a channel for while UV texturing in WebGL.

And the second one is a zero copy editing in 2D. So we had a lot of tricks on that. And we have some documentation. I'll come to that. Don't worry. More features because we may have a few more. Encryption is on the fly now. It's a filter. So you can plug that live into your chain.

We have support for segment-based encryption. Before, we only had complete files. Now you can do that with segments. We have more tools on HEIF. We have HEVC tie splicing and merging. So that's basically bitstream processing. HEVC is now in place. We have media inspection. So it's a basic stuff. But the nice thing is you can move it around in your filter.

So you can put it anywhere in your filter chain to try to debug what's going on. We worked on HLS support. And we also have now dual dash and HLS output. And other features, ISO-BMF customization, because we couldn't do that before. And all the features we had in GPAC 0.8.

So more features. Well, I'm not sure if I will have all the time I need to talk about that. But the nice stuff is we've finally been able to add a splitter inside, an MPEG-2 TS splitter inside the chain. We didn't have that before. You no longer need a huge compositor to play audio and video. Oh, that's crazy. Yes, you have like basic audio-video support, so that's good.

We have a source concatenator, so building a video out of JPEGs, for example. Or concatenating different videos and having everything output as a single video stream. Things like that. Not going into all those details, but you can have some more details online. So these are a few examples.

I'm not going to detail all of them. It's just to give you an idea of the syntax. We tried to keep it quite compact. We inherited from Dash all the templating mechanisms for URL. So that means you can dynamically create your file names based on the properties you want from the input strips.

These are all from our cookbooks. I'll come to that in a few seconds. But you can play with it. And some of the syntax has been extended also for MP4Box. So we can also use some features of this new infrastructure to MP4Box and extend the existing set of functionalities. That brings me to tests and coverage.

I don't think we need to comment on that. This is actually pretty bad. So master and filters. Don't get me wrong. We had tools to do all the testing on masters. The problem is testing the player was basically equivalent to loading an OpenGL context. And so on our VMs it was killing everything. So it took hours and hours to try to build and test.

So we basically couldn't do it. But now since everything is broken into very independent pieces of code, you just have a few tests for OpenGL. And all the rest is completely independent so you can finally have decent coverage. So we have now all our bit system in place. You can check it out. We have a new test suit report.

We've been splitting it out so that we can in the near future handle pull requests and integrate files into our test suits if you guys want to contribute on that. And the last point I told you about is the documentation. People were always complaining like, what? This option is here. It's not on the website and so on.

What we did is basically have completely automated generation of the documentation. So what you get on wiki is what you get in your main pages, is what you get in your command line .h help. We started a set of how-tos. We're migrating all the posts and so on we had on our former websites into our wiki. It's a work in progress obviously.

But we're trying to put more things here, extract them from forums as well. And we worked on improving a bit our dev documentation for LeapJPAC so it's also available over there. So, what's next? Well, after ski vacation obviously, final merge and release. So that would be March or April, something like that.

In terms of ongoing work, so we have right now started an automated fuzzing of GPAC. So we have a bunch of fuzzers running. It's not public yet, but it should be public in the next two weeks. And in our future roadmap, we want to have more hardware codec support. We don't have encoder hardware, so we want to look at that a bit.

And we want to extend also decoding on other platforms. We want to add more remote session monitoring. So we do have an integration of remotory.js. I don't know if you guys know that, but it basically allows you to monitor from a web socket on a navigator what's going on. So we want to add more GPAC specificities in the visualization.

And we want to work a bit more on higher level APIs. So we haven't worked on how you would use a feature inside your app directly without going into too much details of the code. So that's basically what we're planning on working right now. And I think that's about it for me.

So, if you have any questions? There is one online. You can read it here if you want. In what does GPAC distinguish itself from VLC? Maintaining a player is hard. Another possibility to reuse code between the two players. So that's how it distinguishes itself.

It's very hard to say it's a player. I would say that the main difference is for us, our focus is really on all these 2D and 3D interactivity things. It's really something that we've been working a lot on. All this composition and scene description, which makes GPAC quite special for that. Maintaining a player is hard.

Obviously, this is an euphemism. It's terrible. Maintaining a player is something completely crazy. You need to be aware on all the possible platforms. VLC is more a product than we are in terms of player. We don't claim to be a product. We claim to be a very good experimental platform. Are there possibilities to reuse code between the two players?

I would hope so. I think the code in both is pretty well isolated in terms of the plugin system. So it's more a problem of having time to work on that. We're planning to look at that. We're planning to look at the Jstreamer as well plugins. We've been integrating FFmpeg, so there could be also ways of integrating other filters and other modular systems like that.