Can you enable your microphone now? Thank you. Better. Okay.

So I'm with Diamonds. I work for Atlas at the graphics team. And I'll be working on a project called Pipewire. What is it? It's basically to share multimedia between processes.

So that's a big concept, I guess. But what is the point? So if you look at the current multimedia stack, we have kernel, drivers, and APIs, which may or may not come with libraries or not, like also for Bluetooth it comes with a

And then you have your applications. And in between there is, well, there are several things like, for example, Wayland is composed to manage different applications having access to the graphics.

There is also something for audio, also audio. It's only one app that can capture video and that's it.

So what is the idea? Well, the idea is to put the video there between the devices.

That allows you to have a bridge between the devices and the apps actually. It also allows you to share the same device between local apps. And also as a result of how it builds, you can actually share multimedia between apps.

So app one can exchange data. So it's not only between devices. You have the input for video for video. So sending out video also, also between apps. So these things, Jack and Wayland, they stay there.

There's also a session manager, which is important because all of this plugging and all of the things happen in the graph. It needs to be matched by something that's an external process.

Because traditionally in both audio, for example, everything was managed inside the view and it's very hard to configure these things. So we'd like to have that outside. So yeah, I'm listing all the features. But it's like, it uses a lot of concepts from PC.

It uses a lot more things like shared memory with memory. We are made of event of these two single apps and data clean apps. So one interesting thing there is security per application.

So in the world of containers, like the traditional UNIX security rule where a device and user have permissions. The user has permissions on the device. It doesn't work so well when you have multiple containers all running as that user.

They all need to have different kinds of security. So you need to manage that in a different way. So it is built from the ground up to be a real-time data. It's a low latency. For video, not very important yet.

We'd also like to do audio. So the session manager is responsible for setting up the initial graphs of all the devices. For example, you might want to have some ESP posts going on.

Mixing with all the students or applying equalizers and things like that. All this plumbing in a graph.

So I don't have a lot of time, but I'm going to show you a couple of things that are currently working. So if you install Fedora 29 or something, I think it's implemented. Then you have things that work.

So first of all, video for UNIX capture and sharing. It is not only capture and sharing between applications. But underlyingly, it's also capable of doing port and check. So that means that if you have a sandbox application that wants to access the camera, you can go to the portal and use a firewall to open up.

This application wants to access the camera. Will you allow yes or no? If you say yes, then you can actually go to the camera. Otherwise, it's permission denied. But also, you can have multiple clients basically getting the stream. So it works a bit like G scheme.

You have a source, video for being a source, that has ports available. And they are linked to other ports. And basically, data is flowing from one port to another. But what is important here is those red boxes are basically ports where data goes out of the view into the app.

That's just the likely mode. So if you start cheese, for example, you can use auto-video source in G scheme,

which will automatically switch to a barbed wire source, which will go into video for UNIX. You don't really see a difference. So it uses system D software activation there to automatically start barbed wire in UNIX.

There's also a device monitor, so you can monitor all of the devices from G streamer API that are available in barbed wire. So the thing is, you can restart cheese, and then you can also start some other G streamer barbed wire

and actually get the same picture before the processing has been done on Gs. So this is probably going to be a bit nicer if you're having cheese open and you try to do a video chat,

and then it's like, ah, this camera doesn't work, what's going on? Ah, yes, my cheese is open. And then you have to close that one, and so on. So now it's going to be a bit more. So another thing that's currently working for 29 is screen sharing.

So with Wayland, you cannot really make an app that grabs the content of the whole screen. That's the point of Wayland. There's a big security rule. So there is another way now to get to the bottom of the screen, the screen,

that in a much more controlled way through the portal. So basically, if you want to have, for example, a screen recording app, or like a remote desktop or something, you have to ask the portal for a pinewire session where you can get that screen. And the portal does all the checks and stuff.

And it basically sets up the screen in coordination with the router to the barbed wire. Basically, the proposal then sends the screen to the barbed wire where the other apps then connect to it and get the screen. But all of this setup is owned by Trusted Components,

so you can make that secure. So the latest bits on this, for example, is you have the cursor information, a set of network data as well, and a format of together with the bitmaps and stuff. So there's fun stuff.

It looks like this. So Firefox and Chrome 2.0, I think, are being patched to use Firewire so that you can do video conference or do a capture of your desktop inside the WebRTC session.

So what I've been working on next was audio support. So this video supports me for the work that I do now, but then I thought, okay, maybe I can try to do audio as well, but not to de-wrap it all.

But everything had to be moved and re-written, but it takes a while. But I'm trying to do that now. So I've been looking at Jack and Paul, to see how things could work in a different way.

So I've come up with the system that I choose to work. It's basically all the session manager that sets this up. So it sets up a session where we are floating points and the monotone streams are like a monotone format.

And basically there is what's called a DSP node that is created for all of the audio sources and things. So basically the score of the version and the conversion to floating point and speed of the channels.

So basically the session manager that sees those hardware devices will also be moving the device in the sense of this node. And then basically if you have a jack-up, there is a new collaboration there, and it's also recommended to do this, but you basically implement directly your jack-ups

in those safety boards. But for like the regular audio streams from the desktop, there is what is called an audio stream, which basically is an asynchronous stream because you don't want to have those APIs that work synchronously

and you find them for various reasons. And it also does all resample and channels and so on. So basically what this is, it's like all the APIs, like for example, and also all the APIs,

they create what the stream nodes are there, and basically they take the format as it comes, and then transform it into this community of flows, floating point, floating point, floating point, and then send it to the rest of the platform. So there is a bunch of stuff here

on this stream, you can change the volume of this, so yeah. So a lot of experimentation that I did. So this seems to work pretty well. So what have I done?

For post-audio applications, I have the libpost.so.replacing library that basically maps all the post-audio API to equivalent Wi-Fi API. So basically all the post-audio apps, like the volume controller and all of these things,

they just work a little bit different, but that can be tuned. And then there's also the alt-lives, that's easily derived for you. So basically you can run existing post-audio apps on top of Wi-Fi, and also apps as well,

when this is ready to be packaged up. And so for jack support, I'm not exactly sure what to do with that. The third thing that I have, I also have a new jack in place that basically maps all the jack API to post-audio apps,

post-audio API, the pipeline API, and then you just run your jack apps, and you can do really nice things to see all these things work together. So another idea is probably maybe better, because there are things that come to like jack,

to have jack when the jack starts up, that we stop our own source and things, and actually pump into the jack's name. It's very important. But yes, some screenshots. I'm not exactly going to do a real demo.

I can show that in real life as well. So you can run like Para. There's no jack running here in the system. So this is using the jack API with basic library to directly go to post-audio. So you see all the same things there,

the maps come to one, and you can start like VLC or like post-audio. There's actually an also API, a post-audio API, and there are things you can also use in the post-audio API. And you can see from the jack app how it's post-audio apps integrated.

So you can, of course, like use Para to change those things, all these filters and do all this cool stuff. We have all these jack scripts that will automatically interfere with these things. We don't know exactly what to do with all of that.

So you can have some effects and things in there. So what is this state of story? So apart from what is in Fedora

and what is working for the video stuff, other things, the latest changes, they work in the branch. I haven't merged it yet, because it needs to mature a bit more, I guess. So I'll be working on that audio part,

the Permission API. I actually spent the last month doing unit tests and vanishboards and API keynotes. It's MIT crisis. It used to be LPTL, but now I see it.

And I worked on the last piece for the post-audio API, so there is a lot of work still to be done, especially if... Well, it would be nice if this...

Because, for example, if you compare this to post-audio, it uses somewhat less CPU than post-audio, especially once you start going to lower latencies. Post-audio has a tendency to start using 20% CPU, like if you do a SHAP or something, and you do WebRTC, it spikes the CPU,

whereas otherwise it stays at 5%. It would be good if we could move away to that, I think, because of the policy and the security that people have, but it's a low policy.

Yeah, so... Yeah, there is a whole to do here of all the items that we don't. We actually have the same feature as post-audio. But other than that,

we even talked about video effects and so on. So what I'm going to do is

I'm just going to continue doing all the audio. It's adding them one by one, so there's a little bit of support from YouTube that needs to be enhanced with the same feature running and so on. So part of the idea that we have as well

was to move that to a separate legacy and support the user or something to do all the old post-audio stuff. But for now, it's wild out here. So you can find more stuff onto the wiki page here

if you want to read more, or on the website. And, yeah, that's all I have if there are questions. Security-wise, who is going to be maintaining the essential money?

So is that an external process, or are you going to provide an API to connect to someone who writes something? Yeah, so the question is the session manager, who's going to maintain that? So the session manager as an external process was a question actually from good old people,

because they wanted more control over how these things work and how the volumes work and how things connect and what happens when you add the removing devices. So good old people are going to maintain some sort of good old session manager. So the question is, essentially,

so when you have a new contract, and the company, as you said, the actual company is exclusive. But if you're using something that might work, is someone to just run something in my laptop?

No, I'm sorry. The thing is that the idea for the future

is that you don't be able to run random apps like without any protection at all. You run them in a container, and in that container, they simply won't have access to anything required except to import the API.

And there, you check it and it asks you if access is granted to the devices as well. So it becomes more like what is done on the... So the possibilities of the session manager? Yes, so the session manager has complete control over you. You can see what, and can take away from issues or grab them.

So the session manager is very important in that, and you sort of trust it as well as you write it. We're quite sure how to make that. You've got so many hands. Is there one key running in the system? Or is this just for single users? I mean, with many things today, it's always a problem

if you want to run actual, multiple users on the same machine. Is this a global deal that's running here, or what's the plan here? It would be a per-user deal. A global deal is too complex to do with all the infrastructure that we have.

For you to have rules to allocate devices can be seen by using stuff. If you do that as a system deal, you have to be implement all that logic yourself, depending on what user connects to the unit. So right now, you know what to do.

So in certain cases, probably, it would make sense if you only have one user. Would it be the task of the session manager then to give up control of hardware if you switch to another user, so that other user can then use the camera? Well, the session manager would also be per user.

Otherwise, that would work. So if she's open on one user account, and then you switch to another, then... I think that would... Well, I don't know. Do you have to define the policy for that? I think in any case, yeah.

There's going to be two pipeline deals, and one is not going to be able to get to the hardware anymore, so I guess the one that is floating is. So since it's very similar to Jack, do you think it would replace Jack in the future? If it does, do you plan to support other platforms and things like Jack does?

So the question is, can it replace Jack? I don't know. It's difficult to do, but I have to see.

Will I support other platforms and things? Well, not me. Not me. It touches on it. But it uses a lot of unique specific things, so I don't know. Hi, I know the integration with Jack will be quite complicated,

but regarding the session management, there are some concepts that basically exist within the Jack ecosystem, such as non-session management. Have you looked into that? And also, have you looked into further back-ends, such as Vado, because there are still a lot of people

using Firewise devices with Jack? Yeah, so for Jack, the session management definition of Jack is quite different from this one, because for a session manager in Jack, it means like restoring a whole bunch of apps into a certain state, and run into a certain state

so that you can start your project. While this is more like a dynamic reconfiguration of your app, for a session manager itself, I don't plan to implement anything. There are many other session managers available, so the session manager can restore your settings,

so I can implement that API and use those tools. So I don't think I want to include this in the Firewire itself. And then the only question is support for more kinds of devices, like Firewire and cards and stuff. So yeah. So Firewire itself has a plug-in system,

so you can write plug-ins for devices quite easily. So it would be a matter of writing that. This is also used not just for webcams and things,

but for hardware-accelerated decoding and videos on some embedded platforms as if in a web-advanced environment. So yeah, so that's the question. Do you plan to have encoders and decoders running on the Firewire graph? I don't know. I don't think so, to be honest.

I don't know. Well, theoretically, it would be impossible to have a whole bunch of filters behind the capture nodes and the power balances. But you could just as well move that part to the consumer. So I don't know exactly where it should be.

The same we can go with the decoders. Can we expect a worse latency than Jack? Or is it going to be the same? So, frankly, it's worse because what Jack does is he configures the hardware in a very small buffer in two periods.

And if you do that, then also driving will also do different kinds of configuration of the USB packets that it sends out and stuff, which reduces the latency. But since Firewire has dynamic latency, it needs to configure a big buffer and no periods. And then the driver says, well, I'm going to delay it when it needs it.

So I can't do it in the same way. So I have to fall back to the Jack way of doing that. But then again, you have lots of interrupts per second. And that is not good for Firewire. Yeah, well, I mean, this is, I really hate the fact that I cannot use, you know,

Arduino and then like start a video somewhere. So sometimes, yeah, if it was the only application that was using, so let's say Arduino is the only one that is generating audio, then we could probably reconfigure the hardware for this. And then go to another mode when we stop being in real time.

Well, thanks a lot for working on this. Well, we have time for more questions, so thank you.