PipeWire in the Automotive Industry
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47515 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020288 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
MultimediaSoftware engineeringMultimediaComputer animation
00:32
Open setSource codeComputing platformSoftwareInfotainmentElectronic visual displayTelematikFunctional (mathematics)System programmingDevice driverGradientTask (computing)Physical systemProjective planeTask (computing)BuildingGeometryVideo gameDemo (music)Computer animation
01:07
Physical systemComputer hardwareDevice driverDifferent (Kate Ryan album)Pulse (signal processing)Numbering schemeComputer animation
01:36
Open setComputer hardwareGradientComputer networkBefehlsprozessorPattern recognitionArray data structureDigital signal processorMixed realityStreaming mediaInformation securityBootingSoftwareContext awarenessImplementationMusical ensembleDevice driverMetreMultimediaMobile appStreaming mediaInformation securityComputer hardwareGame controllerBefehlsprozessorData managementDifferent (Kate Ryan album)Numbering schemeMultiplication signFilter <Stochastik>Sound cardPulse (signal processing)MultiplicationPhysical systemInfotainmentArray data structureUltraviolet photoelectron spectroscopyHypermediaPlastikkarteType theoryDisk read-and-write headKnowledge baseMereologyBitOrder (biology)Video game consoleComputer animation
06:34
Data managementPulse (signal processing)Decision theoryInformation securityOpen setDemonInterface (computing)Euclidean vectorLoop (music)Projective planeDecision theoryCASE <Informatik>Information securityMultimediaData managementPulse (signal processing)Streaming mediaModule (mathematics)Mobile appComplex (psychology)Cartesian coordinate systemGame controllerComputer animation
08:28
Cartesian coordinate systemCASE <Informatik>Computer architectureRoutingHypermediaBitComputer animation
08:59
DemonArchitectureOpen setExecution unitLink (knot theory)Pulse (signal processing)VideoconferencingModul <Datentyp>Information securityServer (computing)DemonCartesian coordinate systemPlug-in (computing)Graph (mathematics)Server (computing)Process (computing)HypermediaGame controllerPulse (signal processing)VideoconferencingCodeLink (knot theory)Data managementSoftwarePresentation of a groupConnected spaceObject (grammar)Different (Kate Ryan album)Arithmetic meanInformation securityQuicksortWordComputer animation
12:40
ImplementationDemonLibrary (computing)Data managementModul <Datentyp>UsabilityObject (grammar)Keyboard shortcutOpen setPoint (geometry)HypermediaVertex (graph theory)Graph (mathematics)AbstractionOperations researchGUI widgetPulse (signal processing)Representation (politics)Computer hardwareBridging (networking)Function (mathematics)Digital signal processorComputer networkProcess (computing)outputLatent heatFeedbackMenu (computing)Streaming mediaComa BerenicesPhysical systemData managementLibrary (computing)Distribution (mathematics)Computer hardwareSynchronizationHypermediaModule (mathematics)Parallel portCartesian coordinate systemLink (knot theory)Streaming mediaCodeRoutingSoftwareDemonUsabilityBefehlsprozessorGraph (mathematics)Process (computing)Speech synthesisFormal languageImplementationWritingObject (grammar)AbstractionRepresentation (politics)Personal digital assistantPoint (geometry)Arithmetic progressionMarginal distributionCASE <Informatik>AuthorizationLatent heatTwin primeField (computer science)Power (physics)Decision theorySource codeComplex (psychology)ArmTheory of relativityComputer animation
18:10
Personal digital assistantOpen setSpeech synthesisVertex (graph theory)Graph (mathematics)Demo (music)BuildingInformation securityData managementObject (grammar)Run time (program lifecycle phase)VideoconferencingImplementationPulse (signal processing)TouchscreenSource codeData managementComputer hardwareHypermediaGame controllerImplementationMultimediaStreaming mediaSoftwareFunction (mathematics)Sound effectConnected spaceMusical ensembleVolume (thermodynamics)Cartesian coordinate systemCASE <Informatik>Link (knot theory)Graph (mathematics)Demo (music)Point (geometry)Information securityPulse (signal processing)Message passingRun time (program lifecycle phase)Module (mathematics)Moment (mathematics)VideoconferencingSource codeProcess (computing)Code refactoringDecision theoryDifferent (Kate Ryan album)Branch (computer science)WikiBuildingScripting languagePlanningBefehlsprozessoroutputNumber1 (number)Drop (liquid)ArmOpen sourceDivisorPhysical systemLibrary (computing)Coefficient of determinationRight anglePerturbation theoryLevel (video gaming)Representation (politics)CodeCasting (performing arts)Multiplication signComputer animation
23:28
Open setTerm (mathematics)StatisticsBenchmarkGoodness of fitFrequencyPairwise comparisonArmMultiplication signPoint (geometry)Run time (program lifecycle phase)Streaming mediaSpacetimeMathematicsData managementBus (computing)Configuration spaceComputer hardwareBootingBefehlsprozessorFunctional (mathematics)Sampling (statistics)Computer animation
25:53
Point cloudOpen sourceFacebook
Transcript: English(auto-generated)
00:06
Hello everyone. Welcome. I'm George. I work at Collabora. I'm a senior software engineer working on multimedia things like Gistremer and Pipewire. And today I'm going to talk
00:22
about what I've been doing in the past one year with Pipewire enabling Pipewire in the automotive industry. So let's start by defining automotive. So basically I've been working on a project called Automotive Grid Linux. I'm pretty sure you've heard of
00:46
it before. If you haven't heard, ask around. There is a booth at the building K as well where you can see a live demo of Automotive Grid Linux. And my task in AGL has been to make an audio system for cars. Now, the immediate question you might have is what's so special
01:08
about the audio system? I mean, we've had audio systems for years on Linux. The driver layer also works. We have pulse audio on top that works really well for the desktop.
01:25
But yeah, what's missing? So to explain that, I need to explain to you how the hardware looks like because that's the difference basically. So on a desktop, we're used to having this kind of scheme where there is a single CPU and a single audio card,
01:47
might be multiple but it doesn't matter. You basically choose one audio card that you use. You have your speakers connected there and you have your microphone connected there and that's the whole thing that exists. And yeah, you have things like
02:04
pulse audio that can manage this really, really, really well. On a car now, things are quite different because you have different nodes inside the car, different CPUs, multiple CPUs. So you have maybe one CPU that is the in-vehicle infotainment
02:24
system and then another CPU around somewhere else which is maybe, I don't know, a hardware radio device or a CD player. There is dedicated DSP somewhere else which is doing the filters
02:45
and echo cancellation and finally amplifies the audio to the speakers. And there might be multiple DSPs actually, one for the front speakers, one for the rear seat speakers. There are some cars that have even speakers on your headrest. There are also microphone
03:05
arrays that are on your seat belt or on the top on the roof, on the doors, on the side, everywhere basically. And this becomes basically a very complex network that you no longer manage
03:24
in the way that you manage the desktop. So some requirements. Basically, yeah, I mostly talked about it already. You have a lot of different hardware. And the other thing that I didn't say
03:41
is that you might have streams that go around without passing through the main CPU. So you may have, for example, dedicated hardware CD player that directly puts audio on the DSP and that audio never passes through the main CPU. Although the main CPU still has control over what place, you don't actually get the audio data routed through the main CPU for performance
04:07
reasons. And you need low latency because you may want to implement echo cancellation and things like that. And then on the software side, on the iFi system, when you want to play
04:20
something, there are some requirements there as well because you might have, you basically have apps that have a different context. There are apps that, for example, apps that play music that are in the multimedia context the same. And there are other apps that
04:40
may be doing something special like, for example, your navigation app, which every once in a while it says please turn left in 100 meters or something like that. And that sound needs to be treated specially because you want to make sure that the driver can hear it.
05:00
You want to play it on the front speakers, you want to play it amplified, lower down the music while it's playing, and things like that. So you have to treat it in a different path than, and in a different way than how you treat the music. We want security because things
05:21
nowadays are very hard. We want to have containerized apps nowadays that can play music or something else, but you don't want them to be able to do something that they
05:41
are not intended to do. Like you don't want the music app to be able to take over the navigation special stream and things like that. And then you have emergency signals when there is something wrong with your car that needs to play some sound to warn the driver. That's also something that needs to be treated specially. And it's usually
06:06
coming from dedicated hardware that has to be certified and it won't actually send data through the main CPU again. It will just play directly on the DSP and there is a path there that is guaranteed to work because it's certified. But then the IVI needs to understand
06:25
that this is going on and stop the other streams at the time and things like that. So the real problem actually to be doing an audio system in a car is routing and policy management.
06:42
How do we manage the routing and how do we make policies? So there have been some projects that have tried to address this. I'm starting with Pulse Audio. Pulse Audio didn't actually
07:02
ever try to address this use case. It's really meant for the desktop. It makes its own policy decisions so you can't actually tinker with it unless you build a module. And some people do that. They build a module that tries to circumvent how Pulse Audio is internally routing
07:20
streams. But it's not very nice. It is fairly resource intensive compared to other multimedia demos. And it doesn't implement any kind of security so whichever app connects has control over everything. And then there have been things like the Geneva Audio Manager
07:40
which is building on top of Pulse Audio. It uses Pulse Audio as a backend. Optionally, it can use other backends as well. It basically builds an API for policy management on top of that. But it has a couple of shortcomings. You must call the Audio Manager API so the
08:00
applications need to be aware that this exists and things like that. And people that have used it, they generally agree that it's complex and not very nice. And then in AGL, we also had another API called 4A which tried to solve some of these issues of Audio Manager but didn't succeed in addressing all of them and people still didn't like it so much.
08:28
So then there was Pipewire. What is Pipewire doing differently to address these issues? So
08:41
when I started looking at that, Pipewire of course didn't support any of these use cases. But it had something that looked like it was possible to develop it to address these use cases. So let's start with explaining a little bit of the architecture of Pipewire. How does it
09:03
look like? So there is a daemon, Pipewire daemon, that basically handles all the media routing between applications and devices. And everything is a different process. So you can have applications sending data through MFD or DMA buff to Pipewire or capturing data.
09:29
And then you can have devices either accessed directly through a plug-in to connect let's say to also or to video for Linux. But you can also present devices through external processes like the
09:47
Bluetooth manager process that we have that does all the networking stuff with Bluetooth and provides the device to the Pipewire daemon. And then there is another process which is called
10:07
Pipewire daemon. It actually looks at the whole applications and devices graph and decides what is going to be routed where and makes all these connections. And I'll talk more about that later.
10:21
That's actually what I've been doing myself. So inside that blue box, the Pipewire daemon, the things look like that. So inside the Pipewire daemon there is actually a media graph with a lot of little objects called nodes that connect to each other with links.
10:46
Every application is represented by an application node. Every device is represented there as well. Halsa, video for Linux, Bluetooth. And it can also load internal plugins
11:01
that also provide nodes that can do filtering and processing. Pipewire actually started from providing a pulse audio for video. It was initially called Pulse Video. So it can handle video data very well. That's its original purpose.
11:25
Nowadays it also implements routing audio data and it's evolving to become a pulse audio replacement. It is modular so you can customize it a lot. It has built-in security, something that Pulse
11:41
Audio didn't have. There is access control per application, per device, and you can make things, you can give access to certain applications to do certain things and not other things. And that's what I mean with security. It's a very clean code and much more efficient
12:03
than Pulse Audio. It's capable of doing low latency audio and it actually implements the jack API so it can also work as a jack audio server. And yeah, the single most important thing is the external policy management. So applications connect to Pipewire,
12:24
they don't actually get linked to anything, they are not able to do anything until the session manager acts and gives them permissions and gives them links to something else, to a device.
12:41
What was missing from Pipewire, when I started last year, there was no session manager. So I started developing a session manager, basically that was all my work. And I developed a session manager called Wire Plumber. That was the first session manager implementation. It's like Pipewire, it's modular and extensible. The target is to provide a reusable session
13:05
manager for all kinds of use cases, automotive, other embedded, also to provide a desktop session manager. And a session management API as well, so it provides a library that would allow you to
13:21
write your own session manager if you want, or to write your own policies to decide what route where, to write modules for Wire Plumber, to extend its features and tools around that. It's written using GObject, it provides a GObjectified API on top of Pipewire for
13:44
ease of use and for enabling bindings in other languages. That would be super useful. And it also provides an API, which is called the Endpoints API, I'm going to explain that,
14:01
which is very useful for implementing policies. I'm going to explain that. Endpoints is a concept that I brought into Pipewire, it didn't exist before. Pipewire itself, it routes media through nodes from an application to a device or vice versa.
14:22
But the thing is that in a car you have all these kinds of streams, hardware streams that maybe are not passing through the main CPU and you want to control them. So I thought, how can we make an abstraction on top of all of these? And this is what the
14:42
endpoints are. Basically, endpoints are also little objects that look like nodes, they can be linked together and unlinked, but they don't actually have media, they don't route media, they are something more abstract. So for example, you could think of a pipeline that looks
15:05
like in the endpoints graph, it looks like there is a media player that gets connected to the car amplifier and these are the two endpoints with one link and there is no more in that graph. While the same thing on the nodes graph, it looks like there is a media player that goes
15:22
into a filter that does some processing and then it goes to a network sync that pushes things out to the car network and then there is a car network distribution daemon or whatever that eventually ends up in the amplifier. And this is a much more complex path that
15:42
the policy management would have to know and it would become very hardware specific to handle this. So this is why I thought we need to abstract this. And these two are meant to run in parallel. So you have both the endpoints graph and the media,
16:03
the nodes graph, and whenever you make a link on the endpoints graph, the session manager translates that into links on the actual nodes graph and on the actual car network or whatever else is there. The endpoints API is also something useful to bring
16:25
the graph closer to what Palsodio has. So Palsodio has only sources and syncs, there is only two things that you can link together. There is no concept of a very much more complex graph. And this is exactly what endpoints are. So if you were to implement endpoints on a desktop,
16:44
you could still have things like an application being represented as one endpoint and your speakers or your microphone as another endpoint and then you can link them together no matter if there are filters in between or whatever else. And this is how it looks in the endpoints graph.
17:04
So you have your applications, your media player, your voice agent, voice assistant, and they make links to the endpoints. There's an amplifier endpoint there, a speech microphone. And on the top of this graph, we have basically streams that are in the main CPU,
17:26
the main CPU actually routes media data. But then we can also represent a hardware player or something else that is only routed in the car network and not in the main
17:40
CPU. But the main CPU can still have this representation in a graph. And so the policy management code can still look at this and decide, okay, here's, for example, a hardware player that decides to, in this example, so for example, there is a voice agent assistant
18:04
there that wants to play something and it starts playing. And then as soon as it makes the link and it starts streaming, maybe the policy management understands that this happens and it enables some effects on the amplifier endpoint, which could mean that more nodes are
18:25
getting linked at the point and they enable some effects or it could mean that something on the hardware is triggered to enable this effect where the voice is louder than the music.
18:40
And then the hardware player, the user goes and clicks a button on the hardware and there's a radio hardware player that starts streaming to the amplifier without passing media through the main CPU. But because we have this graph representation, the policy management understands that it needs to unlink that media player at the top which was previously linked
19:01
and which is in software. So this is how it works. And then we have how do we make the connection between the endpoints and the actual media nodes? So maybe you have, again, two endpoints, an application that plays something and an amplifier,
19:27
something that plays. And you want to implement volume controls per stream. So you have the multimedia stream and the navigation stream, which are, as I said earlier, they are something that needs to be treated differently. So the software DSP output endpoint that I have written there
19:45
is creating a couple of DSP nodes there that implement volume controls per stream. And as soon as you make this virtual link on the endpoints graph, it goes and creates the actual link on the media routing graph.
20:01
And in the case where there is a hardware dedicated DSP that can support these streams in the hardware and you can trigger the different volumes per stream by sending some comments to the hardware, it's still the same externally. So you can make a link on the
20:23
endpoints graph. And then the session manager, which has a module that implements this endpoint, it understands that it needs to go and send a message to the hardware amplifier to enable that different volume effect. So what's the status of Wireplumber right now?
20:42
It currently works nicely for AGL. On the demo that we have on building K, we use Pipewire and Wireplumber at the moment. The API is sitting down. We are doing some more refactoring on the API level, on the library,
21:00
but it's sitting down. And we've just recently started generating documentation using Hotdog. And I'm in the process of documenting right now. I started, like, last week. I pushed some dogs. There are some shortcomings. Like, it doesn't have flexible policy. Although the policy is configurable, it's not configurable
21:20
enough. And it works only for the AGL demo and nothing else right now. And there is no security management implemented. We just need access to all the applications, which is bad. But we have a plan. So the next steps, what I want to do next is experiment with
21:40
scriptable policy, provide something, provide an API that scripts can be written to influence how these decisions are taken about what to link where. And there is a runtime called Anatoly, written by Bastian Nocera, who wrote this basically for this purpose, for
22:02
managing Pipewire policies. So my next step is to experiment with that, see if that's something that makes sense and if we can write nice scripts like that. But other ways of doing that would be also acceptable and welcome. The other thing is improve desktop compatibility and make a
22:21
drop-in replacement for Pulse Audio that currently doesn't work on the desktop. Another next step is to enable management of video, video nodes, enable camera inputs and screencast inputs, and implement security management. There is a design for that, I just haven't gotten around it to implement it. The source code, it's all on GitHub for
22:46
the desktop.org slash Pipewire. You can find both Pipewire and Wireplumber in there. You can make a merge request there, you can find issues there, the wiki is there, everything is there. And on the AGL front, I maintain a branch of Pipewire with
23:04
the AGL-specific commits that I had to make, although I'm constantly pushing them upstream, so as soon as they are merged upstream, I remove them from the branch and I rebase. And then there is this metapipewire Yocto layer. AGL is based on Yocto, so these are the
23:22
recipes to build Pipewire and Wireplumber inside AGL. Thank you very much, any questions?
23:44
The question is if I have any benchmarks comparing Pulsar to Pipewire. Yes, I had the benchmark that's from last year, so it's not up to date, but it was looking like, I'm sorry I don't have the slide with me, but we had some comparison of Pipewire
24:07
basically taking a stream, a 5.1 stream, so six channels, transforming it in a couple of ways, resampling, changing the format, and taking it out. That was using something like 6% CPU with
24:24
64 samples latency, while Pulsadio at the same configuration was using 100% CPU, it was failing. So a good comparison was like that. I'm sorry I don't have the latest statistics.
24:43
It's looking really, really good. In terms of flexibility over here, you said you were looking to do more flexibility, were you talking about boot time flexibility or runtime flexibility? Because with the adoption of things like A to B in the audio space now,
25:02
you're getting nodes that are coming on and off the buses all the time that are making audio harder to manage. Yeah, Pipewire is very dynamic in how it manages hardware, so it can detect things coming up and moving up all the time.
25:22
Does that include firmware? Because you can have A to B nodes with DSPs in them, the firmware, the DSP, which implements different functionality gets changed dynamically as well. I don't see why it wouldn't be supported, but I never looked at this kind of hardware.
25:46
There's no time for more questions. Thank you very much for listening.