Convergent camera applications for mobile Linux devices
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/62016 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 202392 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
MUDComputer animation
00:28
Computer animation
01:12
Computer virusMatrix (mathematics)Device driverStack (abstract data type)System programmingMobile WebSoftwareStandard deviationDisintegrationMultimediaSoftware testingRaw image formatInternet service providerComplex (psychology)Motion captureComputing platformMiniDiscDecision theoryLatent heatComputer-generated imageryAlgorithmFrame problemStatisticsDigital filterImplementationLibrary (computing)Computing platformGoodness of fitMereologyMultiplication signSlide ruleHypermediaMobile WebIntegrated development environmentOnline chatMedical imagingIn-System-ProgrammierungConnectivity (graph theory)Cartesian coordinate systemKernel (computing)File formatCodeStatisticsAlgorithmGame controllerElectric generatorAndroid (robot)Frame problemSpacetimeParameter (computer programming)Moment (mathematics)INTEGRALLoop (music)LengthChemical equationProcess (computing)Raw image formatWhiteboardVideoconferencingGraphical user interfaceComputer hardwareNeuroinformatikCore dumpKeyboard shortcutSoftware frameworkStandard deviationAbstractionStack (abstract data type)MathematicsDevice driverDifferent (Kate Ryan album)Single-precision floating-point formatReplication (computing)Computer animationProgram flowchart
07:40
Internet service providerDevice driverCoprocessorKernel (computing)Computer-generated imagerySoftwareMachine visionMessage passingLattice (order)Lemma (mathematics)Term (mathematics)AnalogyPatch (Unix)PixelVector potentialRevision controlComputing platformAlgorithmStreaming mediaHypermediaSoftware testingStack (abstract data type)Tape driveMultimediaProgrammable read-only memoryVideoconferencingComputerGraphics tabletMobile WebCartesian coordinate systemLibrary (computing)Standard deviationMultimediaMobile WebMobile appIntegrated development environmentPoint (geometry)MereologySlide ruleTunisGraph coloringMedical imagingSoftware testingStack (abstract data type)INTEGRALMappingProcess (computing)Web portalPatch (Unix)PlastikkarteSoftware frameworkDevice driverControl systemIn-System-ProgrammierungKernel (computing)Multiplication signCollaborationismCategory of beingSoftware developerUniform resource locator
14:08
MultimediaHypermediaView (database)VideoconferencingComputer animation
14:28
Inclusion mapMultimediaVideoconferencingView (database)HypermediaFunction (mathematics)TouchscreenAlgorithmCodierung <Programmierung>In-System-ProgrammierungComputer hardwareComputer animation
15:08
ComputerComputer networkMotion captureTouchscreenVideoconferencingMultimediaHypermediaGroup actionAlgorithmDemo (music)Real-time operating systemComputer animation
15:41
Standard deviationKernel (computing)SpacetimeImplementationConsistencyDevice driverComplete metric spaceAxiom of choiceAbstractionRAIDAndroid (robot)Frame problemFile formatBinary space partitioningSoftware testingSoftware developerDemo (music)VideoconferencingInstallation artDisintegrationInformation securityCurvatureMotion captureWeb browserMobile WebPhysical systemRegular graphBootingPixelGraphics processing unitAlgorithmSoftwareMereologyWeb portalCore dumpSeries (mathematics)CodeSystem callDemo (music)Process (computing)Mobile WebType theoryWeb browserTelecommunicationInterface (computing)Game controllerGeneric programmingSpacetimeMultiplication signElectronic mailing listDevice driverLibrary (computing)Repository (publishing)Frame problemDigital photographyAndroid (robot)Extension (kinesiology)Software developerDifferent (Kate Ryan album)File formatHeegaard splittingSoftwareIn-System-ProgrammierungStack (abstract data type)Kernel (computing)Patch (Unix)Wave packetNear-ringINTEGRALRegular graphPersonal area networkCartesian coordinate systemVideoconferencingIntegrated development environmentImplementationDistribution (mathematics)BitOnline helpBuffer solutionMotion captureTouchscreenDigital rights managementComputing platformFocus (optics)Mobile appSurfaceComputer animation
23:30
Computing platformSoftware developerKernel (computing)Software testingLibrary (computing)Key (cryptography)Moment (mathematics)Point (geometry)Multiplication signDigital rights managementComputer animation
26:11
Computer animationProgram flowchart
Transcript: English(auto-generated)
00:16
Hello, everyone. Thanks for attending. We were doing okay. Is that my device or theirs?
00:38
I don't know if that's me or... Okay. So today's talk is written by Yacob Omondi.
00:51
Unfortunately, he couldn't attend today. He's back, so I'm stepping in. So what I'm talking about today is not work I've done. It's about his experiences working on the
01:02
phone. I don't want to touch it now. Yeah. So my name is Kieran. Just like Yacob, I'm an embedded camera engineer with ideas on board. We've been working on kernel drivers
01:21
and for some time now lib camera. We can be found on IRC, matrix, any way you want to get ahold of us on GitHub if you need or after the chat. And today we want to talk about how we perceive the Linux camera stack on both desktop and
01:43
mobile environments. And starting with the kernel, we see lib camera as being a big part of that to support the platform abstractions. And on top of lib camera, we see lots of applications desiring to use pipe wire. So we're going to look through that. And the
02:02
overall goal is that applications shouldn't care what platform they're running on. They shouldn't care if they're running on a PC, a desktop, or a Libman 5 or a Pinephone Pro. And equally, any application you want to run or camera framework, they should all be able to say, hey, I want to talk to the camera. Give me some pictures, please.
02:23
And specifically today's talk is about the Pinephone Pro. Jacobo has spent some time over the last three months or so, or more, trying to make sure that we can bring up the Pinephone Pro with lib camera and standard applications. And the Pinephone Pro is an
02:41
interesting device because it's, I think it's promoted as like a test ground. So it's like there's no official software, but it's a good device that people can play with and develop their own software. Interestingly for us, it has an RK3399, which is a chip that has an ISP. And it's actually a device that we have already been supporting
03:02
for several years now. It pretty much was one of the first devices we started supporting with lib camera. And part of why we created lib camera is because cameras got complex. This is a slide I presented a few times. But on the left, we can see that beyond having just a single video node where you might say,
03:24
UVC, give me some pictures, cameras started getting more complicated. They have multiple components and you want to configure them. And the one on the left has now been removed from the kernel. And the N900, which already has a lot of different nodes, that's 13 years ago. So if you can imagine, there's a lot of
03:43
cameras out there now that are even more complicated than all the components there, particularly lots of Qualcomm phones. And with all those new components, it's very difficult for applications to know what to do with each of those things. Suddenly, every application has to be aware of every
04:04
platform. And that's going to lead to a lot of replication of code. Each camera application is going to have to deal with media controller to configure the pipeline, has to talk to V4L2 to get frames, has to talk to sub devices to configure parameters on the sensor itself. And that changes
04:22
for every platform. It's different on a rock chip, it's different on a Raspberry Pi, it's different on an Intel. So lib camera's goal really is to fill the gap of that abstraction so that applications only have to look at one API again. And it sits on top of V4L2. It's not a replacement for V4L2. But what we have is a pipeline handler which deals with
04:45
the platform abstraction and we have a component called the IPA. And that's crucial for devices like the PineFoam Pro with an ISP and raw biosensors because you need control algorithms. And the IPA in lib camera provides a space to do that. On top of lib camera itself,
05:04
we have a native lib camera API, which is C++. We've got Python bindings. There's people developing Rust bindings. The Rust bindings are actually giving us C bindings, I believe. Aside from that, we've got Android HAL integration, which is important and comes up
05:21
later, and integration into frameworks like Gstreamer. And as I said, the rock chip LK3399 is one of the devices we started supporting when we started lib camera, particularly on the Chromebook, Chrome tab. But it's actually a really interesting
05:41
platform because it's in a lot of small board computers as well. So it's readily available hardware. You can plug in off-the-shelf cameras from Raspberry Pi and play with it. And what I actually really like is recently we've been working on the IMX8M+, which has the same ISP core in the chip.
06:00
So the same code that we've written for rock chip also works on the IMX8M+. So I've mentioned that these cameras are now complex, and we've got this thing called an ISP, which is kind of getting in the way of people getting images out of their cameras. And the reason for that is the cameras themselves are now raw biosensors, and that
06:22
needs a lot more processing and support to get good images from. Particularly the underlying format is in a format which most applications don't want to process. So that data is fed into the ISP, but the ISP needs to be managed. It produces something called, well, it produces statistics, usually custom to each platform, and there has
06:42
to be code or an algorithm to process those statistics to then generate control parameters to configure the ISP for the next frame. And ultimately then that will process in a loop and produce you some images that the applications will expect, either YUV or RGB. And we
07:01
already have an implementation for this. This was one of the things we started early. I believe a lot of this implementation is derived from Raspberry Pi, so it's quite compatible with the implementation that Raspberry Pi have at the moment. But we've got various components like AGC to handle how bright the image is automatically or set manually. White balance is
07:21
important, then length shading are the kind of three that you have to start with. But all that code is open and already existing in the camera. The kernel driver itself has been in mainline kernel now since, I believe, 2020, and it was de-staged in 21.
07:40
Helen from Collabra was working on that, and since then it's still had active development. There's fixes that go up, and we've been working on it to extend support for the iMX8 M+. And so the kernel side and the lib camera side is looking pretty good. We've got
08:03
support for processing the images. We've got the kernel drivers. But when we go back to the PineFone Pro for quite a long time, there's no driver in the mainline kernel for the front camera, 8858. And even though there was a driver for the back camera, it wasn't tuned and
08:20
it wasn't supported very well. It wasn't tuned, really. So PineFone Pro has been left behind from lib camera for quite some time because no one was actively working on this, and it just meant that you couldn't use lib camera on a PineFone Pro. And then Yakubo has been working on this in collaboration with others who wanted to push this
08:43
forward and make it work again. And Nicholas Roth started this back in October, I think, where he wanted to get WayDroid running on a PineFone Pro. So he was trying to find out what were the missing pieces, what do we need to upstream. And this talk really derived from the work that he kick-started. So he submitted support for the rear camera, front
09:07
camera, to lib camera. And he based that on the kernel driver that was in the PineFone Pro self-hosted driver, not self-hosted,
09:21
Mege's tree. And interestingly, the driver hadn't been posted upstream, so it hadn't had any kind of review process. And it exposed itself as a name with M00F underscore OV8858. So it was encoding properties in the sensor name about
09:41
where it is and its location. And that's not very good for lib camera because it's not generic, because then we can't have a handle that says only match the front camera in location zero when we want that to support every device that has the sensor. So the upstreaming process actually highlights where
10:01
things need to be cleaned up. This has gone through some iterations. And Jacobo, who would have been talking, has taken this on to completion. And it will land in 6.3. It's accepted in the Linux Media Tree now. So that's getting in in March, I think.
10:20
The support required for lib camera, we made a release last week for 0.0.4. So now we've got a kernel with the ISP driver. We've got the sensor drivers and the camera support all upstream and mainline. The other sensor needs a lot of work still.
10:42
Interestingly, it's supported by Raspberry Pi, and the Raspberry Pi kernel has a lot of downstream patches. So if anyone wants to get involved, this is a really good opportunity to look at what is in the Raspberry Pi tree, take some of those cleanups, get them suitable for mainline, and post them up.
11:01
So if everything's there, what's next to make it good? There's lots of patches to upstream still for the 258. The next stages, really, are about camera tuning. And that's part of the process that we're trying to provide in lib camera as a framework. We've got a camera tuning tool.
11:22
And that's really about helping the control loops know how to process the images. So we have a camera tuning tool, which is being developed and can be used already. You can tune the cameras at home, simple things like taking pictures of a white wall. Ideally, you want a color card that was on one of the earlier slides. But with very inexpensive tools, you can do an initial
11:43
start at camera tuning. So if you've got devices and you want to investigate this, that's a great place to get started. So that is not my work here is done, but it's Jacobo's. That's front and back cameras from running on the device.
12:01
This is captured using CAM, which is just a pure test tool in the camera. We have CAM and QCAM. They're not meant for end users, really. It's just for helping us develop the framework. The images probably need more work on the lens shading and white balance, but that's part of the tuning process that we mentioned.
12:22
But users don't want to stop at using test tools. So what's also been going on and progressing nicely is support for application layers on top. And Robert Mader, I met back in Prague, and since then has been also working on this with his device, trying to get the same experience you get on the desktop
12:46
to work on mobile. And that's building up the camera portal in Pipewire, extending support in Gstreamer to handle controls, and mapping that all through the camera portals. So from Pipewire's perspective, this is what
13:02
the Media Stack looks like for desktop environments or anything using Pipewire, where Pipewire sits on top of libcamera, knows how to look at cameras that are v4l2 as well, but if it needs libcamera, it already has that integration. And then Gstreamer and applications can sit on top of Pipewire.
13:20
And Robert has been doing quite a lot of work trying to clean up and finish that integration of that application pipeline, particularly in getting the Gnome Camera app to work all the way through. And I remember when I first saw the Gnome Camera app, I saw it and thought, great, there's a standard
13:41
design for a desktop. I want this to work on libcamera. So this has made me really happy seeing that people have pushed this forward. Gnome Camera is a design, I've forgotten which team was designing it, but then James Westman took that design and created an application for it, which can be part of the standard Gnome environment, and also
14:00
run on mobile devices, which is the key point. If I could have put that in the slide. So, Jacobo couldn't be here today, but he did
14:29
manage to record on his PineFone Pro with a screen grab and encode on the device, running Gnome Camera on the PineFone Pro, running through Pipewire into libcamera, running libcamera
14:45
algorithms, and through the ISP. So this is hardware accelerated camera, he's taking a picture, he, in a moment, will change the camera to the front camera, there. And one of the things I like is, quite
15:01
interestingly, you can see the algorithms kick in. So you can see it starts out green, and then it corrects itself. So you can see that real time action from the algorithms that are in place in libcamera. In consumer devices, that still happens on a UVC webcam, but usually you hide those frames.
15:21
In the camera with these up here, we're just not hiding them, so you can see it. Excellent. So we can have real live demos instead of video ones. Can I get back?
15:57
So, thank you. That demo was running Gnome Camera through the
16:04
Pipewire camera portal. And thanks to Robert, if you have a device running Pipewire, you can install this flat pack, get that application, and run it on your device. I believe that will just all work. The instruction from Robert over there.
16:23
OK. I went with what I had. So that's great. We can now run camera application that's exactly the same on desktop and mobile. But there's more that we want to do on our phones or with communications nowadays.
16:42
So getting browser support is really important there. But now we've got Pipewire integration and portals. Browsers, which most of them use WebRTC, it would be really helpful if we had integration of WebRTC that
17:01
could talk to Pipewire. And Michael from Pangootronics has been working on that. There we are. I want to see you later. Has been working on that tirelessly for a year or more, I think. And I don't think I can point that far out, but what was fantastic is last week it went green and that part is merged.
17:21
There's still a few more series to get in, but the core support is now there. So in some months, that's a very wishy number, we should be able to see browsers able to handle this pipeline and you can make a video call on your Pinephone Pro, which is fantastic. It's not me, that's other people.
17:45
I do talk a lot about other people's work, so I don't want to take credit. Talking some of the distros, I know that even once the code's ready, I believe we can start getting early PPA type packages available so that we don't have to wait
18:00
for it to filter through all the upstream processes. But that's up to the distros, not me. So actually, it's quite exciting. There's a lot of development going on with LibCamera at the moment, lots of application side development. And another one that's been being supported lately since the last three or four months is Adam Piggs is
18:22
working on Sailfish OS. And he had an application called Harbor Camera that ran there, and he's ported that to run on LibCamera. So he's been working on that on the Pinephone, and it's QT-based, so I've been running it on my desktop, my laptop, and an Intel device.
18:44
But even though he's developing it on the Pinephone, because it's based on LibCamera, the platform is abstracted, so it will work on every platform that's supported by LibCamera. So the same application is now going to run just the same as a known camera will run on all the supported platforms, unmodified, which is brilliant.
19:01
I've run it, as I said, on the Surface Go's and my desktop. He has already started plumbing in manual control, so you can do things that you expect from your mobile phone camera app to control the exposure and brightness that you might want to play around with. And then autofocus and manual focus should be coming up soon.
19:24
That is a screen capture from Rafael, who is one of the LibCamera community members, who is just testing Adam's work on the Pinephone. But that's actually running on MIMO-LEST, whereas Adam's working directly on Selfish.
19:43
So again, it's nice to see that the distribution doesn't matter, it's all the way across platform. The talk started, a lot of this work here started because Nicolas Roth was trying to get WayDroid support working. He wants to run WayDroid to get the Android applications running on his phone. And so that is a way of running an Android environment
20:03
in a containerized solution on your device on a regular Linux system, such as we can now run on the phones. And I said earlier that LibCamera already provides an Android camera help. So we've already got integration there for telling
20:20
Android how to talk to the camera, which is great. So that can be reused. There's still a fair bit of work there, unfortunately. Nicolas has got it working in WayDroid. He can capture frames. But due to the format that the buffers are captured in, he can't display them. So that's going to need some more work in Mesa.
20:41
And look at how the buffer management is being handled. There may be some updates from Excellent. So there's recent developments that may improve this in the near future with Pan Frostdriver development. But it might be an opportunity if anyone wants to get dug
21:01
into those layers to work on. Millipixels is a fork from Dorota, where she's been supporting the Libram 5. And the interesting part there is she's working on a GPU-based ISP. So on devices where we don't have support for managing the
21:22
ISP, such as Qualcomm or devices that don't have one, that work would be really interesting to see extend Lib Camera to work on those devices. And Pavel's talk earlier was about sharp photos on a mobile phone, who's creating a software CPU-based
21:40
implementation, which will help get things started as well. These are lessons Jakubo wanted to highlight that he's learned. But they do apply widely. Fragmentation, when it's all split with lots of different stacks from vendors, it's very difficult to use that
22:01
generically. So Lib Camera's goal is to try and pull this all together. And to do that, it needs mainlining. We have to have a single definition of what is true. And mainlining is difficult. It takes a lot of effort. My friend that I traveled up on the train with to FOSDEM,
22:21
he was saying he posted patches to one of the Linux lists. And after four or six weeks, he had no reply. And he found that really demotivating. So it is important that you consider mainlining from the start. You've got to get in so early. It takes a lot of time. And it's always slower than when you've got control of
22:41
your own repositories. We've definitely learned more lessons from developing Lib Camera. A lot of us are derived from kernel developers, so we've been on the other side. And now we're seeing just how important it is on the user space side that these controls and interfaces need
23:01
to be standardized and have a reference implementation. Since we've created Lib Camera, we're finding that the sensor drivers in Linux are improving rapidly, we hope. We've started saying that controls have to be defined by the sensors. There's so many missing parts to the drivers that are already upstream, and they need more work to get them supported generically.
23:23
But doing so means it will all be more consistent and improve the experience for everyone. Thank you. I think I might have two minutes. Excellent. So two minutes if you do have any questions.
23:42
It already does, I think. There may be. Just to repeat the question, the question was, will Lib Camera support the original Pine phone? So Pavel is brilliant to answer that. Wait, wait, wait.
24:03
And there is a kernel on original Pine phone doesn't have required APIs for Lib Camera, so you can either break the Lib Camera to work anyway, or you can fix the kernel, and people are doing both at the moment. It's in development. But Adam Pigs, who was doing the Pinhole Camera app,
24:21
he is working on Pine phone. So no on Pine phone. So it's one of the active platforms being used. So I believe so, yes. Any more questions?
24:41
Earlier you talked about releasing 0.0.4. What's on your roadmap for an 0.1 release? So what do you want to get done? I'm sorry, I couldn't hear the question. Ah, for Lib Camera 1.0. He's ducking. He's getting out of the way.
25:02
We want to. There's key features that we want to support in Lib Camera, and it will break the API. So we already know exactly how we want to break things. I started tagging releases so that we had defined points before Christmas, so trying to do it every two months
25:21
at the moment. It is in the plan. There's a big reconfiguration API that wanted to try and get in first before we go for 1.0. But we need testing to know how to get it right. Once we go 1.0, it feels like we're going to say, that's it.
25:40
But I think some of it's more psychological. But that's versioning. So we're working on it. I'm trying to make sure we handle ABI breakages automatically now so we'll be able to improve on release management. We kind of always just said, use the latest, because we were trying to iterate so fast. And that is hopefully improving.
26:01
So we're working on it. And I'm out of time. Thank you. Thank you. Thank you for a great talk.