Advanced Camera Support on Allwinner SoCs with Mainline Linux
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61999 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
BootingOpen sourceFocus (optics)Source codeDevice driverFeldrechnerDigital rights managementComputer-generated imageryMotion captureElectronic visual displayConvex hullChainShape (magazine)Quantum stateOpticsInterface (computing)Process (computing)Raw image formatDomain nameAnalog-to-digital converterData conversionTime domainPixelPattern languageNoiseBefehlsprozessorCoefficientMatrix (mathematics)Gamma functionContrast (vision)Electric currentInternet service providerComputer hardwareGroup actionDistortion (mathematics)GeometryLatent heatSystem programmingMultimediaParameter (computer programming)AreaDiaphragm (optics)Equivalence relationNumberOperator (mathematics)Level (video gaming)Focus (optics)Artistic renderingType theoryWiener filterMedical imagingProcess (computing)In-System-ProgrammierungSystem on a chipDigital rights managementComputer hardwareTable (information)FeldrechnerSmartphoneKernel (computing)Physical systemReal-time operating systemPixelBlock (periodic table)Beta functionGamma functionMotion captureOcean currentSoftwareGraph coloringMultimediaSampling (statistics)Different (Kate Ryan album)Thermodynamischer ProzessComplex (psychology)Parameter (computer programming)Electronic mailing listRepresentation (politics)Power (physics)Order (biology)BefehlsprozessorAreaLatent heatIntegrated development environmentRaw image formatChainOpticsDomain nameGoodness of fitGeometryRow (database)Service (economics)Device driverInformationStability theoryFilter <Stochastik>BitElectronic visual displayVapor barrierTerm (mathematics)Wave packetCalculationComputer animation
08:31
Parameter (computer programming)Equivalence relationFocus (optics)AreaSource codeDiaphragm (optics)AlgorithmControl flowPersonal digital assistantPhysical systemStatisticsLoop (music)FeedbackLatent heatComputer hardwareMotion captureInterface (computing)Parallel portSerial portDigital signalBit rateDifferential (mechanical device)AnalogyPixelSynchronizationCASE <Informatik>Internet service providerComputing platformComputer-generated imageryThermodynamischer ProzessSystem programmingBefehlsprozessorArmTransmitterRaw image formatFile formatConnected spaceComputing platformoutputDevice driverMereologyMultimediaSoftware development kitType theoryVideoconferencingSpacetimeGeneric programmingElectronic visual displayRead-only memoryMemory managementStrutFunction (mathematics)Buffer solutionConfiguration spaceComputing platformParameter (computer programming)Integrated development environmentIn-System-ProgrammierungInformationMedical imagingMultiplication signAnalogyDemosceneBitMotion captureComputer hardwareInterface (computing)System on a chipWhiteboardRow (database)Form factor (electronics)MereologyType theoryShooting methodLibrary (computing)Electronic visual displaySynchronizationAreaStreaming mediaMultimediaDifferent (Kate Ryan album)Noise (electronics)Electric generatorSource codeDevice driverState of matterChemical equationArmGraph coloringLine (geometry)Electronic mailing listFeedbackNumberProjective planeThermodynamischer ProzessComputing platformKernel (computing)AlgorithmWindow1 (number)Software developerComputer animation
16:57
MultimediaGeneric programmingSpacetimeType theoryVideoconferencingElectronic visual displayMemory managementRead-only memoryInterface (computing)File formatStrutBuffer solutionMotion captureFunction (mathematics)Configuration spaceControl flowComplex (psychology)System programmingBlock (periodic table)Vertex (graph theory)TopologyData managementStreaming mediaGraph (mathematics)Device driverInheritance (object-oriented programming)Network topologyKeyboard shortcutInformationTelephone number mappingBridging (networking)Link (knot theory)SynchronizationImage registrationFrequencyPixelConnectivity (graph theory)Device driverCharacteristic polynomialMultimediaGame controllerLevel (video gaming)Process (computing)ChainGraph (mathematics)Pointer (computer programming)Computing platformNetwork topologyMedical imagingMereologyInterface (computing)BefehlsprozessorImage registrationElement (mathematics)Buffer solutionBus (computing)Link (knot theory)SpacetimeDescriptive statisticsVideoconferencingCategory of beingStreaming mediaRepresentation (politics)Message passingInheritance (object-oriented programming)Keyboard shortcutHeat transferType theoryInformationFile formatSemiconductor memoryDifferent (Kate Ryan album)NumberMultiplication signCartesian coordinate systemResultantComputer animation
25:23
Device driverSynchronizationImage registrationLink (knot theory)Graph (mathematics)StrutSystem programmingComplex (psychology)HypermediaGame controllerMultimediaBlock (periodic table)Source codeSpacetimeTopologyMotion captureVideoconferencingFunction (mathematics)BefehlsprozessorComputer-generated imageryInternet service providerDisintegrationHistogramStatisticsInformationBuffer solutionParameter (computer programming)GUI widgetData bufferQueue (abstract data type)PixelInterface (computing)Thermodynamischer ProzessMeta elementCoordinate systemBit rateControl flowSoftware development kitCodePhysicsMixed realityGeneric programmingDDR SDRAMInterface (computing)Axiom of choiceVideoconferencingGraph coloringParameter (computer programming)Network topologyFunctional (mathematics)InformationMultimediaBuffer solutionElectronic mailing listMeta elementPoint (geometry)StatisticsData structureImplementationIn-System-ProgrammierungMotion captureMathematicsGame controlleroutputDemosceneDifferent (Kate Ryan album)Code2 (number)Multiplication signFunction (mathematics)File formatGraphics tabletSpacetimeNumberComputer hardwareDisk read-and-write headLink (knot theory)PixelBlock (periodic table)Computer programmingRepresentation (politics)FeedbackConnected spaceRun time (program lifecycle phase)ChainINTEGRALSystem on a chipValidity (statistics)MultiplicationElectric generatorUtility softwareFlow separationConnectivity (graph theory)Computer animation
33:49
Interface (computing)CodeGame controllerControl flowBit ratePixelSource codeSoftware development kitPhysicsMixed realityVideoconferencingMultimediaDevice driverBlock (periodic table)Generic programmingDDR SDRAMInternet service providerPatch (Unix)BenutzerhandbuchAsynchronous Transfer ModeDecision theoryComputer hardwareCategory of beingNetwork topologyComputing platformBootingFingerprintIterationFunction (mathematics)outputTopologyDynamic random-access memoryMemory managementFlow separationBridging (networking)DisintegrationComponent-based software engineeringMotion captureModul <Datentyp>Parameter (computer programming)Data structureStrutData bufferCoefficientMotion captureVideoconferencingDefault (computer science)Game controllerComputer hardwareInterface (computing)Sound effectFrequencyCategory of beingMereologyAsynchronous Transfer ModeSeries (mathematics)Different (Kate Ryan album)Device driverElectric generatorData structureFrame problemBenutzerhandbuchNetwork topologyMathematicsPhysicalismBitBuffer solutionElectronic visual displayDirection (geometry)Communications protocolComputing platformFeedbackSynchronizationIn-System-ProgrammierungNumberParameter (computer programming)Configuration spaceBus (computing)ImplementationBlock (periodic table)Generic programmingCoefficientCodeArithmetic meanInheritance (object-oriented programming)Source codeSemiconductor memoryProper mapElement (mathematics)Computing platformSoftware development kitComputer animation
42:15
DisintegrationInternet service providerModul <Datentyp>Parameter (computer programming)Data structureStrutData bufferCoefficientSeries (mathematics)Device driverPatch (Unix)Computing platformMultimediaIterationRotationImplementationComplete metric spaceComputing platformComputer hardwareRevision controlSpacetimeAlgorithmGamma functionMatrix (mathematics)Latent heatAbstractionTape driveAndroid (robot)VacuumKernel (computing)ComputerBootingOpen sourceBuildingRootSlide ruleBuffer solutionConnectivity (graph theory)Device driverNumberFitness functionDirection (geometry)Patch (Unix)Context awarenessComputer hardwareBitSource codeSemiconductor memoryProjective planeInternet service providerWordMultiplication signPosition operatorFreewareSoftware frameworkShared memoryVirtual memoryStatisticsRow (database)MereologyModule (mathematics)RotationCompilation albumAdditionEndliche ModelltheorieMathematicsSpacetimeDigital rights managementIn-System-ProgrammierungImage processingBilderkennungRevision controlSpecial unitary groupImplementationAlgorithmGraph coloringNoise (electronics)Computing platformArtificial neural networkVirtual machineSoftwareFrequencyNumeral (linguistics)Stack (abstract data type)Descriptive statisticsFeedbackSinc functionRepresentation (politics)Kernel (computing)Function (mathematics)Scaling (geometry)Data structureDifferent (Kate Ryan album)Codierung <Programmierung>Structural loadCodeoutputComputer animation
50:41
Program flowchart
Transcript: English(auto-generated)
00:05
OK. OK, so let's start. So hi, everyone. I'm going to be talking about advanced camera support on all Wiener SoCs today. So the topic is quite similar to the previous one, in that we are also going to talk about sensors that need postural care and processing.
00:25
So unfortunately, I will be maybe explaining some of the things that were said by Kieran just earlier. So sorry about that for the people who just attended the previous talk. But hopefully, you'll also learn a thing or two in addition. So I'm Paul.
00:41
I work at a company called Bootlin. We are an embedded Linux engineering company. So we do services around that. And I have contributed a number of things, especially to the Linux kernel. So I worked on the Cidrus VPU driver, a little bit of DRM things on the all Wiener side, but also others.
01:00
And I made a training that we give about displaying and rendering. But today, I am here to tell you about the camera support for all Wiener SoCs using mainline Linux. So we are going to talk first about general things related to image capture technology and just complex camera
01:23
pipelines in general. So let's start with kind of an overview of a typical capture chain. So we start through the optics, of course, where the light is kind of focused on a particular area where we have our sensor. So the optics are usually passive from a software
01:43
perspective, but not necessarily because you can have coil drivers to change the focus and things like that. So it's not always a passive thing. So after that, you have the sensor, which actually samples the lights and produces data from that.
02:01
And we want to transmit that data to something else, typically like your CPU, where you can do something with the data, like display it or encode it. But in between the acquisition and actually receiving the data, we need to do some processing. And that processing can sometimes happen on the sensor itself,
02:22
in which case we talk about some embedded processing. Or it can be on the other side of the interface, so on the side that received the data, typically your SOC or your CPU package or whatever. So this processing step is really
02:41
the one that is necessary to produce good looking pictures from, let's say, just samples coming out of an ADC. So typically, what we call a row or barrier sensor will produce not pixels, but data
03:01
in a biopattern, which is a grid of red, green, and blue filters that is applied on the sensor. That gives you information for each of these different channels. And we kind of need to translate that to pixels. So this is called debioring, and this is how we get pixels.
03:21
But these pixels typically look very bad. So you need to apply a number of processing, a number of operations and enhancements to have something that looks like a nice picture. So a number of things need to be done. For example, the brightness that you get from your ADC
03:40
is linear, and we want to apply some gamma curves to that to make it look nice to the human eye, typically. There's some dark level current that we have to subtract, for example, because the zero value that you get from the sensor is not necessarily the,
04:01
well, the darkest value that you get from the sensor is not necessarily zero, so you might need to subtract an offset, things like that. There's usually a lot of noise, and the colors will be off, so you will need to do some white balancing, things like that. So all of these different steps take place in what we call the ISP, the Image Signal Processor.
04:21
And there's basically three domains in which we apply these enhancements. The first one is the bio domain, so that's really the first step that we apply to the data coming from the raw sensor. At the end of that step, we get some RGB data
04:40
that we also want to enhance, and at the end of that, we typically convert it to YUV representation. And then we can also apply some enhancements to that data. And at the end, we get a YUV picture that is ready to be displayed and encoded, for example.
05:00
So yeah, that's kind of a list of the different enhancements that we apply. So I mentioned a number of them already. I'm not going to go through the list, but you get some idea that there is really a lot of things to be done here. And it actually takes quite some processing power to do that. So that's why typically we consider that it's not something you can do in real time with a CPU
05:23
or it's going to fully load your CPU just to produce pictures and let alone encode them and things like that. So lots of things to do. That's really the base that you need to have something that looks right. There's more advanced stuff that you can have in addition, like the lens shading correction.
05:43
So it's about the fact that the lenses will typically be darker on the edges than they are at the center. So you want to even that out. That's also an operation to do. D-warping, that's when you have a very short focal and things look, well, the geometry looks distorted.
06:02
So you need to readapt that. That's also very intensive in terms of calculation. Stabilization can also be involved if you have very shaky footage, especially from a smartphone or something like that. So you want to also apply a stabilization step to your picture.
06:21
And then finally, you might also want to apply some style to your pictures. So that will typically be a color lookup table where you can decide that you want to make it look I know like sepia tone or something like that. This is also some processing that you will need to apply. So I mentioned that there's basically two types of ways
06:41
to deal with this operation. The first one is to have the ISP in the sensor, in which case it's typically quite simple. And when it's in the sensor, you get the data directly ready from the sensor. But when it's not, you get just the raw beta data
07:01
and you need to do all of these different enhancement steps on some system on a chip ISP. So that's typically a hardware block that is dedicated for the purpose in your SOC. So nowadays, many multimedia oriented SOCs do have such blocks. And in order to properly configure that,
07:22
you might need some specific calibration data that really depends on the sensor, sometimes on the environment that is used, things like that. So it's kind of highly specific to the setup that you have. So it's kind of just an illustration of the different steps. So that's the kind of picture that you would get
07:41
as a raw thing from the sensor and the steps you might apply to have something in YUV at the end that looks kind of okay. But it's not just about statically configuring an image processor to produce something good.
08:00
Some parameters actually depend on the thing that you are shooting. So there's basically three things that you need to adjust depending on the situation. The first one is focus. So of course that implies that you have control over like some coil to change the focus of the lens. But obviously your picture is gonna look very different
08:21
if it's out of focus or if it's sharply focused. So that's one of the things that the ISP is also involved with to basically tell you whether an image is sharp or not. There is white balance which highly depends on the source light that you're using, especially the color temperature of the light.
08:40
So if you're in broad daylight, it's not the same as being in a room with some particular type of lighting. So it needs to adjust to that. It will typically have an impact on how the image looks. And of course exposure. So what is basically the window of luminescence that your sensor is going to sample.
09:02
So if you are in a very bright environment, you need to apply a different gain than when you are in a very dark environment. So that's also something that needs to be adjusted depending on what you are shooting. And this is also something that the ISP is going to help for by telling you basically
09:21
how bright or how dark the scene is. So yeah, you can adjust those, especially exposure. You can adjust with three parameters. So if you do like photography, you probably know about that. You can change the aperture of the lens. You can change the exposure time. So for how long you are waiting for light to come in
09:43
to charge your cells, that will be read by the ADC. And you can increase the gain, which will also increase noise typically. So advanced users will typically want to control these parameters manually to have exactly the picture that they want. But in most cases, people just want to
10:02
take their phone out and shoot at something and just press a button and it just works. So the idea is that we want all of these different parameters to be adjusted automatically. So that's what we call the three A's. So the three A's are automatic exposition, autofocus and auto white balance.
10:23
And that is typically again something that will be done with the ISP. So it works with a feedback loop. Okay, there's a number of algorithms in the literature that exist that are known to be able to do that correctly and efficiently.
10:43
But the way they are implemented in the actual ISP hardware really depends of course on the hardware itself and how it was designed and what the register interface is to configure these different things. And that is often considered to be the secret sauce of the manufacturer of the ISPs.
11:03
So that's the information that is often kind of difficult to get and that they don't want to release and so that's why sometimes you end up with like a big binary blob that does all of this and you don't really know what's going on. Okay, so that was for the kind of parameters
11:23
for the image enhancement. Now let's take a little bit of a look at the hardware interfaces for the capture. So historically there's been different ways to transmit pictures from one side to another. There used to be analog interfaces which are now mostly duplicated
11:41
so let's not really focus so much on those. And then we have typically two types of hardware interfaces that are used for cameras. First one is the parallel, also called DVP sometimes. And that's when you basically just have like one line of data per bit, you have some sync signals
12:03
so it's a little bit like a display, a parallel display if you know about that. And so you just kind of send the data like that and there's also more advanced interfaces which are also more robust to noise that typically work with serial lanes.
12:21
So there is mypy, csi2 and other ones like lvds, sdi, highspy. So those are kind of the high end interfaces that allow you to stream a lot of data. They typically go pretty high speed, they are more robust to noise so they are considered to be like the advanced ones. So that's the one mypy, csi2 that we are going to focus on.
12:46
Through my particular use case involving the old winner platforms. So in case you're not familiar with the old winner platforms, they are arm SOCs made by this company called Old Winner from China.
13:01
They are widely available, especially on these kind of form factors as developer boards. And there's a number of these platforms that support mypy, csi2 and that have an image signal processor. So it means that we can connect a row bayer sensor and get the data from that pipy through the ISP
13:23
and then get a picture at the end. So that was kind of the goal of the project that I had involving these platforms. So the scope was on the v3 and a83t platforms using two different image sensors,
13:41
ov8865 and ov65-5648, which are like I just said, mypy, csi2 sensors that provide row bayer data. And these sensors don't really have an onboard ISP. I think one of the two actually has one but it does very, very little
14:01
so you still need to do a lot on the receiving end of the interface. So that was a goal. What's the state of Old Winner camera support in general with the mainline kernel? Because we wanted to use the mainline kernel of course. Let's first take a look at the general Old Winner platform support.
14:22
So there is a community called Sanxi, or Sanxi I think, which has been working towards mainline support for Old Winner SoCs. So it's very advanced, there's lots of people involved. You can check out this link, the Linux Mainlining Effort, which kind of lists all the features of the different SoCs
14:41
and how they are currently supported in mainline Linux. And it's pretty impressive nowadays. Many of the features are supported, especially for the older SoCs because of course it takes time to get it right. But the multimedia areas are often the ones that come last in support
15:01
because they are typically a bit complex to implement. So when I started the project, there were two drivers for capturing data. The first one is the Sunfly CSI driver, which covers the first generation of these Old Winner platforms.
15:23
It was, the hardware then evolved into a second generation, which is supported in mainline by a driver called Sun6i CSI. Okay, and after that, Old Winner made a new generation of platforms which have a third generation of CSI, which is currently not supported.
15:42
So the devices that I was interested in, so the V3 and A83T, work with a second generation driver. So this driver basically allows you to receive images from the parallel interface, but it didn't support MyPyCSI2
16:00
and it didn't have support for the ISP. So there were actually some, there was actually some support for these features in the downstream vendor Old Winner kernel. So they do have some code for that, but the ISP part especially was implemented as a binary blob.
16:21
Okay, so it was like a static library that was linked to the kernel, which is not necessarily very legal, but nevermind. So it was, there was actually very few resources regarding how the ISP works on these platforms.
16:41
Okay. Right, okay. So generally speaking, how do we support cameras in Linux, at least at the kernel level? So there was this API called v4l2 that I think, well, you've all heard of just before, and probably many people know about.
17:00
So it's really about supporting anything that produces pixels that the CPU can receive. So it supports lots of different types of devices, not only cameras, but also, I don't know, things like scalers, DVBT receivers, lots of different things now,
17:22
decoders, encoders, things like that. So really lots of different devices related to pixels. And typically the way it works is that you have one device node that corresponds to a driver, so typically dev video zero, and that device node gives you access
17:42
to an API from user space where you can do all the different things that are necessary to get a picture from user space. So typically negotiating the pixel formats that you want to receive, doing the memory management, like allocating the buffers,
18:00
how many buffers you want, et cetera. Queuing and dequeuing buffers. So user space provides a buffer to the driver which will fill it with pixels and then return it to the application, and then the application has a buffer that has pixels in it that it can use to again display or encode them or whatever.
18:22
So this video device works well for I would say all-in-one devices where you basically just receive the finished data from a device like a USB UVC camera. Okay, so the camera itself will do all of the processing inside,
18:41
and it will just give you the final result over USB and you get that through this API on Linux. And yeah, typically you need some DMA interface to do that transfer. But in the case of a more complex pipeline, especially when you have multiple components
19:01
involved like with the ISP, with the mypycsi2 receiver, with a particular sensor that you can control directly, then you end up with a situation where you have multiple devices in the pipeline and you kind of need to configure each one of these devices individually. So this called for a more advanced API,
19:21
which is the subdev API, which allows not only to have one big device for receiving the data, but also side devices that you can use to configure each component in the chain. And there is also the media controller API that allows you to kind of control the topology
19:42
between these devices. So the subdevs typically just represent one of the parts of the pipeline and they typically cannot do DMA, so they will be connected from and to other devices through some interfaces that don't involve writing the data to memory.
20:03
So it's like, yeah, it could be a FIFO or it could be an actual hardware interface like mypycsi2. And basically the top level video device will be in charge of kind of calling the next subdev in the chain,
20:21
which we'll call the next one it's attached to and et cetera, so that, for example, you can coordinate starting the stream and starting all the elements at the same time to start receiving an image. But these subdevs still need to be parented to the v4l2 device.
20:41
So basically they need to be all controlled under the same top level entity to be able to, let's say, coordinate between one another. So for that, there is an API in v4l2
21:01
that allows you to register the subdevs with a v4l2 device. So again, that's the parent controlling entity, which is easy to do if all of the support for the subdevs are in the same driver because you have access to that v4l2 dev pointer. But it can also happen
21:21
that you have multiple drivers involved throughout the tree. So for example, you have one driver for your sensor, one driver for your GMA interface to transfer the data, one driver for your ISP, and you could even have more. So in that case, the drivers don't know exactly which other driver they should be attached to.
21:42
So in that case, there is a asynchronous subdev registration interface, which allows you, when basically you have, for example, a sensor driver, to just make that subdev available to whichever driver is going to need it later.
22:01
So the subdev drivers will just make the subdev available to the rest of the world. And then the top-level drivers will need a way to identify which subdevs they actually need and to get a handle of them, which will allow registering these subdevs with the top-level v4l2 device.
22:23
So the way that this kind of linking is done is through the fwnode graph, which is typically implemented in device tree. So it uses the port and endpoint representation that maybe you've seen in some device trees implementing this.
22:42
And this description also allows describing some characteristics of the interface. For example, if you have a sensor that is on a mypycsi interface, it can use a different number of lanes. So in mypycsi2, you can have up to four lanes,
23:00
but maybe the sensor only uses two. So you have to kind of be able to share this information. And this is also done through this fwnode graph description. So you have some device tree properties that you had to indicate that. And then the drivers can call this endpoint pass helper
23:21
to actually retrieve the information about the interface. So to illustrate, on the left side, we have some sensor here. Okay, so we have the port and endpoint representation. The remote endpoint allows you to connect two sides together. And you have these extra properties here,
23:41
like the data lane and link frequencies that really describe the characteristics of the bus. So at which frequency it should be running and how many lanes it should have. And then on the other side, you have the same thing. In this case, the link frequency is controlled by the sensor, so you only need to provide it there. But the data lanes is present on both sides.
24:03
So that's how you can link basically different devices and allow the top level driver to retrieve access to the sub-devs that you want to use. So this is very flexible, of course, because then the same, for example,
24:21
sensor driver can be connected to lots of different platforms and lots of different situations. So it's really, the driver itself doesn't know about how it's connected. It's really the device tree and the fw-node graph that tells you how it works. So back to async notification, just quickly to illustrate how the top level driver
24:44
would gain access to a sub-dev. So first, it has to match, using that fw-node graph representation, it has to match a particular sub-dev. And the top level driver registers a notifier,
25:04
which has a number of callbacks that will be called when a particular device becomes available. And then it can pretty much bind to that device. And then the matching sub-dev will be registered with a top level v4.2 device. And then everything can be linked together.
25:23
And the top level driver actually has a pointer to a v4.2 sub-dev that it can use to apply some actions, like start streaming, stop streaming, or configure the format, or things like that. So this is how it kind of all works together. So yeah, that's also when the media controller API comes in.
25:44
So the media controller API is there to control the topology of how these different devices are actually connected between one another. So it also implements particular functions. So you can say this block attached to this sub-dev
26:00
is an entity of this kind, okay? And each sub-dev has an associated media entity, which lists pads, which are basically in and out points that you can use to connect other devices. And then you can create links between these pads,
26:22
which represent the actual connection in the hardware. So for example, you could have multiple links that are possible for one device, and then you could decide to enable one at runtime. So I don't know, for example, if you have a multiplexer or something like that, that would be a typical case where you would just select one of the inputs and have just one output. So this is really the API that allows you
26:42
to configure the topology of the whole pipeline and how everything is connected together. There's also some runtime validation to make sure that when you connect two entities, they are configured with the same pixel format so that everyone agrees on what the data will be,
27:00
the data that will be transferred. And there is a user space utility called mediactl that you can use to configure these links. So for example, here I'm configuring pad number one of this sub-dev to be connected to pad number zero of this sub-dev, and the one indicates that the link should be enabled.
27:21
So yeah, it's a bit blurry. This is kind of just to give you some kind of big idea or kind of a head start on that, but it's definitely complex, so it's normal that it seems a little bit blurry. It's just in case you have to work on that, then you know what are the things involved in this.
27:42
So in the end, we can end up with very complex pipelines, okay? So each of the green blocks are sub-devs, okay? So they represent to a specific functionality that can be connected in different ways. And the yellow blocks are the actual DMA engine, so the video nodes that are visible from user space
28:03
that programs can connect to to receive the data. But of course, if you haven't configured the rest of the chain properly, then there will be no data available. So this is really what you use at the end when everything is configured and everything is ready, and it works.
28:20
Okay, so that was for the general pipeline integration thing. Now let's talk about ISPs more specifically. So ISPs are just a kind of sub-dev and media entity. And they typically have an internal pipeline with multiple things in it, so we don't necessarily represent the internal pipeline
28:42
unless it's relevant, so there will normally just be one sub-dev for the ISP. But this sub-dev will have highly specific parameters, like I said, it depends on the hardware implementation, so the representation of the parameters
29:01
that you give to the hardware will differ from one implementation to another. So it means that it's actually very hard to have a generic interface that will work for every ISP, and that would be the same. So instead of that, in v4l2, there is actually driver-specific
29:23
or hardware-specific structures that are used to configure the ISP sub-devs. So the way it works is that we have, so one or more capture video devices, that's the same as the dev video zero
29:41
where you get the typical data, the final data that you want. And we have extra video devices that we can use to configure the ISP and to get side information from the ISP. So these are the meta output and meta capture video devices.
30:04
So the meta output is there for parameters. So in v4l2, output is when you provide something to the driver, not when you get something from it, which is a bit confusing, but that's what it is. So with that, basically, you will also use the same Q interface as you have with a video device,
30:20
but instead of having pixels in the buffers, you will have particular structures that correspond to the parameters of the ISP that you are going to fill with a particular configuration, and then you can push that as a buffer to the video device, and the ISP will be configured to use those parameters. For the meta capture, which is the data provided by the ISP,
30:42
you get the typical feedback information from the ISP. So essentially, it will be statistics about how sharp the picture is, how dark the picture is, things like that, so that you can use this information to create a feedback loop and then provide new parameters in the output video device
31:04
to properly configure the ISP to respond to a change in the scene or something like that. So for example, if you switch off a light and turn a different one on that has different color temperature, for example, then you will get the information from these statistics,
31:22
and you will be able to adjust the parameters to respond to that change. So that's how it works. Here is an example from the RK ISP, the Rockchip ISP one, where you can typically find this same topology. So the ISP is here.
31:42
It actually has extra sub-devs before having the video devices for capturing the pixels, but you also find this statistics video device and params video device. So the params will take a particular structure here that you can configure,
32:01
and the statistics will take another one with the information provided by the ISP. Okay, so that gives you kind of a big overview of how all of this is supported in v4.2 and mainline Linux. So now let's take a look at the thing
32:21
I actually worked on for the all-winner cameras. So using, again, these same interfaces for the particular use case of all-winner cameras, or cameras interfaced with all-winner SOCs. So in the all-winner,
32:41
second generation hardware implementation, we have MyPy CSI2 controllers, which are really the components connected to the actual bus, the actual MyPy CSI2 bus, which are separate hardware blocks that are connected through a FIFO to the CSI controller,
33:01
which is really just a DMA engine that will get some pixels in and write them to memory, basically, with some formatting and timing things, but essentially that's what it does. So this CSI controller, again, was already supported in mainline, but not the MyPy CSI2 controllers.
33:21
So the CSI controller actually also needs to be configured specifically to take its input from the MyPy CSI2 controller instead of the parallel interface, which is the only choice that was supported before. So that's one of the things I had to add support for. So there was a lot of kind of reworking of the CSI code to support that,
33:43
even though the biggest rework was actually to support the ISP. We need to get some information from the sensor to properly configure the MyPy CSI2 interface on the receiving side. So for that, we use a default to control that the MyPy CSI2 controller is going to retrieve
34:01
from the sensor driver through the subdev interface again, so it knows what the clock frequency of the bus will be. And we also use a part of the generic Linux PHY API
34:20
to do that, because MyPy CSI2 works with a physical, let's say, protocol or physical implementation called DPhy from MyPy, which is kind of like the physical layer implementation that is used by this interface. So there needs to be some configuration about that.
34:42
And yeah, for that, we use the Linux PHY API. Now, if we look more closely at the platforms that I got interested in, first, for the A83T, there was actually some source code provided in the all we know vendor releases
35:00
that we could use as a base to implement a driver, a proper mainline driver. So it has lots of magic values in registers, so sometimes it's just writing things through registers and we have no idea what it means, but we basically just took that in and did the same, and it just worked. So there's still some magic involved,
35:20
but that's unfortunately not so uncommon, so we just have to deal with it. The DPhy part is separate, so it has different control registers, but that was also supported in that all winner SDK downstream code, so we could also just reuse the same thing, and it worked.
35:42
For the A31 and V3 support, so it's like, again, the second generation of all winner SOCs, we have a different MyPy CSI2 controller from the A83T, so it was necessary to write a separate driver for that one. There was also reference source code available
36:00
and some documentation in one of the user manuals of the platforms, so that was, again, sufficient to write a driver. It turns out that the DPhy part is actually the same controller that is already used for MyPy DSI, which is a display interface
36:20
that uses the same physical layer encapsulation, I would say. So there was actually already a driver for the DPhy block used for MyPy DSI, in which case it's in transmit mode, because when you want to drive a display, you push pixels out, but in that case,
36:40
we reused that driver, but configured it instead in receive mode for MyPy CSI2, so we could get pixels in. So that was also a change in this driver. But it was then necessary to indicate in which direction it should be running,
37:01
so there were different approaches that were possible for that, so I think at the end, we settled for a particular device tree property to configure this mode.
37:21
Okay, so the kind of outcome of this work was first some series to support the MyPy CSI2 controllers, so about 2,600 added lines, so pretty big, that's two new drivers here and here,
37:40
some changes to the DPhy, like I just mentioned, and some device tree changes, so that's most of it. I started this work in October 2020, and it was merged in Linux 6.0 in June 2022, so now these drivers are upstream, and you can use them, and they work, and I actually got a number of people writing to me
38:03
and saying that they actually have been using this in different situations, and apparently it works pretty well, so I'm pretty glad about that, it's pretty nice. Okay, so that was for the MyPy CSI2 part, then let's say the big part of the work
38:21
was supporting the ISP, so the ISP is connected to the CSI controller as well, but on the other side, meaning that the data will flow from MyPy CSI2 to the CSI controller to the ISP, so there also needed to be some configuration
38:40
to be able to support that, especially big rework was required because when you start using the ISP, the DMA engine that is used to write the data to memory is no longer the DMA engine of the CSI controller, so the CSI has to act like a regular sub-dev, okay, it's no longer the final,
39:02
let's say the final sync for the data, but it's just one more element in the chain, so the driver had to be reworked to support this different mode of working where it will basically not register itself as the parent v4.2 device, but instead it will register itself as a sub-dev and the parent v4.2 device will be the ISP driver,
39:24
which is again a separate driver, so that required quite some rework and also to support both modes obviously because not everyone is interested in using the ISP or not every platform even has an ISP, so there needed to be some rework to support that.
39:53
What else to say? It has, I don't know if I put it here, but it has some weird way of configuring it,
40:01
basically in a typical hardware you would just have some registers and configure them and then the effects will be applied like on the next frame or something like that, but in that hardware it actually has a DMA buffer where you write the new values of the register and then you trigger some update bits
40:22
and the hardware itself will go and read from the DMA buffer and copy that data to its registers synchronously with the vertical synchronization so when you receive a new frame, so it's very odd as a way of working, but that's how it does, so if you write directly to the registers
40:40
it won't actually do anything, you need to write to a side buffer and then tell the hardware to update its registers from that buffer, so it's a little bit weird. If you look at the driver you'll see that there is this buffer that is allocated for that, so that's the reason why, that's how it works and that's what the old winner code is doing,
41:01
so that's how it's done. So that's the final pipeline that we have with the sensor here connected to the MyPy CSI2 subdev, so which is a separate driver, then it goes through the CSI driver which in this case is configured as a subdev only
41:22
and then it goes to the ISP subdev which provides a DMA capture interface where you have the final data that was processed and that should look good and it also has another video device for the parameters, like I described with the Rockchip ISP, this one is implemented the same way,
41:42
so we also have a specific structure to configure it. Currently there is no support for the statistics but in the future when such support is added there will be another video device connected to this ISP subdev to be able to provide the feedback data out.
42:03
Okay, so yeah that's pretty much what I just said, few details about the currently supported features in that config parameters buffer. Currently we support the buyer coefficients so we can translate from the buyer row data to actual RGB data
42:22
and we can tweak how much of each color channel we put in so that will typically allow different color temperatures basically. We also support 2D noise filtering which is called BDNF, so it's bidirectional noise filtering which basically is like a low pass filter
42:44
so it will remove the high frequency stuff in your picture and that will make it look smoother and nicer and also easier to encode which is one of the big reasons why you need to do noise filtering.
43:01
And yeah that's the main two features, so there's still a lot to be added, that's just the scope of what our project was at the time but there's definitely a lot of room for improvement so the ISP itself has numerous hardware capabilities and so those could be added later in the driver.
43:20
So it was for that reason submitted to staging in Linux because we don't yet support all the features so we don't yet have a complete description of that structure and since it's part of the API we want to make it clear that it's not finalized yet. Okay so there will be some additions to this structure
43:41
to support other features that are currently not implemented. So this code was submitted in September 2021 and it was merged in November 2022 so this is also in Linux 6.2 so you can get that with the update so that's pretty nice.
44:01
This change was much bigger, you can see it's 8,000 lines of additions. So it's a whole new driver and a big rework of the previous Sun which was more or less a complete rewrite of the driver so it's like pretty big.
44:23
Okay just to finish on what is left to do in this area. So currently the ISP only supports the v3 platform but the same hardware is found on the AT3T and there's a few other chips that have previous versions of the same hardware
44:41
so they could be supported in the same driver so that's something that could be done in the future. I mentioned that there is no statistics currently so that is also something that could be added in the future. It has numerous other features that we could support. Stating rotation and of course all of the modules
45:02
inside the ISP for all the different features that I mentioned. And we don't have any 3A algorithm support in user space to do this feedback loop implementation so that is also something to be worked on.
45:21
And of course, doing that would be a great fit for libcamera. So Tyran has just talked about it so I won't go over it again but that's definitely a good fit for supporting an ISP with mainline Linux. So hopefully it will soon be well integrated
45:40
in libcamera. Someone recently submitted patches about this so it's like going towards this direction so that's pretty nice. That's pretty much the end of this talk. I just wanted to mention that Brooklyn is hiring so if you are interested in this kind of stuff, hardware support, everything, you can reach out to us
46:02
and we have positions available, also internships so feel free if you're interested. And that is pretty much it for me so thanks everyone and now I'll have questions.
46:20
Hi there, that was fantastic, thank you. Who knew it was so complicated? The last time I looked at some of this with NXP free scale parts, we were using Gstreamer with v4l sources coming into it and a lot of the headache was that there was loads of buffer copying all over the place and there were different memory maps and different access for different components to different memory maps.
46:41
So with what you're explaining here, typical use case might be we do this image processing then I want to encode it with h.264265, maybe I want to push it into a GPU to do some kind of image analysis with AI machine learning techniques. Could you say something about how that hangs together with buffer copying and so forth?
47:01
So basically nowadays the v4l2 framework has great support for DMA buff which is the technology used for buffer sharing across different devices. So with that driver you could absolutely reuse the same memory where the ISP is producing the picture and use that as the source for an encoder or even the GPU
47:21
because DRM also supports DMA buff pretty well. So you could do all of that with zero copy, that's definitely all supported and I didn't have to do anything special to have that work, it's just the v4l2 framework has that now. So unless your hardware has weird constraints like the GPU cannot access this part of memory or things like that,
47:41
which are not really well represented currently but in the general case it should work pretty well. So yeah, basically when we have an encoder driver for this all winner platforms, we will definitely be able to directly import ISP output to encoder inputs and no copy and low latency.
48:01
So anyone else? Yeah, thanks for your talk and for supporting hopefully more mainline Linux so we have more phones available. I have a question about the support for artificial network accelerators.
48:23
Do you have any idea if this is somehow integrated into the kernel stack in this way? I mean, it's a lot of work like this as is but well. Yeah, so the AI accelerator stuff that's not really the same scope as the camera stuff
48:41
but that is definitely moving forward. There is an axle subsystem that was added to the kernel quite recently which is based of DRM for some aspects and I think more and more drivers are being contributed towards that. So the main issue currently with that
49:01
would be that the compilers to compile the models into the hardware representation are typically non-free and probably going to remain so in a number of cases. So feel free to push for free compilers for these models to your hardware provider or whatever.
49:21
Any more questions? You mentioned patches for the camera for the ISP. Could you point them to me? Sorry? Could you point me to the patches you mentioned? Do you plan to work on the camera? It's Adam Pigs. That's just for the CSS receiver as far as I'm aware.
49:43
Just not for the ISP. Right, oh sorry, yes, okay. Maybe I went a bit faster with that so it's actually patches on the driver, the some six ICSI driver side to implement things that libcamera expects. So I think you know the one thing I'm talking about. So do you plan to work on the ISP support in libcamera?
50:02
So personally I would be very happy to do so. So we're just looking for someone to fund that effort. So if you know someone with lots of money and interest please come and talk to us. No but seriously I mean I know that people would definitely be interested in that so it's good to spread the word that we are available to do that.
50:21
We just need someone interested and serious about funding this but we would definitely be very happy to do it. So yeah. Okay cool, thank you for a great talk and that's the end of the question. Thank you.