V4L2: A Status Update
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | ||
| Autor | ||
| Lizenz | CC-Namensnennung 2.0 Belgien: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/44623 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
FOSDEM 2019471 / 561
1
9
10
15
18
19
23
24
27
29
31
33
34
35
38
39
40
43
47
49
52
53
54
55
58
59
60
63
65
67
69
70
78
80
82
87
93
95
97
102
103
104
107
110
111
114
116
118
120
122
123
126
127
131
133
136
137
139
141
142
148
153
155
157
159
163
164
168
169
170
171
172
173
174
181
183
185
187
188
193
196
197
198
199
200
201
205
207
208
209
211
213
214
218
221
223
224
226
230
232
234
235
236
244
248
250
251
252
253
255
256
257
262
263
264
268
269
271
274
275
276
278
280
281
283
284
288
289
290
293
294
296
297
300
301
304
309
311
312
313
314
315
317
318
321
322
327
332
333
334
335
336
337
338
339
340
343
345
346
352
353
355
356
357
359
360
362
369
370
373
374
375
376
377
378
383
384
387
388
389
390
391
393
394
395
396
406
408
409
412
413
414
415
419
420
425
426
431
432
433
434
435
436
438
439
440
441
445
446
447
448
453
455
457
459
466
467
471
473
474
475
476
479
480
484
485
486
489
491
492
496
499
500
502
505
507
508
512
515
517
518
529
531
533
534
535
536
539
540
546
550
551
552
553
554
555
557
558
559
560
561
00:00
Streaming <Kommunikationstechnik>Auflösung <Mathematik>GoogolCodecInformationSoftwaretestCodeSelbst organisierendes SystemVirtuelle RealitätDruckertreiberFramework <Informatik>System-on-ChipKnotenmengeObjekt <Kategorie>Puffer <Netzplantechnik>Syntaktische AnalyseE-MailHardwareCodecVideokonferenzProzess <Informatik>Spannweite <Stochastik>KettenkomplexMultiplikationsoperatorSoftwaretestWeg <Topologie>VirtualisierungHumanoider RoboterQuick-Sortt-TestProjektive EbeneMAPVersionsverwaltungEinsMinkowski-MetrikKernel <Informatik>Ordnung <Mathematik>EmulatorUmwandlungsenthalpieOpen SourceKartesische KoordinatenTypentheorieInformationStrömungsrichtungDruckertreiberSchaltnetzAggregatzustandRahmenproblemPunktSpeichermodellKonfigurationsraumProgrammierungPuffer <Netzplantechnik>Komplex <Algebra>HypermediaSelbst organisierendes SystemTopologieAuflösung <Mathematik>BitSoftwareGamecontrollerCASE <Informatik>Framework <Informatik>YouTubeExtreme programmingMathematikKonfiguration <Informatik>TermIntegriertes InformationssystemTouchscreenCodeRechter WinkelTreiber <Programm>Cookie <Internet>Higgs-MechanismusVerband <Mathematik>VollständigkeitSpieltheorieGeradeBasis <Mathematik>p-BlockEinfach zusammenhängender RaumNichtlinearer OperatorBildverstehenDateiformatObjekt <Kategorie>VolumenmodellGruppenoperationGraphische BenutzeroberflächeMixed RealityÜberlagerung <Mathematik>LoopMehrwertnetzMeterDimensionsanalyseFaktor <Algebra>Stabilitätstheorie <Logik>QuadratzahlWeb SiteQuaderDämpfungSchnelltasteFigurierte ZahlComputeranimation
08:14
Funktion <Mathematik>PufferspeicherWarteschlangeInformationEreignishorizontPuffer <Netzplantechnik>Motion CapturingSoftwareVideokonferenzROM <Informatik>EmulatorProzess <Informatik>Analog-Digital-UmsetzerKomplex <Algebra>CodecVirtuelle RealitätDruckertreiberSoftwaretestElektronische PublikationRahmenproblemParametersystemVideokonferenzPuffer <Netzplantechnik>Codierung <Programmierung>GamecontrollerHardwarePunktSichtenkonzeptGrenzschichtablösungWarteschlangeFunktion <Mathematik>Mailing-ListeCodecMotion CapturingZeitstempelDifferenteHalbleiterspeicherAggregatzustandDatenfeldDigital Rights ManagementCASE <Informatik>BitMereologieFahne <Mathematik>ResultanteTV-KarteDruckertreiberEin-AusgabeKontinuierliche IntegrationOrdnung <Mathematik>InformationSoftwaretestKomplex <Algebra>Treiber <Programm>HypermediaProgram SlicingDatenstrukturObjekt <Kategorie>IdentifizierbarkeitEreignishorizontE-MailRichtungMinkowski-MetrikStochastische AbhängigkeitKonfigurationsraumBetriebsmittelverwaltungWürfelMultiplikationsoperatorCanadian Mathematical SocietyVektorraumCodierungFormation <Mathematik>DatenkompressionCodeFatou-MengeSoftwareBenutzerbeteiligungWiderspruchsfreiheitOffice-PaketWeltformelFreewareEinsEgo-ShooterAbstimmung <Frequenz>Natürliche ZahlCaptchaDreiecksfreier GraphGRASS <Programm>Profil <Aerodynamik>BEEPGebäude <Mathematik>IntegralApp <Programm>Digitale PhotographieComputeranimationVorlesung/Konferenz
16:20
Lineare RegressionSkriptspracheSoftwaretestKernel <Informatik>Treiber <Programm>Notepad-ComputerVirtuelle RealitätGoogolHypermediaSpeicherabzugDruckertreiberCodecVersionsverwaltungDokumentenserverHilfsprogrammOpen SourceE-MailModul <Datentyp>Wurzel <Mathematik>Motion CapturingRahmenproblemVideokonferenzMAPWidgetStandardabweichungDateiformatEin-AusgabePufferspeicherSoftware RadioPuffer <Netzplantechnik>Fortsetzung <Mathematik>Total <Mathematik>EindringerkennungAdressraumMathematische LogikMengentheoretische TopologieTypentheorieDatentypBenutzerprofilPASS <Programm>MaßstabHilfesystemHalbleiterspeicherCASE <Informatik>SoftwaretestElektronische PublikationKomplex <Algebra>Virtuelle MaschinePuffer <Netzplantechnik>DruckertreiberHypermediaUmwandlungsenthalpiePatch <Software>Kernel <Informatik>BimodulMAPAusnahmebehandlungDemo <Programm>BitrateEinsTreiber <Programm>Grundsätze ordnungsmäßiger DatenverarbeitungFuzzy-LogikSkriptspracheHilfsprogrammGrenzschichtablösungVirtualisierungVerkehrsinformationDatenstrukturCodecWurzel <Mathematik>DokumentenserverComputerarchitekturMereologieSoftwareentwicklerUmsetzung <Informatik>Lineare RegressionMomentenproblemVerzweigendes ProgrammDateiformatBus <Informatik>DatenfeldVersionsverwaltungMultiplikationsoperatorTrennschärfe <Statistik>Codierung <Programmierung>Prozess <Informatik>Mailing-ListeProjektive EbeneLastPixelVideokonferenzMaßerweiterungGoogolGruppenoperationKettenkomplexSchnittmengeInhalt <Mathematik>FitnessfunktionStabDruckverlaufPunktVollständigkeitHardwareSichtenkonzeptExistenzsatzZeitstempelData MiningBildverstehenQuaderTOEDigitales ZertifikatStreaming <Kommunikationstechnik>Web logCodeURLBeobachtungsstudieSpieltheorieLesen <Datenverarbeitung>RelativitätstheorieZirkel <Instrument>BitBildgebendes VerfahrenVierzigGeradeRechter WinkelComputeranimation
24:27
Weg <Topologie>NormalvektorInformationCodeCASE <Informatik>Vorlesung/Konferenz
25:07
PunktwolkeGibbs-VerteilungVorlesung/KonferenzComputeranimation
Transkript: English(automatisch erzeugt)
00:05
Welcome. So, after my bonus talk earlier today about CEC, now I can talk a little bit about video of Linux, roughly what happened in the past year. I don't have a lot of time, so I am sticking to two main pieces that, in my biased opinion,
00:26
have been the most interesting and most important, and that's hardware codec support and testing. Let me first start. There are two types of hardware codecs. We call them stateful and stateless.
00:40
Stateful codecs have been supported for quite some time now, and basically you just feed them, say, an ACE 264 bitstream, and they do everything, and out come the decoder frames. So they keep track, the hardware will keep track of the reference frames, they will keep track of the state, and they are really simple to implement, because all the complexity is in the hardware.
01:05
A problem with that is, that takes a lot of, that moves the complexity towards the hardware, and of course hardware designers don't like that, so they've been pushing back, and what they basically did was more like accelerators. So you give the frame to decode, you give the reference frame associated with it, or frames, whatever,
01:27
you basically give the whole state together with what you want to decode, and the hardware no longer just takes that, processes it, returns it, and it no longer remembers any state. Those have not been supported until very recently.
01:43
But first stateful codecs, as I said, we support quite a fairly big range of them already. What was missing was a precise specification how to use the video phrenics API for that, particularly in corner cases like what should happen when you do a seek in the middle of a stream,
02:01
or you have a midstream resolution change, and Google, specifically the Chromium and Chrome OS team, they have been working on writing all those details down. Incredibly useful, it is not yet merged, we are going through hopefully at least one more version I think.
02:23
But there's a good chance that we'll get into the 5.1 kernel, or is it 5.2 by now, 5.1, where are we? In order to test these things, so testing with actual hardware codecs is difficult because you need the hardware.
02:40
It would be nice if you had a hardware emulator. We have that for various other devices, types of devices as well, I'll come to that in the next section, but we also have ViCodec, it's a virtual codec driver, it implements in software a codec, not any of the standard ones due to IP issues.
03:01
It's a very simple codec that actually was used decades ago by NASA in some of the space probes, and it was a student who did a project on that, who coded this, made the codec itself, and I turned it into a driver. But it is not compliant to the specification that we're working on.
03:23
And Outreachy is an organization that provides money for people who want to get involved into kernel programming for a project of three months to work on particular issues. Not just kernel, it can be any open source project. I'm co-mentoring together with Helen Koike from Collabora, Daphne Hirschfeld,
03:44
who is working on making the virtual codec compliant to the spec. It's a moving target since the spec is not fully specified yet, so that makes it a bit interesting. And I've started work, so this is slightly out of date, by a few days, I've started work on adding compliance tests as well for stateless codecs, stateful codecs.
04:05
I already, while doing that, I already saw some issues with the specification that are, there are certain exceptions and they're already difficult to program, so it will probably change a little bit. But this is great to have this combination of documentation, compliance tests,
04:23
and basically an emulator for a codec to figure out these issues before you make them available in MadeLine. The other big one is stateless codecs. There was no hardware support in the kernel for these devices.
04:41
It was, however, used already for three, four years by Chrome OS because they had an early version of what is now called Request API that they made many years ago, and they've been using that on Chromebooks. One thing that they're also doing is writing documentation on how to use it.
05:02
That's a bit earlier, stateful codec documentation is further advanced than this one. Daphna is also working on turning the Vi codec driver into a stateless codec so we can test all this. She just started on that this month, or last week actually.
05:25
That needs to be compliance tested as well. But the big thing here is Request API. It was merged in 4.20 together with the first stateless decoder driver. It's in staging. The Request API is in mainline, but the driver is in staging
05:43
because there are still a few things that are being developed. Work has actually been done already by various people to adapt it to the rockchip codecs. I don't know what the state is, but someone started on work for the Tegra T-goder.
06:02
They're all both stateless codecs. There's also work on the way for H.264 and HEVC. Some people are already looking at VP8 and VP9 or at least thinking about it. There's a lot of interest in these because the hardware that uses these codecs
06:26
they have been blocked for being mainlineed for quite some time. You couldn't get the drivers in because the Request API was missing. What the Request API basically does is that it creates... The name Request actually comes, I believe, from Android Camera Hall.
06:42
You make an opaque object, you put a buffer into it and all sorts of formatting data configuration and you give it to the hardware. So you have basically shorthand for per-frame configuration. The hardware processes it and at some point the request is completed.
07:01
You can dequeue the buffers and get any updated state information out of it. This is needed not just for stateless codecs, although they are the first ones that start to use it, but also by complex video pipelines. The Co-post talked earlier today about libCamera.
07:20
It mentioned that they want to use it for Android as well and then you need this. This is basically per-frame configuration, which is a requirement from Android. The current Request API framework is... It probably gives between 80 and 85% of what is necessary for that type of application. So more work will be done in order to support that.
07:42
It's not quite there yet, but a very large chunk is finished. So how do we use it for stateless codec? We have two device nodes. The video node, that was always there, but now you also need to use a media controller node.
08:03
Media controller, very quickly, it basically shows the internal topology of your hardware. It is primarily meant for complex video pipelines, but we also use it for stateless codecs. The reason we use this, instead of just the video node, is exactly because the Request API will also be used for the complex video, complex camera pipelines.
08:26
So this keeps it consistent. You have the same method for both. If you have a stateless codec, that means that user space is responsible for parsing the bitstream. No longer the hardware.
08:42
So it parses out the headers, it figures out the state of the buffer to be decoded. I'm primarily talking about decoders. Encoders is very similar, just the other direction. But I'll make it easy, I'll just talk about decoders here.
09:04
So what you need to do is, first of all, you need to create a request object. And usually you make one for every buffer that you allocated, but it's really up to you. Depends a little bit on the codec that you're using and other factors. So use the media file descriptor of the media device node.
09:24
You allocate the request and you get the file descriptor back. It's just an opaque handle to the request object. You always refer to request objects using that. So it's a decoder, right? So you have a buffer with a to-be-decoded frame.
09:44
And decoders are implemented as a memory-to-memory device. So you have things that you put into the hardware and you get buffers back. So you actually have two buffer queues. One is called output, one is capture. This is very confusing. The point of view is the software.
10:01
So an output queue means that you give memory to the hardware. A capture queue means that the hardware fills in a buffer and gives it back. So for the decoder, the compressed buffers go as an output queue and the decompressed buffers come in in a capture queue. For the output buffer, so it's a codec, so you always start with the output.
10:26
You say it's a video output buffer. You have a request file descriptor. So that means that you don't queue it directly to the driver, you queue it in a request. So it doesn't go to the driver yet. It's stored in the request.
10:43
You have to set a flag to indicate that as well. And there is a timestamp that you can set. Now for memory-to-memory devices, timestamps are not used by the hardware. They are copied from the output buffer into the capture buffer. That's all that is done. Timestamps, basically you are completely free to stick in whatever data you want.
11:04
And in the case of codecs, they are used to basically name buffers. It's used as a tag. So the problem is you have reference frames for a decoder. A decoder, when you have iframes, they are decoded all by themselves.
11:22
And then you have PMB frames, and they are basically the difference between a reference frame, an earlier iframe, and what it should be. So they are a lot smaller because of that. That means that a stateless decoder needs to basically get two buffers. This is more complicated with H.264 and more complicated codecs. You can have multiple references.
11:45
Let's keep it simple. You have the buffer that contains the P frame, the difference, and you have another buffer somewhere that has already been decoded that contains the iframe, the reference frame. The decoder needs both. And the tag is what you use to later, after this frame has been decoded, the decoded frame that you get back has that tag as well.
12:08
It's copied from the output to the capture once decoded. So it gives you a unique identifier in the future of this decoded buffer.
12:22
I'll come back to that in the next slide, but the last bit is that you queue the buffer into the request. The buffer is now in the request. The next step is to add the configuration, the state, to that request. For MPEG-2, there is a slice params control.
12:41
There's basically a fancy structure with data that you can set in your application, and it can be stored in the request and read out by the driver. There are actually two controls related to MPEG-2, but no room on the slide. It's too complicated, so I just addressed that one. We're using the control API.
13:01
Again, you need to specify, where do I do that? Yeah, here. This is the request file descriptor. This is a parameter that tells you that the control shouldn't be applied directly, but should be stored in a request. Here you set it. And the critical bit is actually, there are a lot more parameters, but I'm trying to concentrate on the important parts.
13:23
The backwards reference timestamp, so that is the one, the field that refers to the reference frame that this buffer is the difference of, that you have to refer to. And that's been the earlier timestamp of an already decoded buffer.
13:42
So you reuse the timestamps in order to refer to it. So this is the output part. Then you queue up capture buffers, because the result has to go somewhere. Capture buffers are not parts of a request. They're just independent buffer queue.
14:04
You put buffers in, the decoded result gets put in there, and you get them back. And the final one is that you queue the request. So now the request object is queued to the buffer. The buffer has all the information. The driver, sorry, is queued to the driver.
14:24
The driver has the buffer, has the configuration, the state, and it can start to decode. You can wait using select for an event from the request object. When an event arrives, that means that the request has been completed.
14:40
All the data, the buffer is in a validated state. The parameters, configuration is validated and updated, and you can read it out. And finally you need to dequeue the buffers. The output buffer, that's now can be recycled.
15:01
And the capture buffer containing the decoded frame. So when you dequeue this buffer, the timestamp in that buffer was a copy of the timestamp that you set in the output buffer. And this timestamp can always be used to refer to that buffer.
15:21
If you need to do that in a following p-frame, you can refer to that capture buffer through the timestamp. I concentrate on this because it's relatively complicated and easy to mess up. Not to mention that it took a very long discussion on the mailing list to come to a way that works and that everybody agreed with.
15:46
Testing. So video is complicated. Just like in the DRM system, you had several talks about continuous integration and testing. The same thing is true for video input and for hardware codecs.
16:05
We have already quite a few virtual drivers. Vivid, that's the oldest one. It emulates a webcam, HDMI receiver, S-video, even a TV tuner. Also radio, software-defined receiver, even CEC is being emulated there.
16:24
ViM2M, that emulates a simple memory-to-memory processing device. There are some upcoming patches being made by Mauro that improve it quite a bit. It basically emulates a scaler or a pixel format converter. ViMC emulates a complex camera pipeline.
16:42
There too there are improvements planned. And then there's the latest one, there's the ViCodec, the virtual codec driver to emulate codec and encoder. Encoder and decoder, both stateful and stateless. We had recently several, let's say, embarrassing regressions when we pushed some patches to the kernel.
17:11
That could have been easily caught if we had some regression tests. We have a compliance test that has continuously improved.
17:23
And what we basically wanted to do is to create a media, a script, that would just run the compliance test for all these features.
17:50
Quality drivers, again because of the complexity, it's difficult to test your driver. And these compliance tests, they test all these corner cases. Make sure that all the fields that you are supposed to return in structures that are actually properly filled in and you didn't forget anything.
18:07
It's actually more strict than the specification itself because it checks what should be done. And sometimes the specification is more relaxed due to legacy drivers.
18:20
But this is really meant for new drivers, so new drivers should really do the best quality that you can do. It's also very useful, as I mentioned, for when you're developing documentation and making a new API to also write a compliance test. Because if anything is not quite right, you will very quickly figure it out.
18:44
Fairly new is that there is a fuzzing checker from Google and several drivers have been added. These virtual drivers have been added there. In the beginning we got a whole pile of book reports out of it, but about 80% have been fixed.
19:02
For the remaining 15% we have patches that need some more attention and occasionally I think there's one or two new ones that appeared. I haven't looked at that yet. So this has been very useful. I'll do the resources and then have a very quick demo of the test scripts.
19:25
API documentation. There is a version that I maintain that contains the Google specification for codecs. Since it's not mainlined yet, I built it myself. For easier reading, easier to read than the patches themselves. Git repository.
19:41
For the Cedrus staging driver, there is a test utility to test it from Bootlin. P4UTILs that contains a compliance test. Mainly it's the IRC. We're always looking for volunteers to improve the virtual drivers.
20:03
Since they're virtual, you don't need special hardware, so you can run it on any PC or virtual machine or whatever. It's quite an extensive list of projects from just a few days to a few weeks. So if you're interested, let me know.
20:27
So I have a little script. You have to run it as root. This tests all the media drivers. So first it unloads all the existing media modules and then loads the virtual drivers.
20:40
Hopefully I can avoid that in the future. The main reason is that the video numbering, video zero, always has to be the same device at this moment. So that's why I need to clean it. And then it's running vFrog compliance. It's testing all the various streaming modes. It's testing it with blocking mode, with unblock mode, select.
21:08
Testing DMA buff as well, single-planer, multi-planer. And it's doing about 1,700, oh it's pretty quick actually. It doesn't take too long. So there's 1,700 plus tests.
21:23
And this is not yet including vi-codec. It is currently, there are no failures at the moment. This is just the latest kernel, master branch from us. There are a bunch of warnings. I know what those are. Someone is working on at least a few of them. So this should really be zero as well.
21:42
It will be, I promise. And I've done this earlier. So this is the full test. This includes CEC. You see that up there, CEC test. But CEC test, as I mentioned in my talk earlier today,
22:01
CEC bus is 40 bytes per second. It's extremely slow. And if it emulates CEC, including the speed or lack of it, so doing the CEC test take a fairly large, I think this whole test probably takes about five to ten minutes. I haven't timed it. But you can see that it's close to 2,000 tests.
22:23
I hope that this will either be included in kernel CI and or I'm doing a daily build of our latest master, making sure that it compiles for a large set of architectures. I would really like to, at the end, do a run on the virtual machine to test whether everything is okay.
22:43
But I need some time to figure out how to do that. So with that, questions?
23:01
That's one year of Linux development. Well, at least part of it in 20 minutes. No questions? Everything clear? Ah, look, thank God. We're using the timestamping for referencing buffers, like how we manage the memory for buffers.
23:23
Wouldn't be the freedom. It's right. So memory, usually you allocate memory beforehand. No, let me rephrase it. You have a request buffer step that buffers are created. The actual memory for a buffer can come from outside,
23:42
but internal structures for the buffers are created. And basically those buffers stay mapped once they're queued. It checks if it knows about the buffer already, the buffer memory. If not, it will map it. If, it stays mapped, the memory, until you explicitly free all the buffers.
24:06
So internally you always know that given a timestamp, it belongs to that memory, and you also are guaranteed that that memory stays. The only exception would be if you completely free all the buffers, but then you stop streaming anyway.
24:22
Or if, and that would be the case for, for example, a DMA buff, a buffer, internal buffer structure, gets a new DMA buff file descriptor. And in that case you actually release the internal information. You disconnect it. So we keep track of that internally.
24:47
This is for request API, yes? Vivid and ViM2M and ViCodec is being worked on.
25:01
No more questions? Okay. Thank you.