LAVA + OpenQA = Automated, Continuous Full System Testing
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 287 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/57111 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Engineering drawing
00:37
Gamma functionTerm (mathematics)Image resolutionStatistical hypothesis testingKernel (computing)Statistical hypothesis testingSystem programmingAnalytic continuationSoftware maintenanceResultantData managementTerm (mathematics)Observational studyBitProjective planeContent (media)CASE <Informatik>Basis <Mathematik>Open sourceSoftwareKernel (computing)Computer animation
02:32
Open sourceSystem programmingKolmogorov complexitySoftwareVolumeProduct (business)Function (mathematics)Term (mathematics)Self-organizationSoftware maintenanceKernel (computing)TrailRevision controlStatistical hypothesis testingOverhead (computing)Patch (Unix)Carry (arithmetic)Linear regressionComputing platformStatistical hypothesis testingPlanningKernel (computing)SoftwareComponent-based software engineeringMessage passingCASE <Informatik>CausalityMultiplication signVolume (thermodynamics)Computing platformSystem programmingFocus (optics)Stability theoryExpert systemLinear regressionVulnerability (computing)Statistical hypothesis testingOpen sourceTerm (mathematics)Slide ruleRevision controlProduct (business)Context awarenessInternet service providerSelf-organizationOrder (biology)CodecWordMathematicsSoftware bugConfidence intervalComplex (psychology)Connected spaceStatistical hypothesis testingFrame problemLine (geometry)CodeOverhead (computing)QuicksortMechanism designProcess (computing)Projective planeComputer fileSoftware maintenanceLatent heatComputer animation
08:52
System programmingStatistical hypothesis testingStatistical hypothesis testingReduction of orderChannel capacityKernel (computing)Channel capacityVirtual machineElectronic mailing listCategory of beingError messageStatistical hypothesis testingStatistical hypothesis testingEmailSoftwareSystem programmingArithmetic meanLink (knot theory)Revision controlMoment (mathematics)GoogolConfidence intervalTerm (mathematics)Focus (optics)Kernel (computing)Computer animation
10:27
Operations researchSystem programmingDisintegrationComputer hardwareVirtual realityStatistical hypothesis testingRevision controlKernel (computing)Client (computing)Server (computing)Pairwise comparisonSoftwareLatent heatStatistical hypothesis testingMaizeOpen setProcess (computing)Element (mathematics)Computing platformPairwise comparisonStatistical hypothesis testingServer (computing)Centralizer and normalizerPhysicalismOffice suiteKernel (computing)Statistical hypothesis testingMultiplicationBranch (computer science)Instance (computer science)Basis <Mathematik>Configuration spaceDirection (geometry)WhiteboardType theoryContinuous integrationComputer hardwareGraphical user interfaceSoftwareResultantProjective planeNumberConnected spaceSystem programmingDifferent (Kate Ryan album)Surjective functionVirtualizationClient (computing)Computer animationSource code
13:25
Computer hardwareDisintegrationStatistical hypothesis testingUniqueness quantificationComputer-generated imageryDifferent (Kate Ryan album)Repository (publishing)Lie groupMIDIStatistical hypothesis testingPatch (Unix)Gastropod shellNormed vector spaceLinear regressionType theoryComputer hardwareStatistical hypothesis testingStatistical hypothesis testingServer (computing)BitDefault (computer science)Instance (computer science)Process (computing)Medical imagingCASE <Informatik>Multiplication signSet (mathematics)Branch (computer science)Centralizer and normalizerSoftwareLevel (video gaming)MathematicsComputer fileCloningGastropod shellInstallation artProjective planeString (computer science)Repository (publishing)BuildingRevision controlGoodness of fitComputer clusterFile formatSystem programmingResultantINTEGRALVirtual machineComputer animationSource code
17:48
Asynchronous Transfer ModeIntegrated development environmentMedical imagingFunctional (mathematics)Statistical hypothesis testingMereologyTime zoneMatching (graph theory)
18:52
Computer hardwareEmulatorStatistical hypothesis testingStatistical hypothesis testingLinear regressionFeedbackView (database)MereologySoftwareStatistical hypothesis testingComputer hardwareProjective planeStatistical hypothesis testingKernel (computing)Set (mathematics)LaptopClient (computing)Centralizer and normalizerProcess (computing)WhiteboardServer (computing)Instance (computer science)Linear regressionCodeImplementationTelecommunicationFront and back endsAdditionProfil (magazine)Game controllerBootingFunction (mathematics)AreaRevision controlConnected spaceComputer animationProgram flowchart
21:41
Musical ensembleBuildingStatistical hypothesis testingBootingKernel (computing)ResultantStatistical hypothesis testingWhiteboardPairwise comparisonGame controllerBranch (computer science)Process (computing)Structural loadLine (geometry)PasswordStatistical hypothesis testingNumberComputer animation
22:38
Electronic mailing listComputer fontEmailVideo game consoleKernel (computing)Statistical hypothesis testingPairwise comparisonBasis <Mathematik>Revision controlResultantXMLComputer animation
23:11
FeedbackKernel (computing)TouchscreenStatistical hypothesis testingLinear regressionFunctional (mathematics)Moment (mathematics)BenchmarkFunction (mathematics)Computer animation
23:47
FeedbackStatistical hypothesis testingSynchronizationWhiteboardStatistical hypothesis testingRecursive descent parserStatistical hypothesis testingoutputConnected space1 (number)Server (computing)WhiteboardBitKernel (computing)SoftwareControl systemComputer hardwareCentralizer and normalizerRevision controlLinear regressionFront and back endsBasis <Mathematik>VirtualizationMedical imagingRepository (publishing)SynchronizationInstance (computer science)Source codeComputer animation
26:18
Computer virusComputer animationMeeting/Interview
27:15
BitPresentation of a groupLink (knot theory)Kernel (computing)WhiteboardServer (computing)Different (Kate Ryan album)Process (computing)Moment (mathematics)Statistical hypothesis testingScripting languageSoftware frameworkGraphical user interfaceProjective planeComputer hardwareMeeting/InterviewComputer animation
29:44
Statistical hypothesis testingSystem programmingBootingStatistical hypothesis testingComputer animationMeeting/Interview
30:35
Web pageServer (computing)Statistical hypothesis testingConnected spaceType theoryTerm (mathematics)Software bugIsing-ModellStatistical hypothesis testing
32:10
Statistical hypothesis testingCuboidConnected spaceDefault (computer science)WhiteboardComputer hardware1 (number)Kernel (computing)BootingInstallation artComputer animationMeeting/Interview
33:42
InformationStatistical hypothesis testingOnline chatComputer animationMeeting/Interview
34:29
Connected spaceTouch typingCommunications protocolFront and back endsoutputProjective planeDemonHookingStatistical hypothesis testingComputing platformStatistical hypothesis testingMoment (mathematics)Slide ruleNumberLimit (category theory)Software frameworkImplementationVirtualizationStreaming mediaQuicksortBuffer solutionOpen source
37:42
HookingVariable (mathematics)Cycle (graph theory)Confidence intervalWhiteboardSoftwareStatistical hypothesis testingKernel (computing)Absolute valueSystem programmingComputer hardwareFreewareBefehlsprozessorComputer animation
38:48
WhiteboardLinear regressionStatistical hypothesis testingFlow separationKernel (computing)Branch (computer science)Type theoryDeterminismPatch (Unix)SoftwareBootingReduced instruction set computingMeeting/Interview
41:06
Presentation of a groupControl flowMoment (mathematics)Computer animation
41:53
Online chatMeeting/InterviewComputer animation
42:53
Computer animation
Transcript: English(auto-generated)
00:42
Hi everybody and welcome to this talk which is going to be about automated and continuous system testing with Lava and OpenQA. So for some introductions my name is Lauren Soregi, I'm a project manager here at CodeThink and my colleague here who will be giving the talk with me is James Thomas,
01:02
a senior engineer here at CodeThink. And both of us are passionate about open source and we have a lot of experience working in commercial software projects in the automotive industry and in particular we have experience of how the introduction of cutting edge software concepts from open source into the commercial
01:25
industries can help businesses gain a competitive advantage in what they're doing in the markets. So in terms of the contents of this talk, firstly I will introduce long term maintainability and talk a little bit about what this is.
01:41
Then we'll talk about some of the problems that we can see with the current situation and the gap between upstream and downstream. And then we'll talk about how that has led us to thinking about ways in which we can help to solve this and in particular how we can introduce some automated testing to upstream communities and we'll talk about the technologies we use to do that which
02:03
is Lava and OpenQA. Then we'll do a deep dive on some of the case studies in particular where we have actually implemented the work that we're going to talk about, the testing that we've pushed to the GNOME upstream community and also the testing that we're doing of the Linux kernel.
02:24
All of that work is being done in conjunction with upstreams and the results are being posted on a continuous basis to those projects. Then finally we'll talk about the future plans. So what do I mean when I say long term maintainability?
02:41
So I'm going to use this slide to outline some of the home truths and try to outline some context for what we're talking about here. So there are a lot of software systems or devices or platforms that we work with at CodeThink which are intended to function over a long term time frame and that's generally
03:01
considered to be anything that's 10 years or more and all of these systems rely on open source components and as we know software is increasing in complexity all the time. We all demand more features and that means more lines of code and that means more bugs and more testing needed. So in essence the cost and complexity of maintaining such systems will continue to grow
03:25
over time and it will continue to grow exponentially as well and organisations need to be able to handle the increasing complexity and at the same time handle the increasing volume and all the time staying competitive. So in other words businesses need to be prepared to adapt and to accept change and
03:43
remain competitive. So why do we say long term maintainability? Don't we already have long term maintenance? Well yes and no. So typically we see the same thing happening again and again. You'll have an upstream component which is supported for a shorter lifecycle than that
04:02
of your product and when companies are developing their products they will choose a version, integrate that, do all of their testing until the product is developed and they're happy with it and they're happy that the customers are going to be happy with it and then it's released and it's considered stable. And over time, two years later, four years later, whatever it may be, you'll still find
04:24
that the same version is used and basically there is a huge reluctance to upgrade and move on to the next version. If you have to do so, it might be due to a critical security vulnerability. Generally a lot of the time that will be backported from upstream and integrated
04:41
specifically and then pushed to customers via an update which is a lengthy process and quite an expensive process but generally full system upgrades are avoided. So what we have here with this reluctance upgrade is that we have a situation where systems are just kept on life support. We're doing the minimal possible that we can to keep them to be a viable product.
05:04
And this isn't necessarily to cast blame, it's not casting blame on upstreams or it's not casting blame on downstreams that can consume the software. It's just to say that's the situation that we've found. So we have to be able to do better between us. Back to long-term maintainability, so again basically this is the phrase that we've sort
05:25
of developed to think about how we can help to improve and essentially instead of having this fear to upgrade, how we can design systems from the outset to be able to upgrade. And again just using this slide to say some home truths. What we want systems to be able to do is to be as close to upstream as possible and
05:44
integrate all of the advancements and all of that, basically absorb all of that benefit from upstream with the least amount of overhead required. We want systems to have robust deployment mechanisms in place to be able to roll back in case anything ever goes wrong. And we want to have processes in place on the project to make sure that we can repeat
06:02
any builds done at any time. So that in six years time if we need to know what version went into a specific build we can always do that by checking the manifest files. And the focus of this talk in particular is that we want to have robust testing pipelines to make us comfortable to be able to update.
06:21
So just in summary, everybody wants the same thing. Stable software with the latest features for the long term. And to achieve this we need to have confidence both in your software products as a business and confidence in the software that you depend on. So both downstream and upstream. And this has to come from testing.
06:41
So let's hear from the experts. What does upstream say? Well the message is quite clear. Always use the latest component that you possibly can. For example when we had the Meltdown and Spectre issues some years back the message from the kernel community was very clear. Upgrade to the latest kernel and you will make yourself immediately more secure and
07:02
more protected against these issues. Upgrade to the latest kernel. All of that makes complete sense. But in reality is that something that is easy to do? Let's hear from what the downstreams say. Downstreams will say that the kernel is only supported for two years or with the
07:23
case of the LTS kernel, six years. Our product needs to be supported for 10 years. We definitely don't want to maintain our own kernel fork for obvious reasons. That's to be avoided at all costs. And the problem is we're not confident enough to upgrade to the next kernel version. We don't know that it won't cause regressions until we have a large body of testing
07:42
evidence in specific contexts. In particular if you think about the BSP providers they need to have a lot of confidence before they can rightfully claim that they can upgrade to the next kernel version. And then we see this disconnect which is where the problem really lies really.
08:00
Organisations want support for a longer term. Upstreams want people to go to the latest kernel. We don't have the confidence in order to have that ability to upgrade to the latest. And at all times not all organisations are able to commit to the resources that are needed to contribute to that testing maintenance. So we have this slight disconnect.
08:22
Downstreams that need to put more resources into the testing to have that confidence to upgrade and upstreams that need to help with the testing so that we can upgrade. So hence the gap. Testing the software platforms is extremely complex and time consuming and the gap comes from
08:40
this phrase we're using, mind the gap, basically lots of organisations are missing out on the latest features for fear of upgrading and fear of the time consuming manual regression tests required. So what can we do to help? This led us to thinking about how we could implement automated and continuous embedded
09:02
system testing. So our research showed us that a lot of the testing that goes on at the moment especially with the latest kernel versions relies on human beings and is completely manual. So especially, well in particular the stuff that's done in the open at least. You have individuals emailing mailing lists and saying yes I booted this on my laptop,
09:22
it works. So if we want to have stable software over the long term we need to have better testing to give us that confidence. And having automated and continuous testing in a pipeline will allow us to have that confidence. Automation can increase the robustness of tests because we're not relying on any human
09:41
error, it can increase transparency of tests and in reducing the manual efforts we increase our testing capacity automatically because machines can be utilised for longer than human beings can do. We can catch major errors more easily and earlier and eventually increase the focus on more complex testing and this is by no means something that is a new idea.
10:03
People have been saying this for a long time, basically the link here is to an article from Google where they've said that the way to do testing to make sure that systems are secure is not to focus on downstream testing like each supplier may do but to focus
10:20
on testing directly in the upstream, continuously test upstream and give the confidence there. So now we're going to take a quick look at the tooling that we were using to implement some of this in the open. So the first tool here is Lava which describes itself as a continuous integration system
10:44
for deploying operating systems onto physical and virtual hardware for running tests. So this is essentially a giant test orchestration platform. So there is a central Lava server which dispatches jobs to clients. Now these clients can be anything from a PC running QEMU or it can be a Jetson board or
11:08
a Raspberry Pi or any type of physical hardware. So this is actually currently used by the kernel CI project and as you can see here there's a number of jobs available testing various different platforms, testing various
11:22
different branches of the kernel on a commit by commit basis and you can see that for each one of these particular branches they're building there are several different configs that are being built and tested. So this targets the hardware directly. So this could be a board somewhere in someone's office running and communicating
11:43
with the central server and Lava can then dispatch jobs to that particular board. The other tool we're using is OpenQA. Now OpenQA is a platform for testing the software, specifically the graphical user
12:00
interfaces of the software. So the main element of these tests is a screenshot comparison. So a test would consist of a known good screenshot, some definitions of what needs to be matched in the screenshot and then you run the test using a VNC connection usually
12:22
in QEMU which will then take screenshots and compare whether or not this is actually the same and these are called needles in the OpenQA world. Again there was a central server, a central OpenQA server. Now this is similar to Lava. This can be used to view the results of the tests.
12:40
It can be used to edit the needles but it's also responsible for dispatching test jobs to workers and again this is a similar concept to Lava. A worker is a known hardware type. So example, it's a worker that will provide a QEMU instance that runs 64-bit software.
13:01
You have many of these so if you're running multiple tests, OpenQA can find one that's available, dispatch the job and then run it. And this is a very, very good way of actually testing software because the vast majority of users, this is the only way they'll ever use this. They're not going to be hacking around in a serial console or on a terminal.
13:22
They're actually going to be clicking things and looking at stuff. So the first thing we looked at was integrating OpenQA into the Gnome GitLab pipeline. So if you're not familiar with how GitLab does its CI, it's essentially a YAML file
13:41
which defines a set of steps. This could be a build step, this could be a deploy step and we've added a testing step. So after the build is done, we can now manually trigger a OpenQA job to perform a set of tests to make sure there have been no regressions in that particular piece of software.
14:01
Now the way we've done this is slightly different to how OpenQA usually works. So OpenQA by default assumes that the testing hardware is always present. So you would have a machine running QEMU which is always available to take test jobs. We did it in such a way that we wanted to use the existing build hardware for the Gnome project
14:24
of which there is quite a lot. And so we didn't want to have the need to have dedicated testing hardware sat there all the time. So this makes it very, very easy to maintain as well because it's essentially just running a Docker container on a Gnome good server. And the only requirement for the actual workers there is that they do need KBM
14:44
because this is a QEMU job. It would work without it but it would be quite slow. So the format of the testing integration is that we actually cut the tests themselves inside the build repository for the Gnome projects.
15:02
And so they're using build string to build and deploy their systems. So we can just literally put a folder in the repository which contains the tests. So this allows you to link the tests to a very specific version of whatever it is you're building.
15:21
So you can add them, you can remove them, you could add a new test and it will always be associated with that shard. Now the needles themselves, the screenshots, they're kept in a separate repository. This is just generally the way OpenQA wants you to do this. And it makes sense because the needles could, in theory, get quite big.
15:44
You might have a lot of screenshots in there for multiple different versions. And so it's probably best not to pollute the main build repository with them. So when we trigger a job, you can see it in the pipeline here. The first stage of the GitLab pipeline is we build Gnome OS for ARM, for this Ubit and AMD64.
16:09
And that's mandatory, they're mandatory jobs, so you have to run them. Now the second stage is an ISO installer. This generates the CD image which contains the Gnome OS live CD.
16:21
So that's built from an x86 one here. And then the final stage is actually to run a test on that produced ISO image. So when we trigger that test job, the workers clone the needles from that repository, the needles repository, and it also clones it from a specific shell.
16:44
So someone could, in theory, change the needles repository, add some more, test out some new types of tests in there without necessarily having to push these to the master branch of the needles repository. So once we clone that into the working directory, the workers then register
17:03
with the central OpenQA server with a unique machine type. So let's say we had a QEMU instance. This would be QEMU x86-64 to tell OpenQA that this is going to run 64 bit AMD64 jobs and the unique ID, 12345 in this case.
17:22
So this ensures that OpenQA can only dispatch that test job to the worker that has access to the ISO image. So there's no worry that this test job could go to a previously spun up OpenQA worker. So as you can see there, that's the integration of the CI, that's the test job.
17:44
And here's the results. So this is currently live. You can visit it right now. It's openqa.gnome.org. And here you can see all of the previously built and previously tested images. I'll go to all tests here.
18:02
You can see one was run about five hours ago. And you can see the tests are quite simple at the moment. There's not a huge amount, but we run through the install stage. So we ensure that the actual install of the ISO image is completely correct. And you can see with this UI, you can slide this and see the percentage match.
18:22
And that's looking very good. And then we go right through to the GNOME first setup. So we can check the time zones work. We can check users can be added. We can check all of the functionality that you'd want to test as part of an initial setup. And then finally, we actually get to the desktop there.
18:44
And so the first test is did we actually get to the desktop? And as you can see, that's working 100 percent. So, yeah, as I just mentioned, it's available to test right now on openqa.gnome.org.
19:04
So the second part of this project was trying to then take openQA's testing of the software side of it and put that on physical hardware. So this is why we use Lava in this case, because we can leverage the existing tests
19:23
that exist as part of the kernel CI and the existing board profiles and all of the actual existing tests that they do for the kernel side. We can take that and then run additional tests with openQA. Now, the Lava tests that currently exist in the kernel CI, they're very thorough
19:42
and they do actually go onto the board and you boot it. But they're mostly just checking the serial output to make sure that things are booted. So just the kernel boot is a very good test for a new version of the Linux kernel. Does it log in? Can we see the output? Is that expected?
20:01
So that's all very well and good, but there's a huge area there that isn't actually captured by those tests. So running these openQA tests in a graphical environment, you can now test for things like, is there any regressions in the graphics driver? Does certain software work? Does the network up in a visual way?
20:22
Can we change the settings of that? So the access for this is we have a laptop which is connected to a Raspberry Pi board. And that's running the Lava client, which communicates with the central Lava server. We have an openQA client on that Raspberry Pi, which is listening and dispatching jobs.
20:45
And then openQA connects that over a VNC connection. So you can then control the graphical output, you can send keystrokes, you can actually control this and then get the screenshots. We also additionally deploy openQA on that control laptop. QEMU, sorry, on that control laptop.
21:02
So we can also run tests in QEMU to test even more. And again, this is now live. This is being deployed on our own instance there. We hopefully will be able to submit some new code to the openQA
21:26
because we've written a back end that allows for a communication with a Raspberry Pi board. But that implementation is actually quite generic. So it really should work on multiple different devices. But at the moment, it's only been tested on a Raspberry Pi.
21:42
And so you can see the results here. You can see the Lava job, which takes the kernel, flashes it to the board and then boots it. And then you can see the results of the openQA tests, which then boot Ubuntu and perform some graphical comparisons. And you can see we've got a number of branches out there.
22:00
We've got some long-term support kernels. We've got the master branch. We've got the latest release branch. And we do that for multiple different devices. So we have a Raspberry Pi job that's running on the board and is also testing the main line kernel on there. So we check, let's check the 419. You can see the results there, they're very good, it's all green.
22:24
And you can see the results of the tests here. So you can see this is a QA new job. This is running on the control laptop. And we installed the kernel. We checked the boot load is correct. We log in, make sure we can enter the password.
22:42
And additionally, the openQA tests don't have to be screenshot tests. So here we're running a test over the SSH console to make sure that the version of the kernel is correct. You can still get a log in and all of that. And that can be done on a text comparison basis.
23:01
So here we go. You can see the results of it actually putting the new kernel on. And then we reboot it. Let's see, it's performing the same tests. And then did we actually get to a graphical screen is one of the tests. And you can see there that the needle is matching 100%. As previously stated, they are basic tests at the moment.
23:22
But we do hope to add more. But in theory, we could actually add a lot more functionality to this. So we could run GL benchmarks and test the output of the frames per second to make sure there's been no huge regressions in any of the kernel side.
23:43
And we have received some positive feedback already. As I said, it's being used. Linus is supporting the more tests the merrier. So it's good to get positive feedback about that actually being used.
24:01
So what next? We need more tests. As you can see, the GNU ones need to be fleshed out a bit more. So contributions are very welcome there. We would like to connect our OBQA instance to the main kernel CI, to the main kernel Lava CI, so we can start performing these tests on many more devices.
24:22
We can hopefully test as many of the boards that are using OBQA and the graphical tests that we have there. Some features that need to be added to OBQA itself, namely the test needle synchronization between the workers and the central OBQA server.
24:41
So at the moment, whilst the workers can clone the needles that it needs for its test by giving it a Git repository and a SHA, this does not work on the central server. So the central OBQA server is currently synchronized with a cron job, I believe, that runs every minute to pull the needles into a specific folder.
25:00
So adding the ability to OBQA to target the tests in the SHA per test would be very much needed. We'd also like to add some of this board deployment work to the GitLab pipelines for GNOME. So on top of actually testing the ISO image in QEMU,
25:23
we would like to then add a pipeline stage, which can then flash that to a piece of hardware. So we're thinking about using a RISC-V board for that to make sure there's no regressions there and make sure that the versions of the software actually still keep on working. And maybe that won't be done on a per-commit basis.
25:41
Maybe that could be done on a release basis, but there's still work to do there. And finally, more backends. OpenQA is very geared towards using VNC as the actual control mechanism. It would be nice to add an IEP backend for that. It would be nice to add something way more generic.
26:03
So examples of virtual kernel input device that you could trigger remotely using OpenQA, which would open it up to many more devices that might not perhaps have the ability to have a VNC connection to it. And that's it. Thank you for listening.
27:00
Hello, hello. I cannot hear you for some reason. Is this working now? Oh yes, we can hear you. Awesome. Okay, thank you for your talk. And we do have some questions.
27:23
The first one is can you please elaborate a little bit more about how Lava links into OpenQA? I think the presentation kind of answered this, but maybe you have something else to add. Yeah, so the use of Lava there
27:41
was to use both the existing kernel CI, framework tests in kernels and boards. So the Lava orchestration of board deployments and deploying kernels, et cetera, is pretty much what you have from the kernel CI project.
28:03
So we do those tests, we deploy to board using the existing Lava tests that were there. And then once those tests complete, once those have been booted, we then run the OpenQA client,
28:21
which is a script that you can run on the actual target hardware, to then inform the OpenQA server that we're now running over QA tests. So it's essentially using two different orchestration frameworks there,
28:41
but we use the existing kernel CI Lava tests to do the tests that already exist and then inform OpenQA to then start running GUI tests on a target hardware. And so that's actually,
29:01
you can see that within the GNOME CI, where once the Lava's booted the board and run its existing tests, it actually then calls an OpenQA script to then start running the jobs on the board.
29:22
So it's essentially a two-step process. We're using Lava because that's what currently exists in the kernel CI. But pretty much all of the tests that I've done using Lava could be done in OpenQA. We're just using Lava at the moment because that's what currently exists.
29:43
Okay. And so the next question is about the Lava. Is there any test orchestration support for ESP32 WROM boards? I have no idea, sorry. So the Lava side of this was not something I actually did,
30:01
so I remember the wrong person told us about that. Okay. So next question is, next to questions about OpenQA. First one is can you comment on the speed of execution of OpenQA tests? Because on my own home lab, they are too slow, not too fast, says Eric.
30:23
And the follow-up on this is, we are talking about execution of the testing steps, not the booting of the system. So in terms of feedback, I had the opposite problem.
30:42
So there's been several tests that we've actually had to slow down because they were executing too fast. I would imagine OpenQA is limited by the connection that you actually have over VNC to whatever it is you're trying to test.
31:02
For things like sending input, and this was the question that was answered within the ISE channel on Freenode on the Vera, sorry, the SUSE channel there.
31:20
OpenQA can actually be useful to exposing bugs where you do something too quickly. So sending input, for example, where you type something in too quickly was my experience of that. I certainly have the opposite use of that.
31:43
Perhaps it won't be limited in the speed of your connection via VNC to whatever it is that you're testing. So if you're testing something on a remote server on the other side of the world, then it might be an issue. But personally, in our testing,
32:00
never had an issue with any type of speed of execution of tests. It was the exact opposite. They were going too fast for us. Okay, and we do have a follow-up on this. The testing is then inside of QEMU and KVM box within the GNOME desktop. And so maybe that's not ideal.
32:22
Can you share what's your setup exactly? So we have two setups. The lava-based one, which is testing the kernel performance using the kernel CI, that's actually on hardware. So that's booting on a Raspberry Pi.
32:41
I think we've extended it to a RISC-V board. And so that's actually on hardware. So we do the deployment using lava, we run the initial tests, and then we boot some Ubuntu desktop with VNC installed.
33:01
But the default for other QA itself is to use QEMU. QEMU and utilizing QEMU's VNC connection to connect to the guests there. So that's the setup in the GNOME pipeline.
33:27
QEMU, that's the default for OpenQA in general. So certainly never had a problem with that. The lava-based ones are actually on raw hardware there.
33:41
Okay, thank you. So next question is again about OpenQAs. What are you missing from the OpenQA perspective? Don't quite get the question.
34:01
Is there any additional? There is additional information. I'm not sure if that was directed towards you. But yeah, that's what we have in the chat. Okay, let me ask it in another way. Are you, as a tester and person who's using OpenQA,
34:21
is there something that's missing for you or some features that you would like to see or are you happy with what you have right now? Yes, okay. So I think OpenQA is working very, very well as long as what you're testing provides a VNC backend. You have to connect to what it is you're testing over VNC.
34:47
And so for things like running virtualized guests in QEMU, where QEMU itself will provide the VNC connection to that, that's not really an issue. But let's say we were trying to use this
35:02
to test some sort of embedded automotive platform or what have you, VNC might not readily be available. So the limited amount of backends available to connect to things
35:22
is my only real gripe with OpenQA. And this is absolutely something that I think can be improved. The actual implementation of it is such that you just need a backend that can get the frame buffer
35:43
and a backend that can receive touch inputs to then relate to whatever it is you're testing. So you could do this incredibly generically, right? It could be some daemon that's existing on the hardware under test. As long as that can grab the KMS buffer
36:01
and EV dev, whatever it is that you're trying to forward on to the UI, then as long as you've written a backend for that, it would work. A colleague of mine, and I believe we're actually about to open source it, has written a framework to be able to connect to
36:25
the Google Chromium DevTools protocol. So this is a framework that can receive a stream and then send inputs over the DevTools protocol. So ready to have an embedded project that's using a Chromium based UI,
36:41
we could hook into that in the future. But that is my only limitation I have at the moment with OpenQA is that it is very much bound by VNC. So you can't even connect to Western, for example. That's the biggest grab I have, but I think it just needs more work.
37:02
And I mentioned on the last slide is that I want to do, and when we want to do more work on actually expanding the number of backends so that we can actually start using this on more platforms. And doing it in such a generic fashion,
37:22
I think is the way forward. So then I can use it on any project that I'm working on in the future. But yeah, that's the biggest missing feature from OpenQA is the fact that it has to be reliant on VNC at the moment.
37:41
Yeah, okay, thank you. Let's see, we do have six more minutes for questions and I don't see the audience asking more questions. So let me, in the beginning of this presentation, you talk about, you know, so there are a few things that you touch upon
38:01
and maybe that's not your focus, but from what I see it's, you do want to push a lot of testing upstream to be happening in the upstream community so that we know the software is stable and then we can kind of benefit of the latest features and everything and we can upgrade with more confidence. But then you also need a lot of variability in hardware
38:24
to be available upstream and you do want to hook up your system with kernel CI to make whatever you have available. But is this gonna be enough? Like, is there, do you think we should entice more people to connect to those various systems,
38:42
connect the boards that they have and kind of contribute the free time, the free CPU cycles that they have? Absolutely. So more tests is always better. This is already being used within the Gnome pipeline on GitLab.
39:01
So it's being used there and it's being used independently on a kernel-based CI. So those tests form what has been currently run on RISC-V on a sci-fi board and a Raspberry Pi. They're the exact same tests that could run on a Jetson
39:23
or they could, you know, they can run on any board. It's basically just here's the OS, so you're going to boot this and then run the tests. So the tests themselves are completely generic. So if more people would like to start using this
39:41
and contributing even more tests to that pipeline, then that would be very much welcomed. And they actually have yielded results already. So the person who says of the Lava pipeline on the Raspberry Pi and RISC boards has already detected issues and submitted patches to the kernel
40:02
based on regressions that he observed testing the master branch of the kernel. Because if the test is the same and the software that's running on it is the same, then you've detected a regression in the GPU driver, for example, if something stopped working or the test fails after that.
40:21
So they are actually yielding results already, but the tests themselves, as long as you have a Lava deployment to a board, they should be completely non-hardware deterministic. Had to be specific,
40:41
they should absolutely run on any type of board. So if anyone else wants to come and join this and start using them, that would be very much welcome. I think the actual goal is to try and get this into the kernel CI itself, rather than having it as some separate entity
41:02
that's by the side of it. Okay, okay, yeah. Awesome idea. Thank you very much. Yeah, we do have a few more minutes, three more minutes to be exact. I'm not seeing other questions for the moment.
41:26
Okay. Yeah, so we are going to wait to see if some other questions appear, but for anybody listening to us, we do have a small coffee break coming up before we continue with several lightning talks.
41:43
And then after that is the last presentation for the day. I probably should have said this at the start of the Q&A, but apologies for the other person
42:01
who's supposed to be speaking here. He had an emergency, so he couldn't make it today. Yeah, no problem. Okay, yes, we are saying thank you for the talk already in the chat. So I presume there will be no more questions for you.
42:21
Thank you very much. We are going to still be in this dedicated lava and OpenQA chat room. If you don't wanna catch up with James to ask more follow-up questions. So you're welcome to do that. The room will be open to the general audience
42:40
in less than five minutes. And if also then both will send notifications to the main Dev Room chat. Yeah, so thank you very much. Okay, thank you, bye-bye.