Let openQA test you own stuff
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 63 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/54564 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Different (Kate Ryan album)Computer animation
00:22
BitOpen setMetropolitan area networkGoodness of fitDifferent (Kate Ryan album)
01:12
Process (computing)Statistical hypothesis testingMatching (graph theory)SoftwareStatistical hypothesis testingDevice driverMiniDiscServer (computing)Broadcast programmingSoftware repositoryKey (cryptography)Video game consoleInstance (computer science)Virtual machineProduct (business)MultiplicationDevice driverPhysical systemPoint (geometry)Statistical hypothesis testingMiniDiscOrder (biology)Statistical hypothesis testingMedical imagingInstance (computer science)File Transfer ProtocolProcess (computing)MathematicsSound effectLatent heatSoftware developerGastropod shellMultiplication signParameter (computer programming)Virtual machineSoftwareServer (computing)Scheduling (computing)Open setSinc functionOnline helpUser interfaceSlide ruleTime zoneSelectivity (electronic)BootingComputer animationLecture/Conference
08:03
MultiplicationProduct (business)CASE <Informatik>Lattice (order)Standard deviationDistribution (mathematics)Flow separationParallel portCodeTime zonePoint (geometry)MiniDiscAnalytic continuationSlide ruleResultantStatistical hypothesis testingHoaxDifferent (Kate Ryan album)Variable (mathematics)CloningStatistical hypothesis testingDirectory serviceProduct (business)Virtual machineRepository (publishing)Scheduling (computing)Instance (computer science)TouchscreenFunction (mathematics)Data structureOpen sourceBitOpen setPlanningLetterpress printingSoftware developerOnline helpParameter (computer programming)Computer animation
14:54
BuildingComputer-generated imageryStatistical hypothesis testingScheduling (computing)Open setPhysical systemCASE <Informatik>Multiplication signFrame problemWeb applicationBitPoint (geometry)Statistical hypothesis testingBootingStatistical hypothesis testingCartesian coordinate systemInclusion mapOperator (mathematics)Sign (mathematics)Formal languageWeb 2.0Lattice (order)Software developerInstance (computer science)Web browserCore dumpSineMedical imagingComputer animation
18:46
Open setCASE <Informatik>Lattice (order)Statistical hypothesis testingData conversionInstance (computer science)Template (C++)Presentation of a groupBit
20:13
UsabilityComputer animation
Transcript: English(auto-generated)
00:10
My name is Christopher Hofmann, I'm working for SUSE for quite a while now, it's actually 16 years now, and after I spent a while in different other teams,
00:29
for example also the OpenSUSE team, I think it was about four years ago when we kind of started or continued developing also OpenQA,
00:43
we took it over from Bernard Wiedemann, and after I changed to the YAS team two years ago, it was kind of a good coincidence that we also decided in the YAS team to use OpenQA a bit.
01:05
So actually this talk should not be about the basics of OpenQA, just to explain it in a nutshell, OpenQA tests our installation process,
01:23
it more or less does this by just sending key presses to a QAMO instance where an installation is running, and it checks back whether that happens, what should happen, by matching screenshots.
01:42
This is more or less like the famous monkey and a typewriter, more or less. And since the installation process is, let's say, nearly 100% YAST,
02:01
we as the YAS team and what we are developing affects a test run of OpenQA quite a lot. So whenever we change something, it is most likely that we even break a test run in OpenQA,
02:21
because even a slight change in the UI, for example, lets the test run fail. And since we are in the meanwhile also working together with the UI design team, this happens quite often that we change actually a UI.
02:43
And what to do now, how to avoid those breakages as much as possible and help our QA department. We should test as soon as possible.
03:02
So if we change something in YAST, then we should even always keep in mind that this might somehow affect OpenQA and QA department. And so we can even warn the people from QA that we change something,
03:24
what they should expect to break a test run. And what we also try to do is that we can even maybe deliver an adapted test run
03:40
or the matching screenshots together with a new feature. So to be able to test what we changed, we of course need to get our new software,
04:03
our new packages into OpenQA. Since OpenQA tests ISO images, the obvious thing would be let's create ISO images and test it. And maybe even on every package submission of any YAST package.
04:22
But since mastering a DVD image takes quite some time and needs some resources, maybe the better approach would be just pack together what we changed, put it in a driver update disk. I hope you know this feature actually.
04:44
I'll show you later. You put all the changed stuff on the driver update disk and this just contains a few packages. It's easy to master and then you can feed it into OpenQA.
05:05
To create such a driver update disk, there's a nice command. It's actually developed by one of our team members and there's an extra package for that. Just use this command line.
05:20
Then your driver update disk falls out, put it on an FTP server. And then if you are kind of familiar with OpenQA, schedule a test run with this parameter to let Yast include the driver update disk into the instance.
05:49
To do this, you of course need shell access to the machine who actually runs the test. And if we would use the official instances of OpenQA,
06:02
every developer who wants to test something needs access to that machine and this is obviously not a good idea. To develop tests and enhance tests, you need this access more or less anyway.
06:22
The most obvious, best idea would be just run your own OpenQA instance on your machine. But if you don't develop OpenQA tests that much, you don't need it that often and after a while of not using it, you need to update everything.
06:45
Let's say after two months, you need to update everything, the testing software itself and the test runs and all the needles and this is an annoying effort that not everyone has to do on its own.
07:04
So, what we did in our team, we decided to get an own OpenQA server for our team where we can more or less offer our team members a running OpenQA instance where they also can go and develop.
07:22
And the side effect is that it's much more powerful than most of the desktop machines you have on your desk. So, I was maybe a bit fast with this slide, but how can one develop tests on a machine
07:43
or how can several people together develop tests on a machine without breaking the other guy's stuff because the installation process needs a specific order of tests to more or less get a running system out of it.
08:00
And if at some point it breaks, for example, if it breaks in the timezone selection, the installation cannot continue. And if I change the timezone tests and it breaks for a reason, someone else that wants to change another test cannot test his own stuff because he doesn't get to that point.
08:23
So, we needed a way how several people can have their own code on the same OpenQA instance. And there's a nice feature in OpenQA that we, of course, obviously need. We can test several distributions there in parallel
08:42
which have their own test repositories, their own needle repositories. And we can abuse this just to create kind of a fake distribution per user where we can in parallel develop without affecting each other.
09:05
Okay, this console output might be a bit strange, but it's more or less the directory structure of OpenQA or the tests and the test directory. And let's say user Kenny wants to create his own distribution
09:23
for himself to develop. He creates just a directory in this test directory where, for example, there's also the OpenSUSE and SLE, the racial distributions we are usually testing. And so, Kenny creates his own distribution.
09:43
He even can fork the tests of OpenQA in his own repository where he can develop. And after he finished his development, he can open a pull request to upstream OpenQA.
10:05
After cloning all that stuff in there, you still have to create a few other directories which is somehow not so straightforward, but as it works,
10:20
create also a subdirectory under the products subdirectory where the main PM, which is the main schedule of OpenQA, needs to be copied from the distribution you want to base your stuff on, which in this case is OpenSUSE. And then you also need the needles directory.
10:42
Short explanation, the needles are those things that you try to find in an haystack, which means those are the screenshots who you compare the screen output of QAM with to find out whether you see what you expected to see.
11:04
And those are obviously different between those main distributions, SLE and OpenSUSE, because we have a different theming there. The Yast on SLE is black backgrounded and the one in OpenSUSE is until now at least gray backgrounded.
11:21
So we have different screenshots for different distributions and so if you want to base your fake distribution on that, in this case use the OpenSUSE needles. And this is what the result of all this. And after you created your fake distribution,
11:49
you can schedule jobs. In this case the best offer is usually the tool CloneChop,
12:00
which clones already existing old test runs with some tuned variables. In this case you have to clone the existing, already old existing test run 1.0.7.1 and overwrite the parametered history
12:23
with your new own distribution name, which is Kenny. And so you cover the OpenSUSE distribution you want to test that it uses its own test directory, your own needles and your own tests,
12:43
and you get a nice test run. Yeah, let me just check where I am. And so we can share one machine for several people to develop tests.
13:06
Of course this is not the only thing. Definitely what helped us a lot is that we talked much more with the QA department.
13:24
So in all of our sprint meetings, so means sprint planning, sprint review meetings and standard meetings, we have someone from the QA department that we can really directly contact. This guy here, Joseph.
13:44
And this helped us also a lot to avoid duplicates in work. Also this doesn't work everywhere. We had this case where two, recently this case where actually QA and ourselves developed the same test, but we found this out quite early.
14:02
Yeah. Then, what else can we do with our own testing machine? Of course, if you have a powerful machine you want to automate stuff, what I am planning for the future is actually, I also want to create those drive update disks in an automatic way,
14:26
maybe even on check-in, so something with Jenkins, that we create a drive update disk and feed this automatically to our test runs. This might help us have even less work with testing
14:44
and support QA department with that, but not yet at this point. Oh, I even have a slide for this. So, I was a bit faster than expected, but I guess you have some questions about that
15:03
and feel free to ask them now. Oh, yeah, someone needs a microphone.
15:27
You can also use sign language if you want.
15:47
So, hang on. It's off, okay. Maybe the PA guy can raise the slider. Okay, now it's working.
16:00
So, I'm not too familiar with OpenQA, but I was wondering if it's able to test things which are not the installation, but rather, for example, a web application running in a browser. Can you simulate a test of clicking and using a web application within this operating system? So, from the JAST side, from the JAST view,
16:25
actually, we only care about installation more or less and that the system is usable. OpenQA, if you ever checked it, if you checked our instances, we even test applications. One of them is also Firefox.
16:42
But testing a web application would actually include something like use boot an existing installed image. This is possible. Start your application and then try to use your web application. But mouse clicks is something we did not use yet.
17:01
It does support mouse clicks. It does work. We have used it for testing the OpenQA web UI. So, it can do it. The short answer is apparently you do. Back then, when we developed it, mouse was not really good to use. It had its bad days, but it's better now. You had a question? Coolo has questions.
17:23
What I'm wondering is on who is responsible and on what schedule are you updating the instance and the tests you place your development on? Is this coordinated during the sprint meetings or are you just floating? The point is that I did not really very good understand you because the speakers are in front of me.
17:47
On what time frame and who are you coordinating the updating of the system and the tests that you're developing on? The test development is more or less just a manual thing.
18:02
So, someone gets the job. Someone in our team gets the job. You implemented a feature. Please write a test for it. And then someone needs a test and does this manually. So, after manual scheduling, manually developing and testing the test, you can do your submit request.
18:20
So, it's not that we automatically use it for now. What I want to do is in the future, of course, is doing something like that. But it's not yet the case. But the system, like the Open QA installation? You mean how I update the Open QA installation? Who does this and how do you coordinate this in the team?
18:43
Usually, so far, I do this. And we have a crunch up for using fetch needles. This stuff is updated kind of every 15 minutes, something like that. But the Open QA itself, I update when I see it as appropriate,
19:03
which was recently the case when you changed the UI a bit. As soon as we noticed that something does not work as expected or different than on the official instance, I know that I need to update something. But the tests and the needles are always up to date.
19:30
No one else? Okay. Then, I'm still a bit early.
19:52
So, thank you for listening my talk. Join the conversation as the presentation template suggests.
20:05
Have a lot of fun and join us.