The Amazing Wine Test Framework
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 199 | |
Author | ||
License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/32635 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
FOSDEM 201449 / 199
2
3
10
11
12
13
14
17
18
19
23
24
25
27
29
45
46
47
48
49
50
51
52
55
56
57
58
65
67
68
70
72
74
75
76
77
78
79
81
82
84
86
87
89
90
93
94
100
101
102
103
104
106
107
109
110
111
112
113
114
115
119
122
123
124
126
127
128
129
137
139
141
143
145
147
150
154
155
158
161
163
164
165
166
168
171
174
175
176
179
182
183
185
188
190
191
192
194
195
196
00:00
QuicksortPower setSoftware developerSoftware testingWeb pageGreen's functionAreaMusical ensembleSingle-precision floating-point formatWhiteboardLink (knot theory)Patch (Unix)RobotMultiplication signLaptopWindowSoftware frameworkTablet computerBuildingEvent horizonResultantCapability Maturity ModelTouchscreenPhysical systemRegular graphAdditionTheorySoftware maintenanceUnit testingPoint (geometry)Shape (magazine)Equivalence relationRight angleSet (mathematics)Mathematics1 (number)Complete metric spaceMountain passTask (computing)Projective planeSound effectFood energyUser interfaceWordCausalityGreatest elementStatisticsCASE <Informatik>Line (geometry)Focus (optics)XML
05:47
Student's t-testEndliche ModelltheorieSoftware testingFrequencyUser interfaceLine (geometry)Set (mathematics)Physical lawMereologyWordRobotMultiplication signResultantEmailInstance (computer science)Different (Kate Ryan album)Type theoryBitWeb pageVirtual machineRevision controlAbstract machineWindowProcess (computing)Graphics processing unitNumberScripting languageDevice driverLaptopComputer animation
11:20
WordRight angleMultiplication signPatch (Unix)User interfaceSoftware testingRevision controlSoftware developerUniform resource locatorProcess (computing)RankingIterationElectronic mailing listNumberEmailWebsiteXML
13:21
Rule of inferenceMedical imagingCondition numberVirtual machineRevision controlSoftware bugResultantDifferent (Kate Ryan album)User interfaceNumberPatch (Unix)Traffic reporting
14:24
Form (programming)Vector spaceMedical imagingFood energyMachine visionConfiguration spaceEvent horizonMultiple RegressionAbstract machineFunctional (mathematics)CodeSoftware testingGroup actionFormal languageUniform resource locatorMobile appAsynchronous Transfer ModeRootSoftware bugPatch (Unix)Figurate numberTracing (software)AreaSource codeLinear regressionExtension (kinesiology)
18:58
Endliche ModelltheorieCASE <Informatik>TheoryMoment (mathematics)ProteinRight angleResultantReading (process)UsabilityForm (programming)WindowMereologyBitCache (computing)Software testingComputer fileUser interfaceMedical imagingProcess (computing)Revision controlFormal languageNumberMultiplication signConfiguration spaceCuboidVirtual machineWebsiteFigurate numberScripting languageLaptopPoint (geometry)Computing platformAbstract machineReal numberComputer hardwarePartition (number theory)CloningMiniDisc
26:18
Water vaporMereologyAbstract machineDifferent (Kate Ryan album)Data structureInteractive televisionProteinUniformer RaumHypermediaUniverse (mathematics)AuthorizationPhysical lawState of matterGraphics processing unitWindowVirtualizationSoftware testingComputer hardwareDirection (geometry)FreezingCodeNumberRevision controlPatch (Unix)Scripting languageMedical imagingDevice driverBitResultantVirtual machineLinear regressionLaptopRobotLecture/Conference
30:57
Software testingScripting languageGraphics processing unitResultantUniverse (mathematics)1 (number)Bit rateGroup actionSystem callTouchscreenAlpha (investment)Inclusion mapRight angleAbstract machineServer (computing)Patch (Unix)Web pageWindowElectronic mailing listOrder (biology)Hacker (term)Modal logicPublic key certificateXMLMeeting/Interview
Transcript: English(auto-generated)
00:00
So welcome everyone to the Wine Dev Room for the folks who are new. I think Eric made his pitch for the name tags. We're trying to push positive to have name tags, so please come get one. And what we're going to talk about in this next 30 minutes or so or 25 minutes is the wine test framework. There's a couple of areas there. And I'm going to try to briefly overview, give an overview of the wine test framework
00:24
and then these guys are going to make fun of me because I'm going to get it wrong. That will be entertaining for them, so that works. And then I think Francois is going to go in more detail with the new test bot and some of the situations there. So very briefly, for those of you who don't know, wine has a
00:42
very rich and mature set of unit tests for Windows. So there's a very powerful set of, I don't know how many tests there are now, but it's thousands, right? So every Windows API has a test that goes with it. And in fact we had had, VMware uses the
01:02
wine test framework to verify that their Windows runs correctly. So it's how they prove that a Windows VM is running right is because the wine test runs to completion. It's a very powerful set of tools and it's most commonly done by running just make tests or that's how the developers do it. But there is also a Windows
01:22
executable that is created that you can download and you can run and that executable runs essentially the equivalent of make test and then it uploads its results to test.yhq so I wanted to show test.yhq.org first. Yeah, and this is the point
01:42
of today you wine developers and anyone else who wants to help is we're in atrocious shape right now, our testing I'm sorry, what? Yeah, he didn't want to show this page, did he? So last year there was a lot of green on the board and we were celebrating how much green
02:01
there was. So this is showing, jeez, we're at 10% failures. Why are we failing so badly? Do we have any idea? There's no direct link to that one so we'll look for it. Yeah, but we all know where it is. Every single wine developer here knows about
02:21
this page. Direct music is all green. And that's all you know is your area. So, and what's unfortunate is this is actually really, this is horrible you guys. This is horrible because we were on a path to all green and now we've like backslid.
02:42
Probably Alexander is nice and still accepts patches to break the test. I don't know, we need to, alright. Yeah, we didn't have the make test lecture. Alright, well this is the make test lecture. So I want green next time we meet. So, and actually in all
03:03
seriousness, for any of you that have laptops, one of the things we are hoping to do, what we traditionally do at a wine conf is we will look at the make test status and then we will have a session where that session is scheduled for four this afternoon where we'll kind of look at some of the common tests and see if we can squeeze some of these out. So, and I see that technology has moved on and it used to be at wine conf
03:23
that everyone had a laptop and that's no longer the case, which is interesting. So get running make test on your tablets and your phones, alright. And then we'll be correcting that. So for any of you who are new to wine, can we go to the, you can do something very useful and there's a simple executable
03:43
you can download and if you have wine installed, a git build of wine you can download and run this executable and then it will automatically upload to the web page and it generates great statistics for us. Alright, so you're showing some detail. Okay, so there's some green. Okay, so we can feel better about this page.
04:04
So the link to the wine test executable is there. Okay, that's not all that obvious. There was a better page. There's a better page. Okay, so at the bottom of test.winehq.org there's a link
04:24
that, okay, is off your screen. Maybe we should put it at the top, okay. Alright, so there's another to do. Who's going to take that to do? You're going to do that Andre? Just move this to the top.
04:40
So this is a useful thing you can do and then after you run it your results will appear and you can see your results and then you can even go in and see what area it is and if you're a developer you can run make test and you can submit patches and fix it and that's a glorious thing to do. So I think that was the bulk of what I wanted to say around the regular wine
05:04
test framework and then I think we have a lot of talking about the new wine test bot. Just to frame that for people, in addition to running all of these tests on Linux we now also run all of these tests on Windows. So we have a Windows QMU, we had a VMware
05:24
system for years and years and years that was the test bot and unfortunately the maintainer of that passed away which was a tragic event and so we've been shifting it to a QMU based test system where we're running all these tests actually on Windows and in theory they should all be green on Windows but then what's
05:43
good about it is a patch can't get into wine, a change to the test can't get into wine unless it runs correctly on Windows or so goes the theory. And so the wine test bot is actually a set of Windows VMs that run all of these tests on Windows and unfortunately we're having problems with the new wine test bot
06:03
so the other thing we need to exhort people to do is to help us get those tests fixed and working. I think that's where I turn it over to you. Did I miss anything? So maybe I'll start by saying that the wine test
06:21
bot actually integrates into this site. So like for instance when you look at the results on the test.winhq.com you have so here we still have the whole test bot, it's still active, it's supposed to go away
06:41
like in June last year. Has anyone heard from them recently? Not necessarily, but it's still here. It still works if it's fast. Well they're paying money to host it they were told to shut it down so maybe I'll send them a further. If they want to keep it running that's great but I think. I think we can
07:01
pay them back from the wine test It's really more of a, it's a courtesy thing. We should really you spoke with them didn't you?
07:23
Maybe I should send an email. I think it's time to turn it over. I know we all like it but wasn't Martin maintaining it? He had it but that should do it. The problem is that having two test bots is a bit confusing sometimes people ask me
07:43
the test bot doesn't work well, it's the whole test bot I can't really do anything about it. I can only fix stuff on the new test bot. So on these pages there's a few different types of results so you have results from the old test bot, it's a WTB
08:03
for wine test bot and you have also newTB which is for the new wine test bot so if you have a failure in the newTB column so not too many of them so here for instance, then it means you can
08:25
submit jobs to the new test bot to figure out what's wrong so anything on the newTB machine, anything on there, you can easily investigate. Another set of results is about
08:42
FG- on the Windows version that's a test I run on my machine on VMware virtual machines so you just have to know that it's virtual machines but now you should start to see more regularly
09:01
FG- on the machine type so here's my EPC, because I've set up a number of scripts so that now I can run wine test everyday on Linux and then it travels automatically and runs the test on Windows also. So these are real
09:21
hardware, because I know that some of you want to test on real hardware so you've got some, it's just an Intel graphics card, but if someone wants to do the same kind of thing, I can provide my scripts
09:40
I'll provide my scripts they are kind of ugly, and the graphics are probably specific to my contribution, but I can provide those scripts anyway and the other thing which is interesting is so EPC is a real hardware, and you will find also on Linux, as I said, I run it on both Windows
10:02
and Linux, so you can compare the two results if it works on Windows and doesn't work on Linux, that's interesting, it's not because the graphics card is not capable of doing some things about driver, Linux driver or something in white so I also run a number of tests on Linux
10:23
like on my desktop, and that's another laptop that I've set up to do the tests now on both Linux and Windows 8.1, and that's also one where you'll see more results regularly
10:40
and there's also other tests that are being run by other people and when things don't work, when you see a failure there well, if you can figure it out, it's cool, right? so if it's on the white testbot machine, simplest is to go to
11:03
the white testbot on the summit jobs there the new testbot, it's new testbot if you just type testbot, it's the old one so you have to know to go to this
11:23
to this to have the right URL don't hesitate to create an account I'm not sure why developers have an account on the white testbot, who's got an account on
11:44
not too bad so don't be shy you don't need to be in the top 10 of the jovial rank to get an account you need to register, but it's mostly to
12:02
weed out spammers and people who want to do wine tasting rather than wine testing once you request an account, the account will be approved it might take a few hours or something like that and then you get your account
12:21
and what you get is that you can submit patches directly on the website so if your patch gets rejected well, the wine testbot runs all the patches that are sent to wine patches so if Marvin, that's the wine testbot
12:42
tells you that there's failures rather than spamming the mailing list with a number of attempts to fix the test you can submit jobs on the testbot
13:00
to figure it out and you can not just do iterations of your tests you can run anything you want, pretty much you can upload an executable, so you can really try to figure out what's going on with that windows version
13:20
so we have the testbot it's got a number of virtual machines we try to have a number of different windows versions
13:42
some of those versions are hard to get because it's server edition and it's really expensive no good result so you could ask some more versions if you want to but we may not be able to purchase them but what you can do is if you
14:01
is there a feature you want in the wine testbot you can ask me or you can you can make a bug report on winhq bugs.winhq.org or you can just download it from here and fix it yourself or you can submit patches
14:21
it's not too hard to set up
14:51
so there's a special you have to enter the bugs in the right it's not wine, it's the testbot
15:01
so that way it doesn't get lost it's quite possible to modify the test VM configuration to configure stuff so that you can
15:21
so it's more useful for you I can't add a winhq app to get winhq test running I added a msxml for dls and some other dls like that if you want test to be run on the G drive
15:42
and the root of the drive as well let me know what you want so the action items are
16:02
shut down the old testbot, move the urls and then make it green so there's all those failures I try to maintain the winh testbot I try to improve add functionality to it
16:22
but I'm not going to fix the test there's way too many tests, I don't know the code so I really need you, the winh testbot to work on it, I cannot do it so if you cannot figure out
16:41
why the test fails by going on the winh testbot and running executables or doing some more debug traces you can also ask me about how the VM is set up and maybe I can help, maybe I cannot
17:03
because I don't know much about the area that the test failed it's theoretically possible but probably too complicated
17:21
to do regression analysis, to do bisects and to get to the point, right? some extent the idea would be to get lame and e-mail people the idea is to test the patches before they are committed so there's no need to bisect that's good enough
17:41
you get the source thing, so it's usually pretty obvious which patch was named after you and then I just made people ask can we get a Linux machine that Alexander set up and it's like a majesty machine
18:06
people would just get rejected if the patch would compile because that would be the majesty machine and it would be easy to reject stuff
18:21
I don't think it should match his machine that way you will have two different configurations the thing is you want to test patches send to one patch so that's one of the many things that need to be done for the test mode
18:47
how about foreign language support? yes, for foreign language support what I've planned to do yes, here
19:09
we have a number of windows ultimate versions what's really nice about windows ultimate we have the windows 7 one and the windows vista
19:20
what's really nice about windows ultimate is you get all the languages the windows 7 one is set up with all 45 supported languages the prime is right now in the testpad I have a snapshot of windows 7 and it's set up for one language
19:41
people can only test in that configuration one of the things I want to do is to have multiple snapshots so that ultimately you will be able to test in Japanese, in Arabic, you just pick the right snapshot and test that configuration
20:02
it's also a bit those VMs are sometimes a bit turning to setup like the windows vista I don't have all the languages installed because when I install more than a couple of languages
20:20
then Windows are dead right? but I'll figure that out at some point and for Linux so I don't know I don't know exactly how I'm going to do it
20:40
it's going to introduce performance problems but I guess I'll just ignore it for now and when we have something we're stronger to figure out how to improve performance one thing I may do is now that I have this
21:00
script that I run on my laptops which automatically reboot to Windows I need a Linux partition maybe it will go with real hardware the idea would be to have this real hardware machine
21:22
with Linux and make it so that the testpad can submit jobs and run the test on Linux or tell the machine to reboot to Windows and send the job to run on Windows how do you make sure that this image doesn't change? the idea is that what I do on my laptops
21:41
is I use NTFS clone to take an image of the Windows partition so it only saves the part of the partition which is used so it's on the Linux side, compressed and before rebooting I restart the
22:02
disk image, so it starts to compile restarts the disk image in parallel, waits for all that stuff to be learned and then runs the test on Linux and then sets up the job to reboot to Windows, reboots, and on Windows I have a cache file which downloads my tests, runs my tests, and just downloads them
22:24
I can probably do something like that and integrate it in the Windows part I'm just going to take a bit of time to adjust it
22:47
okay, that's what I really want to see filtering on the side page, so that you don't see all 150 tests
23:02
but only find some of the direct tests and also you can fix it what you can do is click on different device and you can get
23:24
I guess that can be added if you find someone to do it you can work on that part of the site
23:44
yeah, I know, right I'm not sure I know what that's for Yeah, right Generates that additional bit Maybe a bit of JavaScript
24:07
You could actually I mean, you could ask Jeremy Newman to look into it That side is written on the box Oh, that does all that
24:24
Solaris, come on Well, I The Solaris on BSD results It's a virtual machine that I run on a truck that's really
24:43
winded with rocks and solaris Does anyone who works on the fifth making sure that one works on on BSD from time to time I don't think that anyone really wants to take
25:02
the trouble of fixing so far The most central part is getting the test to work on Windows The goal of those tests is to document how the Windows API is working
25:21
So we go to the API and check the results and then we check that wine returns the right results If that test fails on Windows it means it's testing the wrong thing So if you make wine pass the test you're breaking wine
25:41
So you have to make sure that it really works on Windows first So right now it's really not the case, we really have to get this part fixed and then obviously we have to move to the our platform
26:01
I know of one Q&U issue One Yeah, Vientum machines cause some trouble for all the direct 3D and the punchy stuff and they also cause some trouble with sound sometimes because of
26:22
timing issues, but even for direct 3D a test that cannot work on a virtual machine like 3D because there's no 3D acceleration The thing is if the test should be skipping
26:42
the test should be skipped, not failing So even if even direct 3D tests on virtual machines, if they fail it has to be fixed Do we have hardware dedicated to running
27:05
hardware sitting in a rack cold that is available to run these on real hardware So if we want, if we can interject that, that's available I mean you only need to pick one Windows version, but it seems like it would be useful to have that
27:24
No, it's still sitting there, we bought it two years ago Yeah, I've been requesting this ever since game, it was the first test version and it never happened Well I think it's all available for you to work on and I suspect Francois would be really happy to help you get it going
27:42
Yeah, I have a bunch of machines at home with myself that I only use for testing Well it would be helpful for me if I could have touched my machines because Windows is much easier to maintain when you have physical access but I guess it's a bit more tricky because they're going to be offline every now and then when I'm not at home and so on
28:03
and the power consumption would be an issue as well It's a bust Besides lack of time, in the past one of the blockers to working real hardware to the WinTestBot is that I didn't know how to handle the rebooting, restoring the image and stuff like that
28:23
Now that I have it working on my laptops I can, it's not going to be easy to integrate into the TestBot but it's one less robot I have a suggestion for you to get your Windows test fixed
28:43
I have one thing that works wonderfully for regression and it's code freeze, so it would be a good idea to get out of the code freeze on the average 4 or 3% of the Windows tests We have now 7.5%
29:02
The number is zero Zero wouldn't be possible It's possible if you set your mind to it To do one conch later, we will have zero and the conch code freeze will end
29:20
I think that's actually one of the big bonuses of our stable releases I think the big bonus of the stable release is the code freeze Is it supposed to necessarily be the release itself? No, it doesn't require a stable So yeah, whatever The best thing would be if the TestBot is reliable
29:46
and other stuff stops accepting your patches Well we need to get out of the ambiguity between the old TestBot and the new TestBot So we need to fix that so there is only one TestBot and then we need to drive that to green I think two TestBots
30:03
I think people I think people see it as being up in the air Well it's not really resolved, so I'm not going to work on it No, one problem is we don't have SDBMs on the new TestBot That's the one that's running my patches now So we miss failures
30:21
And the other issue is just that we still have failing tests So that we cannot reject the patches because the test is failing Because it was already failing before the patch We just need people to take ownership of the problems and fix them And also we don't have many test results
30:42
for the Mac There are only two test results, one for the Mac driver and one for the excellent driver So that's the machine I run at home And I have a script The scripts I talked about, they also drive the test on the Mac
31:02
So if there are more volunteers to test on the Mac You can test on normal, you can test on graphics cards One last question One last question, do you think I've seen all the test results
31:24
That they didn't bring because of some new wine patches that were wrong, but maybe something With the machines, which the machines have Do you have a look at the results page between When Windows 7 recently turned from yellow to red
31:44
And you have a look at the red ones Then you see there were no patches in wine Yeah, that's because it's running test against the wine server and we upgraded Deviant Wine server
32:00
They were broken by the Deviant upgrade on the server Ok, I see No, you just need to adapt the test
32:23
Yeah, there are a few different things But the thing is People should be looking at that and seeing that One of your tests, you should be fixing Not you personally, but everybody
32:41
We need a way to force us to do that Well, if someone could stop taking patches, I think we'd all fix the test Yeah, but we don't have our tests, there are some that are failing Maybe you should reject patches on tests that are failures Even if the failures predicted the patch
33:00
I sometimes do that too Ok, we don't accept patches for the DLL, for example, if it has something failing, then For somebody else, this means that we stop accepting patches I don't know
33:20
anything about certificates I will never be able to fix that I won't be able to send anything there That's why it's a judgment call That's why it's necessary for the original test order To get involved and fix the test
33:43
Because for a lot of people it can be pretty hard The other thing we will do in the hacking session is go through the list And name names Make people fix the test We can make it green or we can take out that deal, right?
34:03
We can also make it green One patch fixes all that Alright, why don't we go ahead and wrap up