So welcome everyone to the Wine Dev Room for the folks who are new. I think Eric made his pitch for the name tags. We're trying to push positive to have name tags, so please come get one. And what we're going to talk about in this next 30 minutes or so or 25 minutes is the wine test framework. There's a couple of areas there. And I'm going to try to briefly overview, give an overview of the wine test framework

and then these guys are going to make fun of me because I'm going to get it wrong. That will be entertaining for them, so that works. And then I think Francois is going to go in more detail with the new test bot and some of the situations there. So very briefly, for those of you who don't know, wine has a

very rich and mature set of unit tests for Windows. So there's a very powerful set of, I don't know how many tests there are now, but it's thousands, right? So every Windows API has a test that goes with it. And in fact we had had, VMware uses the

wine test framework to verify that their Windows runs correctly. So it's how they prove that a Windows VM is running right is because the wine test runs to completion. It's a very powerful set of tools and it's most commonly done by running just make tests or that's how the developers do it. But there is also a Windows

executable that is created that you can download and you can run and that executable runs essentially the equivalent of make test and then it uploads its results to test.yhq so I wanted to show test.yhq.org first. Yeah, and this is the point

of today you wine developers and anyone else who wants to help is we're in atrocious shape right now, our testing I'm sorry, what? Yeah, he didn't want to show this page, did he? So last year there was a lot of green on the board and we were celebrating how much green

there was. So this is showing, jeez, we're at 10% failures. Why are we failing so badly? Do we have any idea? There's no direct link to that one so we'll look for it. Yeah, but we all know where it is. Every single wine developer here knows about

this page. Direct music is all green. And that's all you know is your area. So, and what's unfortunate is this is actually really, this is horrible you guys. This is horrible because we were on a path to all green and now we've like backslid.

Probably Alexander is nice and still accepts patches to break the test. I don't know, we need to, alright. Yeah, we didn't have the make test lecture. Alright, well this is the make test lecture. So I want green next time we meet. So, and actually in all

seriousness, for any of you that have laptops, one of the things we are hoping to do, what we traditionally do at a wine conf is we will look at the make test status and then we will have a session where that session is scheduled for four this afternoon where we'll kind of look at some of the common tests and see if we can squeeze some of these out. So, and I see that technology has moved on and it used to be at wine conf

that everyone had a laptop and that's no longer the case, which is interesting. So get running make test on your tablets and your phones, alright. And then we'll be correcting that. So for any of you who are new to wine, can we go to the, you can do something very useful and there's a simple executable

you can download and if you have wine installed, a git build of wine you can download and run this executable and then it will automatically upload to the web page and it generates great statistics for us. Alright, so you're showing some detail. Okay, so there's some green. Okay, so we can feel better about this page.

So the link to the wine test executable is there. Okay, that's not all that obvious. There was a better page. There's a better page. Okay, so at the bottom of test.winehq.org there's a link

that, okay, is off your screen. Maybe we should put it at the top, okay. Alright, so there's another to do. Who's going to take that to do? You're going to do that Andre? Just move this to the top.

So this is a useful thing you can do and then after you run it your results will appear and you can see your results and then you can even go in and see what area it is and if you're a developer you can run make test and you can submit patches and fix it and that's a glorious thing to do. So I think that was the bulk of what I wanted to say around the regular wine

test framework and then I think we have a lot of talking about the new wine test bot. Just to frame that for people, in addition to running all of these tests on Linux we now also run all of these tests on Windows. So we have a Windows QMU, we had a VMware

system for years and years and years that was the test bot and unfortunately the maintainer of that passed away which was a tragic event and so we've been shifting it to a QMU based test system where we're running all these tests actually on Windows and in theory they should all be green on Windows but then what's

good about it is a patch can't get into wine, a change to the test can't get into wine unless it runs correctly on Windows or so goes the theory. And so the wine test bot is actually a set of Windows VMs that run all of these tests on Windows and unfortunately we're having problems with the new wine test bot

so the other thing we need to exhort people to do is to help us get those tests fixed and working. I think that's where I turn it over to you. Did I miss anything? So maybe I'll start by saying that the wine test

bot actually integrates into this site. So like for instance when you look at the results on the test.winhq.com you have so here we still have the whole test bot, it's still active, it's supposed to go away

like in June last year. Has anyone heard from them recently? Not necessarily, but it's still here. It still works if it's fast. Well they're paying money to host it they were told to shut it down so maybe I'll send them a further. If they want to keep it running that's great but I think. I think we can

pay them back from the wine test It's really more of a, it's a courtesy thing. We should really you spoke with them didn't you?

Maybe I should send an email. I think it's time to turn it over. I know we all like it but wasn't Martin maintaining it? He had it but that should do it. The problem is that having two test bots is a bit confusing sometimes people ask me

the test bot doesn't work well, it's the whole test bot I can't really do anything about it. I can only fix stuff on the new test bot. So on these pages there's a few different types of results so you have results from the old test bot, it's a WTB

for wine test bot and you have also newTB which is for the new wine test bot so if you have a failure in the newTB column so not too many of them so here for instance, then it means you can

submit jobs to the new test bot to figure out what's wrong so anything on the newTB machine, anything on there, you can easily investigate. Another set of results is about

FG- on the Windows version that's a test I run on my machine on VMware virtual machines so you just have to know that it's virtual machines but now you should start to see more regularly

FG- on the machine type so here's my EPC, because I've set up a number of scripts so that now I can run wine test everyday on Linux and then it travels automatically and runs the test on Windows also. So these are real

hardware, because I know that some of you want to test on real hardware so you've got some, it's just an Intel graphics card, but if someone wants to do the same kind of thing, I can provide my scripts

I'll provide my scripts they are kind of ugly, and the graphics are probably specific to my contribution, but I can provide those scripts anyway and the other thing which is interesting is so EPC is a real hardware, and you will find also on Linux, as I said, I run it on both Windows

and Linux, so you can compare the two results if it works on Windows and doesn't work on Linux, that's interesting, it's not because the graphics card is not capable of doing some things about driver, Linux driver or something in white so I also run a number of tests on Linux

like on my desktop, and that's another laptop that I've set up to do the tests now on both Linux and Windows 8.1, and that's also one where you'll see more results regularly

and there's also other tests that are being run by other people and when things don't work, when you see a failure there well, if you can figure it out, it's cool, right? so if it's on the white testbot machine, simplest is to go to

the white testbot on the summit jobs there the new testbot, it's new testbot if you just type testbot, it's the old one so you have to know to go to this

to this to have the right URL don't hesitate to create an account I'm not sure why developers have an account on the white testbot, who's got an account on

not too bad so don't be shy you don't need to be in the top 10 of the jovial rank to get an account you need to register, but it's mostly to

weed out spammers and people who want to do wine tasting rather than wine testing once you request an account, the account will be approved it might take a few hours or something like that and then you get your account

and what you get is that you can submit patches directly on the website so if your patch gets rejected well, the wine testbot runs all the patches that are sent to wine patches so if Marvin, that's the wine testbot

tells you that there's failures rather than spamming the mailing list with a number of attempts to fix the test you can submit jobs on the testbot

to figure it out and you can not just do iterations of your tests you can run anything you want, pretty much you can upload an executable, so you can really try to figure out what's going on with that windows version

so we have the testbot it's got a number of virtual machines we try to have a number of different windows versions

some of those versions are hard to get because it's server edition and it's really expensive no good result so you could ask some more versions if you want to but we may not be able to purchase them but what you can do is if you

is there a feature you want in the wine testbot you can ask me or you can you can make a bug report on winhq bugs.winhq.org or you can just download it from here and fix it yourself or you can submit patches

it's not too hard to set up

so there's a special you have to enter the bugs in the right it's not wine, it's the testbot

so that way it doesn't get lost it's quite possible to modify the test VM configuration to configure stuff so that you can

so it's more useful for you I can't add a winhq app to get winhq test running I added a msxml for dls and some other dls like that if you want test to be run on the G drive

and the root of the drive as well let me know what you want so the action items are

shut down the old testbot, move the urls and then make it green so there's all those failures I try to maintain the winh testbot I try to improve add functionality to it

but I'm not going to fix the test there's way too many tests, I don't know the code so I really need you, the winh testbot to work on it, I cannot do it so if you cannot figure out

why the test fails by going on the winh testbot and running executables or doing some more debug traces you can also ask me about how the VM is set up and maybe I can help, maybe I cannot

because I don't know much about the area that the test failed it's theoretically possible but probably too complicated

to do regression analysis, to do bisects and to get to the point, right? some extent the idea would be to get lame and e-mail people the idea is to test the patches before they are committed so there's no need to bisect that's good enough

you get the source thing, so it's usually pretty obvious which patch was named after you and then I just made people ask can we get a Linux machine that Alexander set up and it's like a majesty machine

people would just get rejected if the patch would compile because that would be the majesty machine and it would be easy to reject stuff

I don't think it should match his machine that way you will have two different configurations the thing is you want to test patches send to one patch so that's one of the many things that need to be done for the test mode

how about foreign language support? yes, for foreign language support what I've planned to do yes, here

we have a number of windows ultimate versions what's really nice about windows ultimate we have the windows 7 one and the windows vista

what's really nice about windows ultimate is you get all the languages the windows 7 one is set up with all 45 supported languages the prime is right now in the testpad I have a snapshot of windows 7 and it's set up for one language

people can only test in that configuration one of the things I want to do is to have multiple snapshots so that ultimately you will be able to test in Japanese, in Arabic, you just pick the right snapshot and test that configuration

it's also a bit those VMs are sometimes a bit turning to setup like the windows vista I don't have all the languages installed because when I install more than a couple of languages

then Windows are dead right? but I'll figure that out at some point and for Linux so I don't know I don't know exactly how I'm going to do it

it's going to introduce performance problems but I guess I'll just ignore it for now and when we have something we're stronger to figure out how to improve performance one thing I may do is now that I have this

script that I run on my laptops which automatically reboot to Windows I need a Linux partition maybe it will go with real hardware the idea would be to have this real hardware machine

with Linux and make it so that the testpad can submit jobs and run the test on Linux or tell the machine to reboot to Windows and send the job to run on Windows how do you make sure that this image doesn't change? the idea is that what I do on my laptops

is I use NTFS clone to take an image of the Windows partition so it only saves the part of the partition which is used so it's on the Linux side, compressed and before rebooting I restart the

disk image, so it starts to compile restarts the disk image in parallel, waits for all that stuff to be learned and then runs the test on Linux and then sets up the job to reboot to Windows, reboots, and on Windows I have a cache file which downloads my tests, runs my tests, and just downloads them

I can probably do something like that and integrate it in the Windows part I'm just going to take a bit of time to adjust it

okay, that's what I really want to see filtering on the side page, so that you don't see all 150 tests

but only find some of the direct tests and also you can fix it what you can do is click on different device and you can get

I guess that can be added if you find someone to do it you can work on that part of the site

yeah, I know, right I'm not sure I know what that's for Yeah, right Generates that additional bit Maybe a bit of JavaScript

You could actually I mean, you could ask Jeremy Newman to look into it That side is written on the box Oh, that does all that

Solaris, come on Well, I The Solaris on BSD results It's a virtual machine that I run on a truck that's really

winded with rocks and solaris Does anyone who works on the fifth making sure that one works on on BSD from time to time I don't think that anyone really wants to take

the trouble of fixing so far The most central part is getting the test to work on Windows The goal of those tests is to document how the Windows API is working

So we go to the API and check the results and then we check that wine returns the right results If that test fails on Windows it means it's testing the wrong thing So if you make wine pass the test you're breaking wine

So you have to make sure that it really works on Windows first So right now it's really not the case, we really have to get this part fixed and then obviously we have to move to the our platform

I know of one Q&U issue One Yeah, Vientum machines cause some trouble for all the direct 3D and the punchy stuff and they also cause some trouble with sound sometimes because of

timing issues, but even for direct 3D a test that cannot work on a virtual machine like 3D because there's no 3D acceleration The thing is if the test should be skipping

the test should be skipped, not failing So even if even direct 3D tests on virtual machines, if they fail it has to be fixed Do we have hardware dedicated to running

hardware sitting in a rack cold that is available to run these on real hardware So if we want, if we can interject that, that's available I mean you only need to pick one Windows version, but it seems like it would be useful to have that

No, it's still sitting there, we bought it two years ago Yeah, I've been requesting this ever since game, it was the first test version and it never happened Well I think it's all available for you to work on and I suspect Francois would be really happy to help you get it going

Yeah, I have a bunch of machines at home with myself that I only use for testing Well it would be helpful for me if I could have touched my machines because Windows is much easier to maintain when you have physical access but I guess it's a bit more tricky because they're going to be offline every now and then when I'm not at home and so on

and the power consumption would be an issue as well It's a bust Besides lack of time, in the past one of the blockers to working real hardware to the WinTestBot is that I didn't know how to handle the rebooting, restoring the image and stuff like that

Now that I have it working on my laptops I can, it's not going to be easy to integrate into the TestBot but it's one less robot I have a suggestion for you to get your Windows test fixed

I have one thing that works wonderfully for regression and it's code freeze, so it would be a good idea to get out of the code freeze on the average 4 or 3% of the Windows tests We have now 7.5%

The number is zero Zero wouldn't be possible It's possible if you set your mind to it To do one conch later, we will have zero and the conch code freeze will end

I think that's actually one of the big bonuses of our stable releases I think the big bonus of the stable release is the code freeze Is it supposed to necessarily be the release itself? No, it doesn't require a stable So yeah, whatever The best thing would be if the TestBot is reliable

and other stuff stops accepting your patches Well we need to get out of the ambiguity between the old TestBot and the new TestBot So we need to fix that so there is only one TestBot and then we need to drive that to green I think two TestBots

I think people I think people see it as being up in the air Well it's not really resolved, so I'm not going to work on it No, one problem is we don't have SDBMs on the new TestBot That's the one that's running my patches now So we miss failures

And the other issue is just that we still have failing tests So that we cannot reject the patches because the test is failing Because it was already failing before the patch We just need people to take ownership of the problems and fix them And also we don't have many test results

for the Mac There are only two test results, one for the Mac driver and one for the excellent driver So that's the machine I run at home And I have a script The scripts I talked about, they also drive the test on the Mac

So if there are more volunteers to test on the Mac You can test on normal, you can test on graphics cards One last question One last question, do you think I've seen all the test results

That they didn't bring because of some new wine patches that were wrong, but maybe something With the machines, which the machines have Do you have a look at the results page between When Windows 7 recently turned from yellow to red

And you have a look at the red ones Then you see there were no patches in wine Yeah, that's because it's running test against the wine server and we upgraded Deviant Wine server

They were broken by the Deviant upgrade on the server Ok, I see No, you just need to adapt the test

Yeah, there are a few different things But the thing is People should be looking at that and seeing that One of your tests, you should be fixing Not you personally, but everybody

We need a way to force us to do that Well, if someone could stop taking patches, I think we'd all fix the test Yeah, but we don't have our tests, there are some that are failing Maybe you should reject patches on tests that are failures Even if the failures predicted the patch

I sometimes do that too Ok, we don't accept patches for the DLL, for example, if it has something failing, then For somebody else, this means that we stop accepting patches I don't know

anything about certificates I will never be able to fix that I won't be able to send anything there That's why it's a judgment call That's why it's necessary for the original test order To get involved and fix the test

Because for a lot of people it can be pretty hard The other thing we will do in the hacking session is go through the list And name names Make people fix the test We can make it green or we can take out that deal, right?

We can also make it green One patch fixes all that Alright, why don't we go ahead and wrap up