Testing for a Better Web
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 199 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/32633 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201451 / 199
2
3
10
11
12
13
14
17
18
19
23
24
25
27
29
45
46
47
48
49
50
51
52
55
56
57
58
65
67
68
70
72
74
75
76
77
78
79
81
82
84
86
87
89
90
93
94
100
101
102
103
104
106
107
109
110
111
112
113
114
115
119
122
123
124
126
127
128
129
137
139
141
143
145
147
150
154
155
158
161
163
164
165
166
168
171
174
175
176
179
182
183
185
188
190
191
192
194
195
196
00:00
Statistical hypothesis testingWorld Wide Web ConsortiumStatistical hypothesis testingComputing platformContent (media)Computer fileObject (grammar)Web browserSuite (music)Continuous integrationImplementationProcess (computing)Configuration spaceBitResultantPhysical systemSet (mathematics)Right angleAttribute grammarCase moddingComplete metric spaceRepository (publishing)Error messageServer (computing)Exception handlingCASE <Informatik>QuicksortPoint (geometry)Software bugDomain nameFunctional (mathematics)Network socketWeb pageScripting languageIndependence (probability theory)InformationProduct (business)Group actionDivision (mathematics)Element (mathematics)Block (periodic table)Level (video gaming)CodeRegulärer Ausdruck <Textverarbeitung>Computing platformDifferent (Kate Ryan album)State of matterNeuroinformatikSurfaceTraffic reportingSquare numberConnected spaceWeb-DesignerOnline helpSoftwareFile Transfer ProtocolSearch engine (computing)Green's functionRange (statistics)Communications protocolOrder (biology)Near-ringRow (database)Open sourcePatch (Unix)CodeExtension (kinesiology)Type theoryElectronic visual displayExecution unitSoftware frameworkPlastikkarteSlide ruleRevision controlSoftware developerDemo (music)WritingACIDMultiplication signLink (knot theory)Projective planeStandard deviationSeries (mathematics)Structural loadWebsiteGreatest elementNormal (geometry)InternetworkingLibrary (computing)Mobile WebSelf-organizationProper mapComputer hardwareMereologyMoment (mathematics)Software suiteOperating systemEndliche ModelltheorieBoilerplate (text)Flow separation9K33 OsaIdeal (ethics)Data structureTouchscreenNumberSystem callCoefficient of determinationText editoroutputEvent horizonBookmark (World Wide Web)Mixed realityCategory of beingFood energyLatent heatSound effectDataflowNetwork topologyBlogInterior (topology)Graph (mathematics)WordExponential functionJava appletBit rateState observerDevice driverVector spaceTask (computing)OntologyBridging (networking)SubsetFunction (mathematics)DivisorFlagWhiteboardEmailMessage passingIntegrated development environmentStaff (military)Database normalizationTheorySource codeFundamental theorem of algebraCuboidHoaxAddress spaceModal logicFeasibility studyRadical (chemistry)LengthLecture/Conference
Transcript: English(auto-generated)
00:00
James is working in the automation team and he describes his own job as I quote improving interoperability through testing so James has worked for Opera he is a British citizen but he also has a Swedish card and I've been told
00:23
that he was especially fond of hamburgers but the very good one so please welcome James okay we're just having some minor technical problems
00:55
getting the slides working but I'm here to talk to you today about
01:00
testing for the web so the web I think I hope everybody in this room can agree is is pretty great it's got a lot of things going for it it's vendor neutral you can implement it on any device you don't have to go out to someone and get permission to distribute your content on the web but it also has a number of
01:21
problems so one of the problems that I think we've we've really seen recently with the rise of mobile operating systems is that the web doesn't necessarily have all the features that other platforms have so the performance or it might not have access to hardware and there's a pretty
01:44
simple solution to that which is that you go okay we need to spit this this extra feature let's go and convince some browser vendors to implement it let's get together and write a spec and and let's do it so we know we know how to do that but there's another problem with the web which seems to
02:02
sort of be fundamental to the whole model that well if you've ever developed a website you've probably found that you developed it in one browser and then you tried it in some other browser and it didn't quite work and now that's a bit more of a problem and this is this is really something that's been going on for a long time you know ten years ago or
02:21
more we were getting best displayed in Netscape for or whatever and then we got best displayed in Internet Explorer and then best displayed in WebKit and Internet Explorer isn't going to work at all so this is something we'd really like to fix so what's the process for doing this well typically if we have bugs and we want to fix them and then the first step is is to
02:44
write a test so maybe the right solution to improving the web here is to write some tests that we can use to say well there's a bug we need to fix it and you we can all see what it is okay and people have written
03:01
tests before you might recognize these these are the these are two of the acid tests and they were written mainly by Ian Hickson who now edits the HTML spec and the idea was that they would test a whole bunch of things together so that when you implemented a whole load of different features and fixed
03:24
maybe some bugs that you had then you would render the acid test the way that it was intended so the top one is acid 2 and if you got that right you would get the smiley face and the bottom one is acid 3 and you would get this hundred out of a hundred if you did it right and they were great they really
03:40
encouraged people to think about web standards and compete on implementing features at a time when that wasn't really maybe as much of a priority as it is today but they also had a few problems so one of the problems is that an acid test is very sort of self-contained thing like once it's
04:01
finished it's done and you can't like go oh well I found this other bug I want to I want to add it to the test and make browsers all fail again because because they'd be pretty annoyed at you if you did that and so it wasn't really possible for people to contribute to them and also they they kind of set some weird incentives because people wanted to do well for
04:24
them for marketing they would often make quite shallow implementations of the features and just do enough to pass the test and this this was a particular problem with acid 3 actually because there were a couple of browser vendors really competing to see who could be the first to release a build that got 100 out of a hundred and and so there were stories of people literally just
04:44
putting in stub methods that sort of did you know return true for this but don't actually implement the feature at all so in that sense we'd like to do a better so the question is what what do we really mean by better so what would be an ideal test suite for the web well it should certainly be automated or
05:06
automatable and what I mean by this is that it should be possible to either from within the browser or by scripting the browser from the outside work out what the result of the test was without actually having a human
05:22
involved and so I'm in the tools and automation team so I kind of have a vested interest in this but the way that I like to explain it is that if we have an automated test we can run it a hundred times a day easily we can put it on our continuous integration hardware and run it a hundred times a day if we have a manual test that requires somebody to sit down and
05:41
actually do it we can probably run it once every hundred days because we just can't afford to employ all the people to run these tests and it sounds pretty obvious but actually this is something we've got wrong in the past so for example CSS 2.1 spec had a big test suite hundreds of manual tests impossible to run it took one person two full days to run it so the
06:02
result is that nobody does it should be vendor neutral so it should be possible to write a test in this and see if it passes in gecko and blink and webkit and ie and servo in the future and whatever other engines you care about and again it kind of sounds obvious you know the web is supposed to
06:22
be a sort of vendor neutral platform but actually at the moment every browser vendor has their own test suites and they're completely separate and they typically depend on proprietary features so for example blink and webkit have a type of test called a fast layout test which actually dumps internal data structures and then compares it to the expected internal
06:41
data structures and because they're different to the internal data structures of gecko we can't use those tests and the similar tests that we have that they can't use it should be possible to run in continuous integration so this sort of follows on from the point about automation it being automated and more than it should be possible to run in continuous
07:05
integration it should actually be running continuous integration we should do everything we can from the start to make sure that this is something that browser vendors will put on their test systems and run hundreds of times a day so we need to make sure that it's got a completely
07:21
replicatable environment so that when you write a test it's exactly the same as when the browser vendor runs it so they actually see the fail that they expected and it also means that we have to have a certain quality standard we have to have tests that always produce the same results so that we're not continually having to work out well did it just
07:41
fail because the test was a bit dodgy and the last point is anybody can contribute and this is kind of the central point of my talk that actually it's not just people that work at browser vendors that might want to contribute to this or people who you know work on browsers in their spare time although those are also great people to contribute but if you
08:04
develop a website you might be the best person to to contribute a test for something because it's quite likely that you will come across a bug at some point and you will see a rendering difference between two browsers and then you might be you know the the person in the world who has who has
08:21
found this bug and so it would be great if if you could also be the person that calls it to be fixed so that nobody else ever has to find it again so it sounds like what I'm describing is basically an open source project for four tests and so that's what that's what we've created so in
08:43
this in this case we is basically the w3c who are sort of coordinating the effort in the sense that they're providing some infrastructure for it and and they own this brand test the web forward you might have heard of test
09:00
the web forward before it's historically been used to describe a series of events that were originally run by Adobe where people would turn up and they would learn how to write a test for the web platform and then they would contribute their test so previously test the web forward was these events now it's actually the the sort of brand name for the whole
09:22
project and obviously it has a website test the web forward dog and so if you literally listen to nothing else I say for the rest of the talk but you remember test the web forward dog then that's really all you need to know because everything else I'm going to say is covered on that site in much more detail okay so it's an open source project we're running it like a
09:47
typical open source project we have a github page so a bit hard to read on the screen but there's a repository in the w3c organization called web platform tests where we have all the tests for almost everything there's only
10:04
two sort of parts of the web platform really where we're not collecting the tests in this repository one of them is is JavaScript itself because it's run at ECMA and it has its own test suite already we haven't taken
10:21
JavaScript tests and the second one is CSS and the reason for this is basically historical legacy there's a there's a separate repository for CSS tests and that's because they took testing seriously earlier than quite a lot of other kind of w3c working group type people and so they have
10:44
their own infrastructure which their tests depend on but nobody else is used and so we're working on migrating the CSS stuff over to this web platform test repository but we're not quite there yet and so as is typical for
11:02
forget help we just use the normal github workflow if you want to contribute to test or if you want to fix a test or something you just create a pull request and then it will get code reviewed and when it's alright then we'll integrate it into the repository okay so that's enough kind of background let's actually get on and write a test so as I've said it's
11:26
documented on test the web forward org test the web forward org it's another github repository you can contribute documentation fixes that's a very useful contribution if that's what you're into but to write a test we need an
11:43
example of something that's broken really because if you're a web developer this is this is the most likely kind of scenario that you're going to be in so the example I have here is is with an API called file reader and if you haven't used file reader don't worry the about the details of it you just need to know that it has this method called abort on it basically a
12:04
file reader is something you can use to access the contents of say a file that's been picked through input type equals file and in our example what we see is that we create a file reader and then immediately we we call this abort method and in Firefox and I mean blink based browsers two different things
12:22
happen so Firefox here we'll throw an error whereas blink based browsers will will just continue without throwing an error and so obviously these behaviors can't both be right so what are you going to do well you might then
12:42
think okay so which is the right behavior so you put file reader spec into your favorite search engine and you probably come up come back with something like this and it's it's quite hard to read on here but this is this is a w3c specification for for for the file API but there is a trap
13:04
typically the the search results that end up at the top of Google certainly are for what the w3c calls TR versions of documents and these are always out of date it really helps if you just think that TR stands for totally wrong so whatever you see it whatever you see a TR document you
13:25
should look for the bit that says latest editors draft and you should click on that instead because that's the only way to be sure that you're reading the right version of the spec so just to emphasize it okay this is a
13:41
little hard to read okay so then we find this the right version of the spec and we find a bit about the abort method and it says when the abort method is called the user agent must run the steps below if ready state equals empty or Freddy safety was done set result to null and terminate this overall set of steps without doing anything else well okay ready state it turns out here if
14:00
we follow the link that's in the spec is actually set to empty when the file readers first created and we haven't done anything that will change ready state yet so it's empty here and this first step applies and then it says so set result to null and terminate this overall set of steps okay so that that tells us that well it doesn't tell us that we have to throw an error so
14:23
clearly what Firefox is doing here is wrong and what the other browsers are doing is right so now we're going to turn our sort of miniature example into a proper test case that's ready for submission so this is our example from before just put into a document and the first thing we need is
14:42
is a little bit of boilerplate so we have this script test harness dot JS which is provides the sort of testing framework it's quite specific to this this test suite it's not something that people have really used in other JavaScript frameworks and that's basically because it's designed
15:03
with testing browsers in mind so that's why we haven't reused whichever other JavaScript library you've previously heard of and we also have this file called test harness report dot JS and that's basically to help browser vendors integrate this in their continuous integration system so that file is for them to do whatever they like with in order to report the
15:23
results back and then we have a little bit more we have this div ID equals log and that's just somewhere to dump human readable output so that you can tell whether whether the test passed or failed okay and then one more
15:41
thing that test needs is a title in this case we're only going to have one test on the page so we can just add a title element and that will provide the test title okay and then the last step or almost the last step is is to turn the the coding into an actual test it turns out test harness dot JS
16:04
provides this function test which itself takes it takes a function and if the function that you call it with runs without throwing any errors then the test passes and if it throws an error than the test fails so in this case that turns out to be exactly what we want so we can literally just strip
16:21
out all the the try catch stuff and we we immediately have our test and the reason for having this this thing where we pass in a function is that it allows us to isolate different different tests on the page from each other so that we can have multiple tests on the page and they won't interfere too much
16:46
so that's a valid test we could submit that at this point but we can do a little bit better and we can see a few more features so if we look back at the spec it said if ready state equals empty set result to null so it
17:04
turns out if we read about what result is here result is an attribute of this file reader object so if if implementation is really compliant with the spec then at the end of this process we should have result equals null so we can add a testing for that so test harness dot JS it provides a
17:27
lot of assert functions of the type that you might be used to if you've used a lot of other kind of X unit type testing frameworks in this case we just want assert equals and we just assert that reader dot result is not so
17:42
that's pretty straightforward if it's not no then this will actually throw a an assertion error and the test will fail of course we have more than just assert equals in fact one of the design goals has been to have quite a rich range of assertion API API so that instead of everybody having to
18:02
reinvent their own way of testing things we can just use pre-written functions so for example if the right behavior here had been to throw then there would be assert throws already provided for you when you would just pass that a function that you would expect to throw and you could tell it I expect this function to throw this kind of error and it would do all
18:23
those checks for you you wouldn't have to write any try catch blocks or anything and similarly there's things like assert reg X equals assert array equals and so on for more complicated kinds of checks so that's great we've actually there in just a few minutes gone from having a difference between
18:42
browsers to having a test we need to run the test and you might think well this is a bit problematic if I want to run it on my system I've got these past slash resources which doesn't really exist on my computer maybe so how am I going to check it well as I said before one of the big goals
19:05
that we had was to make the test system very self-contained so we actually distribute in the repository a web server and a web socket server that are specially designed for writing test cases and there's a setup
19:23
script so that when you run it you get them launched in the right configuration so if you if you go to the repository and read the readme file then basically all you have to do is you have to set up your exact hosts to
19:40
have the right domain names on your local system and then everything will work just like it would on a browser vendor systems so there's no like fiddling with Apache or mod PHP or any of this stuff that's that's a complete pain and then we can actually run our test and I don't really believe
20:01
in demos because they always go wrong so his his his one I made earlier this is it in Chrome where it passes and this is it's in Firefox where it fails and before you run out and file a bug there is already a bug filed and it's
20:20
in fact already has a patch in it so hopefully in the near future I'm going to need to find a new example to use in in this kind of talk because this one will be fixed okay so now that we've got a we've got a test and we've checked that it works we can submit our test at that point somebody for
20:43
example me or one of one of the other people that works on this will do code review of the test and eventually it will get integrated into the repository okay so my my example test there was a JavaScript test which is the best type to write whenever you can but sometimes you can't and you
21:01
need to test something that depends on layout and when you want to do that the process the test type we have is called a ref test what you do here is you make two different pages that when the test passes should render the same and when the test fails should render differently so typically what you
21:21
might do is say you have a test involving flexbox you would make one page use flexbox and say render a green square and one page use just normal positioning and render the same green square and then obviously when flexbox is broken the you won't get a green square you'll get something else so
21:41
that's how we automate tests that can't be simply automated through JavaScript but depend on layout and when we can't do that either we finally resort to manual tests but this is really a last resort and in the future we're hoping to replace all the manual tests with or as many manual tests as possible with WebDriver tests so if you have experience working with
22:03
WebDriver Selenium then you know we we have a job for you converting tests okay so that's what what you can do for vendors but what about vendors are they holding up their side of the bargain are they actually
22:21
running these tests yet well the news is is good although we're not all the way there yet so at Mozilla we now have these tests we have some of the tests running on production and we have the full test suite now running on our staging surface so as of last week and on Linux we're
22:42
getting like four or five tests that are unexpectedly erroring on each run on other platforms there's more so there's there's more work still to do but certainly in the near future we're going to be running all these tests and we we have scripts so that we can update it almost as soon as new tests
23:01
are accepted I've also seen work in WebKit and heard about work in in blink to get this running so it definitely looks like at least for the open source browser vendors that I know about sort of within the first half of this year everybody should be running this complete test suite on their on
23:22
their sort of continuous integration setup obviously knowing what say Microsoft are doing is a bit harder okay so the last point I have is is that if you're here you're a great person to write a test and you should really want to write tests for this test suite if you're a web developer then writing tests is a way to ensure that the web sucks less in the
23:42
future it's a way to ensure that the bug that you found today isn't still a bug you know six browser versions later and that's doubly true if you're writing some sort of JavaScript framework in my ideal world if you were writing a JavaScript framework you would have a policy that every time you you had to hack around two different browsers having different behaviors you
24:03
would write a test we would provide you with a link that told you when with what the behavior of that test was in the latest version of each browser and then as soon as it passed in all the browsers that you supported you could remove your hack and you could be completely confident that was
24:21
okay to do but if you're just an open source developer or and you're interested in getting involved in spec work or you're interested in getting involved in browser work in particular writing tests is a really great way to to learn about web specs and how they work and it's a really great way of getting involved in browser development without necessarily diving
24:44
straight into C++ code or something okay that's all I have to say so if anybody has any questions now would be a great time and please use the microphone when you ask questions so that it appears on the recording hi to
25:06
what extent is this testing framework geared exclusively around rendering and browsing display side of things and to what extent will it also cover things
25:20
like network connection handling say if FTP of HTTP think protocol differences like expect and continue newer stuff like speedy and all that kind of stuff okay so basically the idea is that with the exception of the
25:41
two examples that I had earlier we're prepared to accept I think any test that you can that you can write that is done in a way that is completely exposed to to standard it like web api's so if you can test it in a way that's browser independent and you know doesn't rely on specific features of
26:06
each browser to get low-level information out or something then then that's absolutely fine to put in this test as far as I'm concerned so I mean I would certainly be happy to accept like speedy tests or something I mean I
26:22
don't know how the the working group for that feel about it but certainly I think the best thing to do would be to put in this test suite yeah does that include performance tests as well as just functional tests so far we have
26:44
exclusively done functional tests so things that we can give a pass or fail to I think performance testing is is difficult so I'm certainly I believe in performance testing I'm not sure that it would necessarily fit well into this
27:05
but it's definitely something that I think a lot of us would like to have a more you know have a centralized place to put that sort of thing so I'm sorry it's gonna be the only question because we are a little bit late and
27:23
there are a lot of people who wants to come as well for the next next talk so what I propose you because I understand you have a lot of questions James I'm sure will be ready to answer to your questions outside of the of the room thank you dance thanks a lot