Testing for a Better Web

Video thumbnail (Frame 0) Video thumbnail (Frame 1091) Video thumbnail (Frame 1658) Video thumbnail (Frame 11190) Video thumbnail (Frame 20529)
Video in TIB AV-Portal: Testing for a Better Web

Formal Metadata

Testing for a Better Web
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Poor interoperability between browsers is one of the main frustrations faced when trying to develop for the web platform. Solving this is essential for safeguarding the future of the open web, and requires a comprehensive web platform testsuite that is run by all browser vendors. The challenge of creating this test suite is being coordinated by the W3C under the "Test The Web Forward" banner. In this talk, I will present the current state of the test suite, how Mozilla are using these tests in their automated testing infrastructure, and explain how to get involved with improving the web by contributing to the testing effort
Process (computing) Java applet Interior (topology) Mixed reality Food energy Statistical hypothesis testing
Slide rule World Wide Web Consortium Category of being Computer animation Statistical hypothesis testing
Statistical hypothesis testing Point (geometry) State observer Suite (music) Greatest element Implementation Multiplication sign ACID Continuous integration Web browser Number Software bug Internetworking Computer hardware Computing platform Operating system Ideal (ethics) Software suite Data structure Endliche Modelltheorie Computing platform Physical system Task (computing) Mobile Web World Wide Web Consortium Standard deviation Moment (mathematics) Content (media) Database normalization Process (computing) Integrated development environment Vector space Blog Order (biology) Website Right angle Whiteboard Quicksort Resultant Fundamental theorem of algebra
Statistical hypothesis testing Suite (music) Group action Hoax Range (statistics) Source code Execution unit Exponential function Neuroinformatik Software bug Different (Kate Ryan album) Feasibility study Error message Exception handling Physical system Touchscreen Block (periodic table) Software developer Sound effect Bit Staff (military) Complete metric space Radical (chemistry) Message passing Process (computing) Order (biology) Text editor Quicksort Writing Web page Point (geometry) Computer file Open source Patch (Unix) Device driver Online help Continuous integration Event horizon Code Product (business) Element (mathematics) Latent heat Bridging (networking) Computing platform Traffic reporting Computing platform Address space Standard deviation Graph (mathematics) Demo (music) Information Surface Content (media) Word Software Personal digital assistant Web-Designer Network topology Library (computing) Code Length State of matter Multiplication sign Modal logic Set (mathematics) Function (mathematics) Mereology Proper map Bookmark (World Wide Web) Subset Bit rate Ontology Square number Electronic visual display Flag Cuboid Software framework Series (mathematics) Extension (kinesiology) Scripting language World Wide Web Consortium Email Flow separation Connected space Type theory Repository (publishing) Website Self-organization Normal (geometry) output Configuration space Right angle Resultant Dataflow Domain name Functional (mathematics) Server (computing) Implementation Divisor Link (knot theory) Web browser Theory Statistical hypothesis testing Revision control Green's function Projective plane Boilerplate (text) Search engine (computing) Blog Object (grammar)
I and Jens's walking in in the automation interior and he describes his own Java as I quote improving the inter repairability through testing so just as works for a pair of he is a British cities energy also has a Swedish job a and I have been told that he was especially fond of and Bulgaria's but there's a very good 1 so please welcome Jens the and
I had is having uh some minor technical problems getting there get the slides lightning but on it's thought you today about testing for the web so who on the web I think I hope everybody in this room categories is pretty great it's got a lot of
things going for it uh is vendor-neutral you can
implement on any device you don't have to go out to someone and get permission to distribute your
content on the web not but it also has a number of problems so to 1 of the problems that I think we've uh the we've a really no seen recently with the rise of mobile operating systems is that the way it does in history have all the features of the other platforms have uh so the performance or it might not have access to hardware and there's a pretty simple solution to that uh which is that you go OK we need this big this this extra feature let's go and cumbersome browser vendors to implemented let's get together writers back and let's do it so we know we know how to do that but there is another problem have with the way of which seems to so the fundamentals the whole model that I will if you've ever developed website you for we found the developed in 1 browser and then you tried in some other browser and didn't quite work and that's a bit more of a problem um and this is this is really something that's been going on for a long time you know that 10 years ago more we were getting best displayed in Netscape for or whatever and then we got best displayed in Internet Explorer and then press displayed in Web kitten Internet Explorer and and work at all and so this is something which we like to fix that so was the process for doing this well typically if we have blogs and we want to fix them and then the 1st step is is to write a text uh so maybe the right uh solution to to improving the way here is to rise and tests so that we can use to say was about we need to fix it and you we can all see what it is OK and people have written test before you might recognize these these are the these are 2 of the acid tests and uh they were written mainly by Ian Hickson you now it's the HTML spec uh and the idea was that they would test a whole bunch of things together so that when you were implanted all would have different features and and fixed maybe some bugs that you had a then you would render the acid test of the way that it was intended so the top 1 is i is acid to an if if you got that right you would get a smiley face in the bottom 1 is acid free and you get this 100 out of 100 but if you did it right and they were great they really encourage people to think about Web standards and and uh uh compete on implementing features at a time when that wasn't really a maybe as much of a priority as it is today um but they also had a few problems so 1 of the problems is an acid test is very sort of self contained thing why once it's finished it's dark and you can't let go all I found this on the board I want I want right it to the test make browsers will fail again because because they be pretty annoyed you if you did that uh and so it wasn't really possible for of peoples could contribute to and also they they kind of set some weird incentives because people wanted to do well for them for marketing and they were often made quite shallow implementations of the features and just give enough to pass the test in this this is a particular problem with acid 3 actually uh because the recover browser vendors really competing to see who could be the 1st release to build the got 100 out of 100 and and so do the with stories of people lately just putting in stop methods that sort of did you return true for this but don't have to implement the feature at all uh so in that sense we'd like to have a better and so but the question is what what do we really mean by better so you what would be an ideal test suite for the way well it should it should certainly be automated automated below and so what I mean by this is that it should be possible to uh either from within the browser or by scripting a browser from the outside work out what the result of the test was without actually having a human involved and uh so I mean the tools automation team so I can have a vested interest in is the way that I like to explain it is that if we have an automated test we can run 100 times a day easily we can put on a continuous integration however environment on redundant if we have a manual test the requires somebody's sit down actually do it but we can probably run it once every 100 days because we just can't afford to employ all the people to run these tests and it sounds pretty obvious this is something we've got wrong in the past so for example CSS to put once back at a big test suite hundreds of manual tests impossible to run it took 1 person 2 full days to run it as the result is that nobody does it should be vendor-neutral so it should be possible to write a test in this and see if it passes Endeca and blamed and Web kitten Aion observer in the future whatever other engines you you care about and again it kind of sounds obvious uh you know the way this was this sort of vendor-neutral platform but actually at the moment every browser vendor has the and test suites and 2 they're completely separate and they typically depend on proprietary features so for example Blinkin WebKit have topic task order of fast layout test which as he dumps internal data structures and then compares it to the expected internal data structures and because they're different to the internal data structures of vector we can't use those texts and the similar test the we have that they can't use but it should be possible to run in continuous integration so this also follows on from the point about the automation of it being automated and more than they should be possible to run in continuous integration it should actually be run in continuation integration we should do everything we can from the start to make sure all this is something that browser vendors will put on their their uh test systems are on hundreds of fun today so we need to make sure that it's got completely replicated will enviroment so that when you write a test if it's exactly the same as when the browser vendor
runs it so they actually see that the failed and they expected and also means that we have to a certain quality standard we have of test always produce the same results so that when continually having to work out what did he just say with the test was a bridge ontology and the last point is anybody can contribute and this is called the central point my talk that actually is not just people work a browser vendors them I want to contribute to this so people you you work on browsers in their spare time of those those are also great people to contribute but if you develop a website you might be the best person to to a contributor test something because it's quite likely that you will come across a blog at some point and and you will see that a rendering difference between 2 browsers and then you might be you know the the person the world to those who was found this blog and so be great if if you could also be the person to cause it to be fixed so that nobody else ever has defined it again so it sounds like what I'm describing is basically an open source project for 4 tests and so that's what that's what we've created uh so in this in this case where the is basically the w 3 saying and who sort of coordinating the effort uh in the sense that they're providing some infrastructure for it and and they have in this graph and test the word forward you might have heard of a test of our forward before but it's a historically been used to describe a series of events that were originally run by Adobe where people would turn out and they would learn how to write a test for the web platform and then they would contribute the test uh so you previously test revolt was these events now is actually the the sort of brand name for the whole project and obviously has a website test of our forward all and so if you literally listen to nothing else I say for the rest of the talk will be remembered test the way forward all uh then that's really all you need to know because everything else and it saves covered all not site in much more detail phone OK so it is an open source project we're running at like a typical open source project we have again page and so bit hard to read on the screen but others are repository in the w 3 c uh organization called web platform tests where we have all the tests for almost everything is only 2 uh sort of part of the web platform really where we're not collecting the test in this repository of 1 of them is is java script itself because it's run effect money has its own test already we haven't taken jobs protests and the 2nd 1 is a CSS and the reason for this is basically historical legacy of there's a there's a central repository of the CSS tests and that's because they took testing seriously earlier than quite a lot of other kind of w 3 c working group type people uh as so they have their own infrastructure which they test depend on that but nobody else is used and so we work kill migrating the CSS stuff over to this web platform test repository that the we don't like that yet and as you the as is typical for forget how pretty she's the normal Gerhard blood flow if you want to contribute a test of you want a fix a test or something you just create a pull request and then it will get code review and and when it's all right then will integrate into the roles of tree OK so that's enough kind of background there's actually get on the writing test at so as I've said is documented on test went forward old Testament forward all gets another Gerhard repository so you can contribute documentation fixes but that's very useful could contribution if that's what your intent the but to write a test with an example of something that broken really because if your web developer this is that this is the most likely kind of scenario is that the URI being so the example I have here is is with an API called file reader and if you have a new star reader don't worry that the about the details of it but you just need to know that 2 of it has this method could abort on basically a file reader is something you can use to access the contents of uh say a file being picked through input type equals 5 and in our example we see is that we create a file reader and them immediately we call visible method and in the Firefox and an link-based browsers to different things happen so Firefox able will throw an error and where is link-based browsers will uh will just continue without Friday and so obviously these pages from both the right so what you going to do well you might then think OK so which is the right behavior so you put file reader spec into your favorite search engine and uh you probably come up come back with something like this uh and it's it's quite hard to read on here but this is this is the w 3 c specification for some FIL thoughtful the file API but there is a trap typically the the search results that end up at the at the top of the Google certainly of full the w 3 c equals T all versions of documents and feasible ways out of debt it really helps if you just T Stanford totally wrong so whenever you see it whenever you see a R a document you should look for the bit that says latest editor's draft and you should click on that instead because that's the only way to be sure reading the right version of the spec as just emphasize that OK so this is a little hard to read OK so that we we
find this with the right version of this spec we find a bit about ball method and it says when the ball method is called the use rate you must run the steps below it ready 60 empty your Freddy say he was done so result and Alan terminators overall set of steps without doing anything else all OK ready stated turns out here if we follow the link those in the spec there is actually a set to empty when the file redispersed created we haven't done anything that will change ready state yet so it's empty here and and this 1st step applies and then it says so you set result to null and terminated several sets of steps OK so that that tells us the well it doesn't tell us that we have to throw an error so clearly what Firefox is doing here is wrong and the other browsers theory is right so now we're going to turn off all sort of a miniature example into a proper test case that's ready for submission uh so this is our example from before just put into a document and the 1st thing we need is is a little bit of boilerplate so we have this script test harness OJS uh which is uh provides the sort of testing framework it's quite specific to this this test suite of it's not there's something that people of used in other jobs script framers and that's basically because is designed for testing browsers in mind so that's why we haven't we whichever other JavaScript library you previously had of and we also have this local test on his report addresses and that's basically to help browser vendors integrate this in a continuous Integration systems said that file is for them to do whatever they like with in order to report the results back and then we have a little bit more we have this divide equals log and that's just some way to dump human readable output so you can tell whether to whether the test passed or failed the the OK and then 1 more thing that test needs a title in this case were needed have 1 test on the page so we can just add a title element and that will provide the test title OK and then the last step almost the last step is is determined by the coding into an actual test it turns out test harness the JS provides this function test which itself takes takes a function and if the the function you call it with a runs without offering any errors the test passes and if it frozen error than the test fails so in this case that turns out to be exactly what we want so we we can literally just strip out all the the try catch staff and we we immediately have or test and the reason for having this this thing what we pass in a function is that it allows us to isolate different uh different tests on the page from each other their so that we can I have multiple tests on the page and they won't interfere too much so who that's a valid test we could submit that at this point but we can do a little bit better and we can see a few more features so if we look back at the spec is said the it's ready state equals empty set result enough so it turns out if we if we read about what result is easier result is that abuse of this file reader of object so if a 1st implementation is really with the spec then at the end of this process we should have resulted was not so we can add uh testing for that so test honest IJS it provides a lot of a certain functions of the type you might be used to if you've used a lot of other kind of X unit types the testing frameworks in this case we just want to set equals and we just assume that we did not result is not so that's pretty straightforward but it is not null and this will actually throughout an assertion around the test will fail so of course we have more than just asserting equals impact but 1 of the design dolls has been to have quite a rich range of assertion API API so that instead of everybody having to reinvent their own way of testing things so we can just use pre written functions so for example if the right baby had been deferred then there would be a search throws already provided for you when you just pass that a function of the you would expect the throne you could tell a dies by this function to through this kind of error and it would do all those checks for you you wouldn't have to write any try catch blocks or anything and similarly there's things like this right exp equals at such a rate equals and so on for a more complicated kinds of checks so that's great we've actually there in just a few minutes gone from having a difference between browsers having a test we need to run the test and so you might think well this is a bit problematic if I want to run on my system i've got these past 2 us slash resources which doesn't really exist on my computer maybe as a how my going to check it well as I said before 1 of the big goal so we had was to make the test system very self-contained so we actually distribute in the repository uh the web server and a WebSocket server that of specially designed for writing test cases and there's a set up script and so that when you run it you get them launched in the right configuration so if you if you go to the of the repository in and read the readme file that basically all you have to do is you have to set up your except hosts the to have the right to domain names on your local system and then everything will work just like you would on a browser vendors systems so there's no like fiddling with Apache or more PHP or or any of this stuff that's that's a complete package and then we can actually run a test and I I don't really believe in demos to stay was the wrong so here is a is is 1 I made earlier this is it in a crime where it passes the and this is this uh in thoughtful travails and before you run out of and thought about 3 is already about filed and it's in fact already has a patch in it so hopefully in the near future and the need to find a new example to using in this kind of talk because this 1 will be fixed OK so now we've gotta we've got a test and we check that it works we can submit our test that point somebody for example meal 1 of or 1 of the other people look what's on this will do code review of the test and eventually it will get integrated into the repository OK so you might be in my example test there was a jobs protest test which is the best to write whenever you can but sometimes you can't and you need to test something that depends on a layout and when you want to that of the process the test of rehab is correct test what you do here is you made to different pages the when the test passed sources should render the same and when the test fails should render differently so typically what you might do is so you have a test involving flex box you would make 1 page uh use of fake sports and say render a green square and 1 page use just normal position render the same green square and then all this UN flags books is broken the UN it a green square you'll get something else so that's how we automate test that can't be simply automated through JavaScript that depend on land and what we can't do that either we finally result to manual tests this is really a last resort and in the future we're hoping to replace all the manual tests with or as many e-mail test as possible with driver tests so if you have experience working with where drivers of solatium uh then uh you know we we have job for you of converting tests OK so that's what what you can do for vendors of but what about vendors of a holding up their side of the bargain about actually running these tests yet as well of the news is is good uh the when all the way there yet so Mozilla we now have these tests we have some of the tests running on production and we have the full test suite of now running on a staging surface so as of last week and on Linux we're seeing like 4 or 5 tests that unexpectedly uh airing on each run on other platforms there's more so this was more work still to do the search in the near future we can be running all these tests in which we have scripts so that we can update it uh almost as soon as you tests accepted I've also seen working WebKit and had about working in blame to get is running so it definitely looks like at least the open source browser vendors by know about uh so of within the 1st half of this year everybody should be running this complete test suite on there but on those of continuous integration set up obviously no I will say Microsoft to doing is a bit harder OK so the last point I have is is that if you'll hear your great person to write a text and you should really want to write tests for this test suite if your web developer the writing test is a way to ensure that the Web sucks less in the future there is a way to ensure the bug that you found today is still about you know the 6 browsing versions later and that's doubly true if your writings something jobs that framework and in my ideal world if you were writing jobs that framework he would have a policy that every time you you had to hack around at 2 different browsers having different behaviors you would write a test that we would provide you with a length that told you when with where will the behavior of that test was in the latest version of each uh browser and then as soon as it past in although browsers that you supported you can remove your hack and you could be completely confident that was located but but if you just open source developer or and you interested in getting involved in spec work we're interested in getting involved in browser work in particular writing tests is a really great way to to learn about words specs and how they work is really a great way of getting involved in the browser no development without necessary diving straighten C + + code or something OK that's all I have to say so if anybody has questions now be great time and please use the microphone uh when you ask questions so that it is on the recording the hi them to what extent is this a testing framework gives it um exclusively around rendering of browsing uh and display side of things and to what extent Willie also cover things like uh network connection handling and say if the FTP and HTTP think political differences like that and expect continue and newer stuff like speedy and all that kind of stuff again so um basically the idea is that of with the exception of those 2 examples that I had earlier we're prepared to accept I think any test that you can you can write that is a done in a way that is completely exposed to to standard at uh light web API is so if you can test it in a way that's browser independent of and it's uh you know doesn't rely on specific the features of each browser to to get uh low-level information out or something and then that's absolutely fine but test for on so you know I mean I I would certainly be happy to accept light tests or something the idea I mean I don't know how the the Working Group that feel about it but suddenly I think the best thing to do would be to put in this test the yeah I mean does that include performance test as well as just Functional tests so far we have exclusively Don functional tests so things that we can give a pass fail today I think performance testing is is difficult and so you a lot I subtly I believe in performance testing uh I'm not sure that it would necessarily that fit well into this but it's definitely something that I think a lot of us would like to have a more you know you have have a centralized a place to put that sort of thing so so it's going in the dual the only question because only a little bit late and a lot of who belongs to come all as well as for the next the the next talk so what I propose you because I understand you have a lot of questions and gems sensor would be ready to answer of your questions outside of the of the room thank you dense centered at the church