Visual Regression Testing
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 53 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/54871 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Visual systemRegression analysisStatistical hypothesis testingUniverse (mathematics)PlanningWebsiteHypermediaContent (media)Multiplication signComputer animation
00:42
RobotStatistical hypothesis testingWeb pageStatistical hypothesis testingCommodore VIC-20Regression analysisVisual systemFunctional (mathematics)Multiple RegressionSoftwareMathematicsLogicCross-site scriptingTemplate (C++)Service (economics)Integrated development environmentDisintegrationSoftware frameworkImplementationThresholding (image processing)Mountain passElectric currentCalculationDrop (liquid)Demo (music)Universe (mathematics)AuthorizationWebsiteAreaMathematicsTunisBuildingMyspaceService (economics)Group actionSineHypermediaYouTubePixelContent (media)AlgorithmSoftware developerMultiplication signFitness functionLimit (category theory)DataflowMusical ensembleStatistical hypothesis testingElement (mathematics)Different (Kate Ryan album)Sound effectUltraviolet photoelectron spectroscopyMatching (graph theory)Physical systemStatistical hypothesis testingMultiple RegressionSoftwareTemplate (C++)Problemorientierte ProgrammierspracheSoftware frameworkExecution unitSource codeUniform resource locatorSystem callLine (geometry)RoboticsGame controllerIntegrated development environmentBitText editorStructural loadCloningFigurate numberStudent's t-testFlagParameter (computer programming)Branch (computer science)Revision controlForm (programming)Demo (music)Plug-in (computing)Web browser1 (number)Flow separationClient (computing)INTEGRALFunction (mathematics)
10:30
Personal digital assistantElement (mathematics)CuboidLink (knot theory)2 (number)Windows RegistryMusical ensembleObject (grammar)Statistical hypothesis testingSource code
11:49
Statistical hypothesis testingSource code
12:13
Image resolutionWebsite
12:48
Statistical hypothesis testingStatisticsSource codeSuite (music)Wechselseitige InformationMountain passMobile appThumbnailWebsiteTowerSummierbarkeitWeb pageSimultaneous localization and mappingTraffic reportingStatistical hypothesis testingPoint (geometry)Source codeComputer animation
13:08
Web pageMotion captureMobile appAnnulus (mathematics)Duality (mathematics)InformationWebsiteComputer-assisted translationLink (knot theory)Computer animation
13:25
Statistical hypothesis testingSummierbarkeitMaizeWeb pageMIDIEquals signConvex hullMobile appUniform resource locatorComputer-generated imageryFinite element methodElectronic mailing listMotion captureGamma functionContent (media)MathematicsMusical ensembleDrop (liquid)Computer animation
14:10
MathematicsContent (media)Source code
14:26
Uniform resource locatorConvex hullStatistical hypothesis testingWeb pageMotion captureEquals signRule of inferenceWebsitePlane (geometry)DecimalComputer animationSource code
15:01
WindowSource code
15:26
Web pageComputer animation
15:52
Different (Kate Ryan album)Graph coloringSource code
16:07
Demo (music)Regression analysisStatistical hypothesis testingVisual systemSystem programmingWeb browserOperations researchContent (media)TimestampMereologyGame controllerMultiplication signWeb browserPhysical systemMotion captureArtistic renderingMereologyStatistical hypothesis testingResultantMedical imagingRepository (publishing)MathematicsPRINCE2Office suiteContent (media)Musical ensembleProduct (business)Covering spaceComputer animationLecture/Conference
18:24
TimestampRepository (publishing)Statistical hypothesis testingTime zoneSource codeFile formatRoboticsSoftware frameworkLattice (order)Lecture/Conference
Transcript: English(auto-generated)
00:00
Thank you. Thanks for coming by. I work at the University of Dresden. It's a media center. We are responsible for building our, for building the website of the university. And our plan site underwent what you can call it a relaunch. We have a new design and
00:24
we got it by an agency and our intent is to keep the corporate design just in place. And also not, also at the beginning of the project but over the lifetime of the website. So here are just some small facts about my university. It's one of the biggest
00:49
universities in Eastern Germany. We have about 35,000 students and more than 6,000 employees. And among those we have more than 3,000 editors. About our clone
01:06
deployment, our stack is somewhat outdated. So we are still uncloned for with archetypes. And you have also some figures about the load of our site. It's somewhat high traffic there.
01:22
We also have a heavily customized clone UI both for visitors and editors. And we also put high emphasis on the compliance on the corporate design. And also we really like testing so we wrote over 400 for our tests just for one site.
01:49
So, regression testing. So what's that all about? I have moved here a little quote from Wikipedia, the method of all credible sources.
02:01
So, regression testing is rerunning functional and non-functional tests to ensure that previously developed and test software still performs after a change. So, if it's not, once this happens, it's called a regression. So what could lead to a regression? I mentioned there are some reasons.
02:24
You can read some on your own. Basically it's many things that are complicated, you cannot control very well.
02:42
So, we know now about regression testing and what is visual regression testing. I put here a little example.
03:00
Let's imagine we have the site here with a picture and some text. And we have a designer which is doing its work, changing some bits on the site. Maybe somewhere totally unrelated place.
03:21
And now we have this change. So, do we have a change? Maybe you can spot the difference. Yeah, it's a line. Is it the same? No, it's not. There are some smaller changes in the line height.
03:46
So you can't make this difference by just looking by it. So we have to make those changes explicit. So, the worst thing that could happen is that your client notices those changes or the visitors of the site notices those changes.
04:11
So it's always a better idea that you use those changes beforehand.
04:24
So, what does the workflow look like for writing tests? So imagine having a designer having made some changes in the CSS or templates. And how does this affect the visual output of the site?
04:44
So we have many smaller bits of the sites. You can call it just elements of the site and you have to check that those will be the same as before. So what just happened?
05:04
You basically check with a reference screenshot and if it's the same, you are good to go. But if it doesn't match up, you have to decide if it's an intended change.
05:22
So the change the designer made was in accordance with the copyright design. Or if not, the designer has to make sure that those changes get revoked. And if there are changes, the reference screenshot has to be updated as well.
05:47
Alright. There are several tools and services at hand you can likely use. I'll put just some prominent ones here.
06:03
You can look those up on your own. But we didn't use any of these because those didn't fit very well in our testing environment.
06:20
So what about our testing environment? We use Jenkins. We run tests every night. We also use the robot framework. If you don't know the robot framework, it's basically a domain specific language which makes it possible to write human readable tests.
06:46
So normal people can understand those as well. And it uses Selenium framework that controls your browser. So you can imitate the user making clicks, filling out forms and so on.
07:08
And additionally we use the Selenium screenshots add-on which is written by Asko. And we also use the integration plug-in for Jenkins to use the robot framework.
07:24
We also test against the Firefox ESR versions. And also we have a policy to build test sites for manual checking.
07:42
So things that are happening in the develop branch are built every day. And when a release is due, we update it whenever needed. Yeah, what did we do? We introduced a new keyword for the robot framework.
08:06
We called it test versus reference. And what you can see here are the arguments it expects. So we have a CSS selector. We have a name to give this element.
08:26
We also have reference paths to tell about the location where the screenshot is loaded or saved. It's in the end, in practical terms, we automatically define the path.
08:45
So the tester doesn't really have to think about it. And we have one important flag which is update. It's a brilliant one and it decides whether to create a new reference screenshot or do not.
09:03
And if do not, just do the check against the reference screenshot. So what does it mean for the workflow? So when we know there are some changes that are expected, we have to set update to true.
09:24
Otherwise, in the normal case, we just let it false. Yeah, and how do we do the detection of changes? It's based on a work of our students. Shout out to Christian.
09:43
It's a rather simple but sufficient approach. Yeah, you can read it here. So it's basically pixel by pixel checking. And if any changes happen, we add those 40 pixels.
10:03
And if it's below a certain threshold, it's good to go. Otherwise, the test fails. But you can imagine any other algorithms there and just use it. So I have prepared a little demo.
10:24
I hope it works.
10:46
So yeah, right. This is one of our tests. Alright, we have an element which is called recommendations. Which is basically a simple box which contains links for interesting places on our site.
11:06
And this test creates such an object and publishes it. And yeah, we have some basic steps to just create this object. And in the end, we have this final keyword which selects this element and checks if it likes the reference.
11:33
And we also have this update set here which is currently false.
11:43
So it's checking against the reference. So let's call this.
12:00
I stripped down every test so it's just one. Yeah, you can see the magic happening.
12:25
So it might be some issue because the resolution is different. So you basically can't see that much. But imagine the whole site is also there and the user could actually see where it is clicking.
12:48
Alright, yeah. Nothing that much to see but I have this nice report here which made all those screenshots. And we have here our test.
13:04
We have just this screenshot for example but it's not the point. So this is the whole site. And this is a recommendations object. So it's just some cat here.
13:21
But there could be also offer links here on the site. And we have here our new keyword. And if we click through it we can just raise it.
13:47
Yeah. So this is a screenshot. It's a cropped out screenshot. So this is being checked. And this check here passes.
14:04
And now we do a little visual change. I keep things simple. I just don't change the CSS but I change some content.
15:03
Alright, it didn't work because I moved resize the window. Just run it again. Okay, this is different. I'm sorry.
15:59
There should be a screenshot which just tells the difference by showing the pink color.
16:12
But afterwards I can show you the changes.
16:21
So what problems we have faced and how we try to solve those. There are some tiny changes in the font rendering. We try to kept it to diminish those problems by using the same OS and browser as in the CI system.
16:44
So there are also parts that you just cannot control at the first hand. So namely timestamps. So we used for this example a free scan to just set the current date at the time when running the tests.
17:06
And if we have just other content, remote content, we just control our animations. We can't capture exactly at a certain time. We also can check out those parts.
17:20
And also some technical problem about storage. So you can use additional sub-module to save the images there. You don't have to pollute your repository with all the binary blob.
17:44
So what are the next steps? Maybe we can improve the workflow for selecting reference screenshots. It's rather cumbersome. We could also implement better algorithms.
18:02
And in the end we would like to publish the raw add-on so you can use it. Alright, thank you. So on the results screen it looked like it would tell you how much time there was in between clicks.
18:36
Can you also have it notify you between tests if suddenly now one takes a lot longer between interactions?
18:45
So you have basically the timestamps at every step. So you can track those time differences as well. I think it's possible. You said you have something like 400 tests, robot format tests.
19:15
When you run them all, how long does it take? Very long.
19:21
We have one big repository which takes just six hours. So we really have to use one source at night and it almost takes more than one night. So if everyone has some hints about speeding up robot frameworks, we are happy to hear those.