Continuous Visual Integration for Rails

thank that and that you would do that and you have an interview 100 doc figuring here and we gotta go go go we have a lot of things to get through and so just a little bit about me you start and my name is

mike for Nike's and kernel founder of perceived and proceed I 0 which is a tool for visual testing some early Saturday shortly some of the things I learned over the last year about how the test apps pixel-by-pixel animal so the author of 2 regions Jason EPA serializes and spider blocks and if you see that those I'd above the talk you after an overview of the questions OK so it's

slowed some bread and so the the come in like 3 parts the problem and general solution and kind of how it works and architectures and methodologies and all the problems that come along with that and so let's start with the problem the so the problem is basically

that unit testing itself is kind of a solved problem and we have a lot of different strategies and techniques and technologies for testing the data of our systems for the behavior of our systems for functionality of systems and integration of artisans of other systems and and then testing as systems and smoke testing our deployments and we have like a a lot of tools and technologies for this right and the how do you test

something like that so I guess the color of the text has become the color of the button nor or that the button is now the the text now uh you know has 0 opacities and something's happened right and this so this was fixed by an issue and were another example here is a

is a for a for page of I used to work on and this is just 1 suppose look like it's pretty simple pretty straightforward we once a feature and then 4 weeks later we were told that our for for pays looked like this right you've all seen this right and of course nobody coppicing QA because no 1 q is the 4 4 page and and you know somebody this is a simple change something it is like moved a CSS file everything else worked on the before for page was formally broken so then I fix and then fixed look like this

and so you know the city is totally covered up and they individuate effects on mobile so you know you're you're still continued fixed and then I went and pulled hold slides for this a while back and look to the 4 4 page it was broken again right I reported this my team and so this in the business this is what we call a regression right and and

specifically is the type of visual regression right so how you Product Teams fix this today China the answers hire more people how do you fix these kind of problems with intense but that 1 of the interns doing there on the clicking on a lot right what they call the area of exploring looking for a specific word here to it thank you here and so Q is the big 1 right so and this can be this can be developer QA system you doing during absence can be you have to a engineers regulate can mean and can mean many things that part of the job of this is to find these kinds of things before they had production right and or you know that you get issues from your customers and you fix them after an acute is is very necessary but it's also very slow and manual and complicated right and it's also pretty impossible to catch everything right even in like a medium-size that with just a tens of models you have hundreds of flows and thousands of possible possible page states and permutations and constant of feature turn right there's there's a lot of uh development it's happening in these apps and you you you can catch all the stuff all the time and so it's also very expensive right QA you're spending manual human often engineering power paying for fixing these kinds of his or her actions so let's go back to this and this

button a problem and what's you mind my standard fixed so this would be like can I write a

regression test for this right I mean to be personal of testing error register basically everything so like what's was good try to write a test for this trait and so here's like and spectator

test that unit test this part of the apparatus simple things like it visits the home page and then it fills and some of the text box of the title and click the button right and then you expect that the page has new content on the page and so there's a problem here right like this test didn't fail that the button so technically works it's just visually wrong right and this manifests in tons of different ways so so I supposed to do here and am I supposed to assert that some like CSS computed style of the color of the thing or maybe that has a CSS class applied but that's not really testing the right thing so I'm just not do this right and no 1 can do this because no 1 wants a write a test that's this like fragile and inflexible right and especially that of the developing products and

so my normal approaches is very useful and so the pollen fundamentally is that

pixels the changing right but were often only testing what's underneath were testing all of our abstractions on top of those pixels but this is an important problem because the pixels are the things that users are actually touching and seeing an interacting with all the time and and to go further than that even with all of our current testing strategies methodologies we still lack confidence in deploys right you can have a million unit tests for all the different data changes in the world but if you move the CSS file or changes CSS you have to go look at it right enough to go check that and test and so both cells move on to the solution to the problem and I like to say that this is this is the solution differences all solution is not the be-all end-all all testing strategies that will make your you know your life perfect and that sort of a new new tool in the toolbox so the question I framing as and what if we could see every pixel changed in any you I state in every PR that we make right and so that basically is like you know what what can we do if we could test ah apps pixel-by-pixel so in order to do that and introduce a new concept you may or may not be familiar with these and called perceptual this called Peterson called visual this and this has been pioneered by many times abrupt slackening rule has done quite a bit of work on this integral conserves consumer surveys you should watch his talk it's about how he accidentally launched a pink dancing pony to production and then they ended up having to like you do this I'll testing in order to prevent that from happening and but so so what is a perceptual there's

a of essential this is relatively straightforward right given to images what's the difference between these

2 images right and what compute the delta between these 2 images and that can be that's right so all the red pixels

are the pixel the change between these 2 images without any context about what the event about right so you can compute this basically for any kind of image and so so how do we compute the there

was little atlas channel examples so shattered the differences in these 2 side-by-side and I will talk show the key difference if you right background color on the top the lost the link Catalan governance data but the right we got so this is the right and

you can immediately see all the changes in an image without having to sort of sift through right all these pixels little

change these are the things of attention as and

so Peterson 30 seconds what's going to keep it it's a pretty pretty straightforward right but X so I have these 2 images and just new older rats was open new and old OK so here are the 2 images right and this is this is just from the skeleton I like demo site so you can see there there's some differences in number would going make a pdf and see

what that actually it's so I have magic the library installed this is the image matter compare tool and compare all the neuronal store the image in diff . png not open that up in the so called we have our 1st Peter right like those are all the pixels of a change and by default it you know applies the the images underneath the makes a translucent and you turn things off you can positive bunch of different flags so this command to like fuzz factor if you don't care the pixels adjacent a certain amount of color are those kinds of things right so contributing Peterson cells are actually relatively

straightforward the so there's a couple of kids in real life there

and so if you try to figure out the difference between these 2 it might take your 2nd but there the difference in the speed if you can kind of immediately see that the do you agree to the Terms of Use section of this page is gone right it just no longer exists and and I kind of love this because this is a test for an error condition but it's it's basically like a back and change manifesting as a front-end failure right this is a rails form object that somehow gotten into a weird state that is manifesting as this sort of front and failure and you might have a test for this but this this form probably doesn't submit now right you probably can actually submit the form and so here's another example

here's a here's like a normal visual changes to this change much and I want like a new person got added to this page so the visual differs OK a bunch of thing

shifted around in like got refloated and you can sort of like

go back and forth like OK I understand that this this page has a new thing added to so so I have to learn how to like

PBS because it can be a little bit noisy right so for example this 1 they look the same but in the further you probably can't read that it says like if type of G queried unequal undefined slash units and so this 1 was somebody added a GM which happens to inject some scripts and the jam was in a broken state right so all the Tessa probably passing all the everything else is passing but the for the hasn't jumped in red and you can often can't catch these kinds of things without little test or you know looking at it but here's just up the

diff art like I found this in some this slow down and you know an image that shifted over just perfectly to create this like nice piece of art and totally useless but chemical and also

the brick priest signaling kinases if there are 0 pixel change that's really important for you right like in a classic refactor of your app in a pure refactor you're not changing anything that some these interacting with all the plumbing shifting around you're changing our architectures you're upgrading something but the actual thing that people are attaching more the API touching is not actual changing right so having a 0 pixel change PDF can be a really strong signal because you get visibility into knowing that nothing has changed in this page right I can safely operate this thing because everything is remaining the same and then you as a racket bigger and bigger you want all the dual sensory factors and for your code health right

so let's go right and this testing system in 2 minutes ready to go but so I had this out of this is different

or which is if you want a brand his talk

else top 2 years ago this is his act and

and random of you in the room but you Paladin exactly what is going to go back and write tests for your demo at from 2 years ago but were images of so area

so here are some uh sunlight

future spectral written for this apple today do simple things like you visit a page expected has some content you click a dialog you expect that the new thing is up and this act has just basic

behaviors it's just a stream of posts right and you can you can upload gifts and you can do simple things like you click submit a gift and it doesn't require animation pull that down and you can type stuff and there's a validation state like a bunch of things that we all do all the time right so so these tests for this

are are relatively straightforward and so let's just go like say a screenshot of the end of this the all I'm doing here is using the capybara uh screenshot say screenshots capabilities and this works with basically every driver that you have except rack tests and the most of their supporters so let's sort say that Moscow

run the tests R is just my bachelor's for bundle exec our spec that all of 3 and then you should all have that that by the way because it's really is the type of the time and so great we run the test was see there's a change here in open all that great so we have a screenshot of like what art state where a test look like in that state prior what AppleTalk in that test right and this is kind of what I call a complex you I state like you've click the button and sundry grand mission is fired in order to open up the top dialog This is not just like a static page you visited right and so but you also notice it doesn't quite look exactly like the P. we were looking at right this border

images all messed up and there's some other

things going on here so I will talk a little bit more about that later about why that section of was totally the same OK so so great so we've saved our old image

looks let's say that what's changing to new for the new 1 also change the background color of the star so here's a CSS socialist eliciting the background color by 1 pixel right and witches of

understated who go tests great so we have an

old and new great let's compare them and

stored in difficulty candy of the death of Hindi cool is a pdf right like all the background pixels of this page have changed and and you might think of this is just noise right but why would anybody care about a background color that you you can't see and but I guarantee you there's a designer in this room who actually would probably want to know if this changes right and they want to guarantee that there is a consistent color palette being used in that we develop exact sort of arbitrarily changing the background change when we think about the new color that we should use rather than using some consistency so I kind of don't discount these kinds of changes as just because you can't see them in the eye doesn't mean that they're not important the so

bright right also let's look do this and so simple users here are catching visual regressions right that's that the kind of obvious 1 but then if you start thinking about this more there's a lot of advantages for this kind of stuff like CSS 3 factors in deletions a big 1 right you're all terrified to delete CSS yes or no right and because it's it's very you don't know where that CSS assessors use you know what legacy parts of your after using CSS so what if you go add a little death uh this test to regression test to your top 50 pages now to delete your CSS and see what happens right and then you if you deleted and nothing changes on the pages you care about great you can probably delete that CSS and testing style guide especially testing living sound there's a pretty a pretty cool the use of this safe dependency upgrade so often your libraries you know there backwards compatible but they're adding new features so you want lip-reader libraries but upgrading libraries dependency is also kind of scary sometimes and you want build to especially if the libraries are providing from independencies of any sort of the providing JavaScript behaviors if they're providing and if your style guide is in its own genome in you're importing and you're pretty out versions like of upgrading dependency safely in having these kind of visual checks to be really useful and this regression testing for e-mails an awesome event those use cases thing testing the 3 visualization something of sort of started experimenting with recently and because testing the 3 is actually kind of hard right like you can kind tests and I'm not going to be 3 expert by any means but you can sort of test like that that data transformations are making like you're how your inputs transform your outputs and so how you expect a D 3 is going to be all visualize that but wouldn't it be nice to just below the like this is what it looks like I know that's right 0 look it's change is it still right you know that's kind what you really wanna testable visualisations and and then going further what I really want here is a visual review process along side coders and on top of the more about that but so if this was all so easy why are we all doing this right now right and endeavor was somebody has said that if it wasn't easy if it was easy when the hot and this is a gets really complicated right there's a bunch of problems and sort of hand wave over a bunch of of the problems that I saw a bucket them and in 3 different categories tooling and workflows and performance and non-deterministic rendering so on the tooling friends and it's it's kind of hard there some open-source projects that do this right now phantom CSS is a is a great example of 1 right and that it's represents all your visual changes as a ton of individual test failures right and that that's kind of a lot of information a lot of and failures for things that are just sort of it can choose a line between like something being flaky and image or a cheesy you that I wanted to be and like an actual test year or for example on you probably shouldn't have to require that your manually storing these baseline images like in your git repo right that that's a big workflow tooling process that most of us are probably is not going to do that that's a lot of work right and the performance 1 I think this is the big 1 across the spectrum of all the open-source tool all the proprietary tool all everything this is the big 1 that probably prevents us from doing this right now and the example I showed are a somewhat contrived right there the pre simple pages but in the real world you have some pages that are when there when you render a full-page screenshot of the 30 thousand pixels high 40 thousand pixels high and and that's not crazy right so rendering screen training that kind of page and uploading and and storing it can take 15 seconds as to render it in another you know 5 to differ so if you have if you a hundred of these tests that you want to do and all run serially that's that's 30 minutes you adding to your test we right and none of us want to do that you're fewer features that security features tracks already there already to slot right and there are already too flaky so that that's a hard 1 I think the performances action the biggest the biggest problem here and and the non determinis eGranary what'll talk about some sort of handing over the other problems if you are talking about this mom I would love to and so on monotonicity rendering from simply like there's a bunch

of things that change in browser try were not static pages sigh emissions the big obvious 1 right answer take this like pure CSS animations right if you visually diff this a bunch of times but what what what differ you going to get you know you might get this that you might get the you might get the stuff that these are useful to you they're just they're just kind of noise so so for example in Percy um what we do to do this is we

actually like freeze animations by injecting this particular CSS style into the into the page that tries to like sample again these emissions from happening so you can say nothing has changed and if you wanna know more about the birth of a post on board proceed I about how we actually do that and or or another 1 dynamic data is a big problem right

if you have anything on your page the changes as in your tests especially especially if you're going to see a visual the from those selected the kicker as a good example right and then you can you can sort of fix these was like fixed a data instead of faker data you can sort of like a move in a direction where you're you're having more like static deterministic things that you're using and test which I think is a relatively good under this is still the problem have some ideas about how to how to address this kind of thing and

so old test browser so like we talked about before this that we see on the right here is what was rendered by capping bar WebKit what you see on the left is like what and my Firefox and these are not the same thing as like another border image doesn't work and the you like you have the web font here is not a web font this 1 and then the problem this is that often the browses the old test browsers that were using and these are not really modeling in any fashion right like cadaver WebKit is an old forgone WebKit doesn't support these things if you if you can't and that's all it until the new like 2 . 0 version but didn't doesn't support these things to use a fork of chrome 14 from 5 years ago right doesn't render the modern web and they also has 17 hundred opened have issues that are like basically entry so so go for it and so that's that's a really hard problem right and

then some of the poems like you can really control for is this sort of like subpixel anti-aliasing problems uh the way the tax is represented on a page is not totally deterministic right these new things like shift by 1 pixel GP don't actually guarantee you that in some ways that like floating point operations will always come out to be the same thing right so if you have a gradient that's rendered on 1 machine and you try to render that same gradient on machine they may not be pixel perfect they probably won't be and if you compile uh some code with different optimization flags GPU floating point operations will be non non deterministic right so the we look at it is as if they are the same all the time but actually eating under the pixel perfect as of the problem and some some tools attempt to solve this with some more sort of like you know open CAD CAM Computer Vision Research things where you suppressor like 0 is this a button has the button moving sort of try to derive the page back from the image right and so

that's that's hard and so user only

half the battle here right and so back to our back to our main goal like what if we could see every pixel change in any you I state in every PIR right and this is really what I think is the difference between like this or regression testing sometimes and what I frame as continuous visual integration right in the same way that like your unit tests or not the entire thing you're going to test your system and then you need processes to like the doing continuous integration you need to be merging changes with all the other developers all the time you need to be testing instantly in CIS fasteners parallelisation units paralyzed as possible and there is a difference between like doing this erection testing sometimes and continuous visual integration and these sort of the the big problems spaces that the creator and so so that would require being only do this would require that things are really fast right there's a basically as fast as a test suite you can you need it will handle complex UI states you can just test a static page were not just here to just like the look at all of our static interval test components and all the different components states and you need and easily continuously integrated into your workflow on basically every commit fresh in my mind this can be saved and tell your reader in production and even even stating is like a little bit too late for me right like I want this to happen basically all the time so entitled then the last part of this talk about how it sort of architected person to to try to address these problems and so here's like his

enter it into like our spec feature spec it's basically the same thing we created right you have a feature specter visit some pages does some some action on the page and and then what you do is you just drop and you know pretty candy bar snapshot the page given a name say this is the home page right so

that's what they're actually doing underneath right when these these things depression to bursty like are we are we pushing up images and I say that with question marks because of those will come along with all the problems that we notice before so we we don't wanna do that and so what we actually do here is we push up Don snapshots and if you think about this like it makes a lot of sense because the most efficient most like lossless encoding format for a website is not an image of the ones that are of the view of the website of the website right at your assets it's you're on state you've created and so a push up the domination of snapshots and technically put push up uncharted 56 fingerprinted versions of those assets so we actually never upload things twice so the 1st thing might differ from might be slow but then after that it basically my my goal is basically the say like you know 0 time is spent in your test we after the 1st round the ontology and so

then we do a bunch of like can we be magic underneath that to actually say like and you know we process of a storage we can talk to get help and sediment statuses we according work with the Cyprus he held and actually this is the big part what that actually addresses most of the performance issues as we can paralyze that's right so you've push a supplement of Don snapshots as fast as the test we can go and what we actually do underneath is we run as fast as you concurrency when the last so we can actually totally out of band of your test suite the paralyzing and running and rendering these uh these Don snapshots in deterministic rendering environment and then be able to like you know show this to you in a nice way so this this was the sort of like mean innovation that helps I think this thing come version and so as of

yesterday actually want to like talk about this we had we have had a million little this rendered in pretty as of yesterday so really proud of that now stand and so here's a couple quick Person example that I talked to excite our customers get permission to show you just a couple of a couple of pages to see the sidebar

precede the product looks like and have sort of and try to address this problem so here's here's charity waters build rewarded Oregon they do this amazing they're very design centre team the big big Rails apps they on on you know they've pushed 162 total snapshot on every build basically and this this particular bill which is called for the updating the new for the market that had 96 little this right and you can sort of like go through each 1 of these pages and is

a look with all these collisions changes and then this is the best thing I can click that and say like 0 great I

notice that this like for the is different on all of these

pages and so there's a lot to go through right so this recently added this like Overview mode work in this like see all of my pages all at once and the

like OK and this confirmed like really quickly judicial review and confirmed all these these changes are the ones that I want right these are the little changes that i've had that I have

actually like you know we want to make as part of so here's

here's another example and so this the stages basically like were updating you know that the PR is new Press creation were trying out a pages and this 1 is just the 1st iteration of that Pr were like they remove some CSS styles and Michael look this pages told the breaking the head is totally broken and and I would never want to like launch this page but it isn't the sort of like iterative review process where the can go here and I can say 0 look at US law case looks like currently in this PIR and then also the important signal of none of the other pages have changes have changed based on this and assess change and you can go through and you can sort of see like what are the other pages here and in the sense but so the on like workflow and tooling and problem so this is the last thing I'll show you

and so we just providers like a CI service right we just like as a test run they actually push information to person and person marks this pr with another cs CS services here and perceive visual that's found right and we can just click details jump right to the page and they're

like 0 look this is the state of the sky are this is that background handling and I can go through I can

decide like yes this is the right visual change that was intentional for this pr looks going market so the little this is the 1 http

request that my demo my talk requires

sorts of the source and so OK

so like I got here I'm like great this is undoing of his review right from that have a lot of these things this is what I want quick approve and then get will you know mark that status is green right so now this sort of gives you like a light wave visual review process for all of the different UI states of your app and at the PR level right not like that some some later stage and so yeah that's that's basically

what's and this this Don snapshotting uh mechanism has helped the sort of like tackle bunch of those different problems so that's it I decided to it in this talk that like little testing is possible it's a thing we should be doing it and it's a new way to think about testing and it can help give you deployment uh confidence right I think of this is like the last stage of the CD pipelines where you just need a like in your acceptance phase you need to make sure that all this stuff is looking correct right and you need to build approved and and that this is still very manual step but we can probably Ottoman quite a bit of this and then also there's like a lot more work to be done to make this a mainstream engineering practice the 1 last thing

and so because of the stance naturally model and able to respond to be a sneak peek of something and working on over the next couple of months and I wanted to do this for regression tests offer for Amber test so if you are an every user I would love to talk to you on this you know me my could proceed I O and let's talk about like getting as a beta tester of this protest because I to think that this is probably the world where this makes the most sense right like know not everyone is writing rails regression tests the rate rose future specs always to their they're really hard to right and sometimes and but we are of writing a lot of JavaScript test nowadays right and as we saw before the separator worlds to like this is just an EPI back in and this is like a single page at front-end and those lines become clear in the ram lot more of these tests and so and to be able to get this kind of power all we need to do is build so that those those non snapshots and and random and so if you're interested not please let me know and I can like about to get your your hands on the beta so think so

much few yeah the questions like what you what is the baseline mike how's the baseline created and so basically anything you want different ways I usually just pick like master like whatever master has last created that is our baseline bread and and then we provide a mechanism to pursue you can select I want more manual version were actually like approve a master bill and that becomes the baseline so I think it's you gotta have cannibal but yet basically I think if if you're really doing like master is always green always deployable in geology testing yes so we don't right now right I think and we don't know the question was do to cross-browser testing so I think that's it that's it would be a great evolution of this kind of testing I doing doing more cross-presentation because of all those problems I mentioned to you and you'd be surprised to learn that most browser don't provide a full-page screenshot idea and then Firefox is the main 1 so I think that you can get like 90 % of the benefits of visual testing with like 1 good modern browser and then that would be a great evolution of this kind of idea would be to do cross-browser testing yeah that's a good customer questions like What tech reasoning and it's all custom built on Google cloud platform I've like stock arise all of the environments it's basically a rails API of full Amber strictly amber front end and the workers like Ron XVI which is a virtual framebuffer at all on 1 and so on Firefox yeah 25 0 0 and 0 the about Percy access control can access that perceive it right now I just tired to like have off so it's if you can see the repo and have you access like team collaborator axis of the day everything you can see and perceive and anybody who can see that with the I have built a new like a complex like role authentication kind of things and yet I totally missed that part so let me just do that quickly for the people who remain OK so so

part of this thing is we have all of these different much screenshots of a particular red right but we have the original Dom of these so we can just resize the browser to a smaller with the natural show here's like responsive testing so here they have a 320

px version of this so now I can see the for the change in all those different ones right so

and I might forward this like this is what this page looks like you know quote on mobile

basically just like at this that this breakpoint size right and so the the dance Action a mouse

or model also takes care of that in that you can decide rendered from what is is not testing on the actual device right but it is like is giving you at least the response size and the question was do this can you disable local test-run only haven't said that after the default behavior and and I had some some people as like I wanted to say literally only specifically this branch and so there's an environment variable we provide called per se per enable which you can set to 0 or 1 and it will force anemometry on or off got takes much the uh

might also on my