Effectively test your webapp with Python and Selenium
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 108 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21097 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2016108 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
Red HatDemonSoftware testingWeb applicationRight angleSoftware testingComputer animation
00:31
SoftwareSoftware developerSocial classWeb pageServer (computing)State of matterProjective planePhysical systemInformation securityProduct (business)CodeSoftware developerPattern recognitionResultantPhase transitionEndliche ModelltheorieVideo gameUnit testingMultiplication signBroadcasting (networking)Power (physics)Source codeSoftware testingSpacetimeForm (programming)Web pageObservational studyPoint (geometry)Asynchronous Transfer ModeSoftware engineeringObject modelBlogView (database)TwitterComputer animation
03:04
Division (mathematics)outputSocial classSoftware testingMereologyDefault (computer science)Level (video gaming)State of matterMultiplication signAlphabet (computer science)Order (biology)Server (computing)Endliche ModelltheorieInteractive televisionScaling (geometry)Social classWeb pageDivision (mathematics)Object modelArithmetic meanBitComputer animation
04:13
Data structureLevel (video gaming)Software testingElectric generatorLatent heatProduct (business)Parameter (computer programming)Software developerData structureIntegrated development environmentObject (grammar)Web pageDisk read-and-write headComputer animation
05:26
Software testingElectric generatorData structureElement (mathematics)System identificationTestdatenWeb pageObject (grammar)Social classSoftware testingMultiplication signState of matterQuicksortLatent heatCASE <Informatik>Element (mathematics)Set (mathematics)Web pageIntegrated development environmentTerm (mathematics)Server (computing)VideoconferencingSystem administratorComputer animation
06:33
Social classWeb pageObject (grammar)Software testingIntegrated development environmentParallel portGoogolFeedbackData structureSoftware developerSoftware testingWeb pageProduct (business)Revision controlPhase transitionBlock (periodic table)MereologyEmailCartesian coordinate systemBlogClient (computing)Multiplication signExecution unitCuboidVirtual machineIntegrated development environmentLimit (category theory)View (database)Sheaf (mathematics)Latent heatObject (grammar)Intrusion detection systemBitINTEGRALReal numberEndliche ModelltheorieIterationSet (mathematics)Medical imagingSoftwareData managementPhysical lawRight angleNatural numberComputer animation
10:57
Social classPasswordWeb pageLoginSoftware testingVariable (mathematics)Mountain passComputer fileTouchscreenDevice driverWorld Wide Web ConsortiumPoint cloudService (economics)DisintegrationInformation retrievalElement (mathematics)CuboidSoftware testingFunctional (mathematics)Software developerMetropolitan area networkQuicksortPoint (geometry)CuboidData structureVariable (mathematics)Web pageComputer fileDevice driverINTEGRALConnected spaceHookingSocial classPlug-in (computing)MathematicsParameter (computer programming)Integrated development environmentMultiplication signEndliche ModelltheorieService (economics)Traffic reportingFlow separationArithmetic meanLibrary (computing)Combinational logicIncidence algebraMenu (computing)Element (mathematics)Presentation of a groupSource codeOpen sourceError messageIdentifiabilityAdditionMetadataBlogOpen setProjective planeWeb browserBitCloud computingCodeSuite (music)Stack (abstract data type)Web 2.0Intrusion detection systemRepresentational state transferGraphical user interfaceLatent heatObject (grammar)Computer animation
16:27
Function (mathematics)Software developerCuboidRevision controlSoftware testingSocial classPlug-in (computing)Linear regressionObject (grammar)Web pageLibrary (computing)Open sourcePattern languageMathematicsMultiplication signDifferent (Kate Ryan album)Cartesian coordinate systemDigital video recorderCodePoint (geometry)Message passingIntegrated development environmentExterior algebraEqualiser (mathematics)Beta functionCore dumpWritingEscape characterElement (mathematics)CASE <Informatik>HookingWrapper (data mining)Goodness of fitVideoconferencingMarkup languageSource codeRow (database)Lecture/Conference
Transcript: English(auto-generated)
00:02
Okay, so let's welcome Andrei Koman, and he'll talk to us about testing the web application with Selenium. Is this working? Yep.
00:21
Cool. So, effectively testing, effectively test your web app with Python and Selenium. That's a mouthful, right? So what's this talk actually about? I've been working on a project for a pretty long time now, and I got to see, like from its beginnings, how the project itself evolved, like the code base,
00:43
and how the test around it evolved, because we reached some conclusions at some points in time, and we tried to improve what we had, and that's what I want to share with you guys, like what we learned from that, and hoping that you don't make the same mistakes
01:01
that we did. That cool? Okay. So, my name is Andrei Koman. I'm from Timișoara, that's a city in Romania, in the west of Romania. I work at 3Peller for PBS. That's the public broadcast system in the US.
01:25
You can find me at, find some of my code on GitHub, or follow me on Twitter. That's cool. How about you guys? How many of you have been working with Selenium? Whoa, quite a lot.
01:40
How many of you are like QA, software engineering tests, something, okay, a lot less. That's not, that's not a bad thing. So, well, let's get cracking then, right? We kind of have, like, I like to think of it
02:01
like the way the tests evolved in three phases, and we're gonna take a look at each phase, and see what we did right in each phase, what we did not so good, and how we try to iterate over that in the next phase. Okay, cool. So, this is how a test looked like in the first phase.
02:23
Bear in mind that this has been, like in the beginnings of the project, where the QA team didn't have that much support for development because we were in crunch mode to get features out. So, this is more like what the tests look like. Let's create a resource, think of a resource
02:40
like a blog post or something. Give it a title, right, pretty innocent. And then we had a test that kind of views that particular resource. Nothing out of the ordinary till now. So, let's take a look at how the page object model for Selenium looked like.
03:02
So, we were good kids, and we separated the Selenium interaction from the test itself, so we have like this page object model classes that model the page. And at the time, we were using Expat, so to get the title, we were doing like,
03:21
getting the div and going by specific ID, then, you know, go a level up, and it starts to get a little bit confusing, right? Cool. So, what this test was hiding, actually, that at the time we were using NoseTest
03:42
as our test runner, and NoseTest, when it starts collecting your tests by default, it runs them in alphabetical order. So, if C goes before V, you can use your test kind of like a means to create the fixture and the state for the test on the server, and then test that in a different test.
04:04
You can see how that kind of doesn't scale and contradicts some good practices of keeping tests separated and independent. And, yep, this is the,
04:23
kind of the problem with the way we use Selenium. So, we used the page object model, that was cool, but we had this really long expat that makes my head hurt when I try to read it, and whenever a developer came in unknowingly
04:43
and changed the way the HTML was structured for this specific page, well, that expat, that test got broken inadvertently. So, all these tests were running like on a environment. We were giving it, hey, run this on production,
05:01
run this on QA, run this on staging. But, yeah, you can imagine that if the tests were failing like midway, it's not that cool that you left leftovers on production. Okay, so, yeah, as I said, we wanted something better.
05:21
We wanted to stop using tests as fixture generators. We wanna move to something more robust in terms of identifying a Selenium element and interacting with it, getting its value, or interacting with that element,
05:40
like if it's a button, click on it or something. So, I think at the time we upgraded to Django 1.4, and then we had live server test case, so we said, yeah, let's give that a shot, why not? And then, all the problems with setting up fixtures
06:02
kind of magically went away because we were using the Django ORM to create the state we were going to test, and using tests to just do a targeted test to see, yeah, this is the title of the page where I can click on a specific element and the video plays or stuff like that.
06:22
And we weren't stuck in that kind of end-to-end test where you need to create something, maybe using the admin and then test it using the published state. So, this was like a better world for us, right? We can run tests independently, we can parallelize,
06:40
which we couldn't do without many headaches before. So, this is good, right? So, we also took note of how we were handling Selenium integration, and we started moving to something more specific. So, we started hooking into IDs or CSS selectors,
07:03
and that gave us also more readability into the code, like, you know, it's gonna be an ID called title, and it was also less brittle because once you moved like sections of HTML in your templates, it's not as likely to break the tests with that.
07:23
Cool. So, what's the problem with this kind of approach? It's not testing a real environment. Now, when you get that email in the middle of the day saying that, hey, we are a team that you're a client of,
07:41
so like we have a team that manages our image, ingestion, and resizing, right? And if they make a deploy, they kind of send us an email, hey, run some tests to make sure your production environment still works. Well, you can't really use these tests for that kind of thing because these tests spin up like a really bare bones environment.
08:03
You just create just enough data to do your test in them, and afterwards, you throw them away, and they're not really designed to test a real environment, whereas in the previous approach, we were actually testing like QA production and so on.
08:23
Cool. This is a good thing that in the page object model, we started using IDs, but then again, we could do a bit better here too, okay?
08:43
So, what we had, what we wanted to do better after all these two phases were separating long-running tests from short tests. You still like to have end-to-end tests that go
09:01
maybe through multiple parts of your application and test the workflow, but maybe something like if you have a blog and you just check that the specific page works, that's good enough for like a smoke test. Google does this, so in how Google tests software, they have like a section where they kind of chunk tests
09:21
in medium or small, large, extra large, and the small tests are the unit tests that the developer runs on his machine. The large tests are more like these integration tests, and an extra large would be like an end-to-end test.
09:41
So it's important to put a distinction between these and put time boxes in which a test with like all my large tests must run under five minutes. Running them independently, that's kind of a conflict between the first phase and second phase. So in the first phase, we can run them independently,
10:03
but we could use an environment. In the second phase, we ran them independently, but we weren't hitting any real environment. So, we started looking around. I mentioned we were using nose tests, but Pytest was like the new kid on a block
10:22
and made sense to try to look at what that has to offer. We put a limit on our Selenium suite, like it has to run in under five minutes, five to 10 minutes. Previously, our whole Selenium was running in 45 minutes,
10:44
and in the first iteration, so in the first phase, it was really hard to debug a test. So, if you wanted to, if it failed, like in the view part, you had to go through all the creation, and it was a very cumbersome and hard effort to debug.
11:05
So, also, what we wanted to make an emphasis on is decoupling the test from the HTML structure of the page and for this, we went a step further from hooking, so identifying elements from existing CSS selectors
11:23
or IDs in the page to setting up a convention between the development team and the QA team, saying, hey, this is a prefix which all the identifiers, so if you have a class like Selenium dash something, that's reserved for testing,
11:41
so for allowing testers to hook up to that, identify that element, and do stuff with it. It's important not to tie any CSS or JavaScript functionality to that because you're just gonna go loophole, and it's important to keep that, those test hooks focused only on that.
12:05
So, how does the test look like nowadays, right? The page is pretty much the same, except that we added stuff like a Selenium prefix to everything, so you don't go in and rely on
12:24
IDs or CSS classes that will put in on developers and maybe make that specific element red or pop out or something. And also, we kept the same page object model,
12:43
and that's pretty much it. And test-wise, because we switched to PyTest, we started using a couple of interesting plugins from the Mozilla Foundation. They open-sourced a few plugins that I'm gonna showcase
13:03
a bit later in the presentation, but they really helped us to ramp up on a new test suite. So, one of the first plugins that I tried out
13:20
was PyTest Variables. This is something that you can put in to your project and keep your fixtures or your credentials or something like that in a JSON file, and then pass that in into the test. This kind of separates your fixture data, your credentials from the code itself. And it has a pretty simple interface.
13:43
Pretty cool for the Mozilla guys to open-source this. Otherwise, there's PyTest HTML. It's a really cool plugin that is not necessarily tied to Selenium integration, but it provides you like with a test report, saying, hey,
14:03
this test failed here, and these were the values that it failed at. And if you were using it in conjunction with the PyTest HTML, the PyTest Selenium plugin from the Mozilla guys, it also hooks in and puts you a screenshot from when the test failed.
14:23
That's pretty cool. PyTest Selenium, this is the PyTest plugin that the Mozilla Foundation open-sourced. It's pretty cool, it has a lot of Chrome drivers, a lot of web drivers, sorry, Chrome, Firefox.
14:41
Fun fact, we initially ran a lot of tests on Firefox, and after switching to Chrome, we saw a significant performance improvement. And it was pretty cool that this plugin allowed us just to change a parameter somewhere, and then all our tests were running on Chrome instead of that. And also, they have support for connecting
15:02
with cloud-based testing services like Sauce Labs or Browser Stack. Sorry? Yeah. Yeah. I think for Chrome, you need to install the Chrome driver. Other than that, you need to install the Selenium package
15:23
in your environment. So what I was thinking for you guys to take away from this talk is, and what we're planning to do on the project going forward is leveraging the APIs
15:41
to create data fixtures. So if you have a REST API to create stuff and remove stuff from your app, try to leverage that, like creating a blog post and then delete it for test purposes. Adding additional metadata into your page.
16:03
So these are the, that convention between the development of the QA team that everything prefixed with something Selenium, it will be whatever, banana, or whatever you want. And last but not least, defining these test classes.
16:21
So I have large tests, I have small tests and I put time boxes around them. And when that specific class exceeds that time box, then you really should look at failing that whole class and looking at why all of a sudden my whole large test case is like,
16:41
maybe that's a smoke test, so it's running slower than before. That was about it. Thank you for your time.
17:05
Do you have any questions? Do you know of an easy way to get video recording of these test runs? Not really, I haven't toyed around with that yet. That's an interesting point.
17:24
I'll try to check up with the Mozilla program, maybe they offer that. If not, maybe it's something worth contributing to open source. I know in the past we were at a previous company, we were using VNC to record test runs and see why it failed.
17:41
To answer the previous question, I know Mozilla has something because they use videos in Pytest HTML, so maybe just check the documentation for that. I hope it's documented, if not open issue,
18:01
then maybe it will be. Any other questions? Have you looked into Splinter at all, and Pytest Splinter plugin?
18:21
Are you? I haven't, but I'm curious. It's a convenience wrapper library in Python for Selenium, which helps you write better asserts without going into the markup. So it's like, instead of putting .SEL markup everywhere, it helps you to write better tests that are less brittle.
18:40
So maybe just a hint, it makes it easier to write nicer tests. Cool, I'm curious to talk to you about that after all. Thanks. How did you version your page objects if you were making changes in the application?
19:02
How did you deal with having different versions of, and like regression testing with Selenium? Good one. So what we're trying now is trying to get an element, and if you can't get to that hook element, you skip the test, and the market has skipped and say that,
19:20
hey, this test was skipped, until that code reaches the environment you're trying to test. Okay, any other questions? Okay, if that's all, let's say thank you to Andrej.
19:43
Thank you.