Dockerized pytests
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33716 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2017108 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
Coma BerenicesSoftwareIntelPlug-in (computing)Demo (music)Software developerWindows RegistryComputer-generated imagerySelf-organizationLatent heatStatement (computer science)BlogDisintegrationSoftware testingCoordinate systemBuildingCodeOrder (biology)Computer configurationArithmetic meanBitVirtual machineProcess (computing)Computer fileSet (mathematics)Web pagePlanningComputer configurationSoftware frameworkConfiguration spaceMedical imagingRepository (publishing)QuicksortMultiplication signSlide ruleSocial classIntegrated development environmentSoftware testingCoefficient of determinationComputer chessParameter (computer programming)Point (geometry)NumberComplex (psychology)CASE <Informatik>Module (mathematics)Sampling (statistics)ForestParsingPort scannerFreewareLibrary (computing)INTEGRALInformation securityPointer (computer programming)Parametrische ErregungView (database)EvolutePlug-in (computing)Structural loadTask (computing)Scripting languageVirtual realityDynamical systemVirtualizationElectric generatorVolume (thermodynamics)Demo (music)Local ringWindows RegistryCartesian coordinate systemComputer animationLecture/ConferenceMeeting/Interview
08:46
Sample (statistics)Plug-in (computing)Point (geometry)Software testingVolumeSoftware repositoryLink (knot theory)CASE <Informatik>Computer fileService (economics)Forcing (mathematics)Virtual machineDirectory serviceData loggerLocal ringMultiplication signOperating systemMedical imagingPlanningStructural loadSet (mathematics)Software frameworkLogical constantPoint (geometry)Software testingContext awarenessIntegrated development environmentMessage passingRepository (publishing)Configuration spaceStatement (computer science)Content (media)Overhead (computing)NumberError messageDemosceneFunction (mathematics)BitPrice indexUniform boundedness principleRootVolume (thermodynamics)Computer configurationWordProfil (magazine)Distribution (mathematics)CodePlug-in (computing)Software repositoryModule (mathematics)Projective planeDebuggerMultilaterationLevel (video gaming)Sinc functionDemo (music)Tracing (software)2 (number)Variable (mathematics)
17:19
Software testingSystem callProcess (computing)outputVirtual machineParameter (computer programming)Instance (computer science)Scripting languageUMLComputer animation
18:22
Function (mathematics)Video game consoleChi-squared distributionMedical imagingInstance (computer science)Scripting languageSoftware testingException handlingComputer configurationComputer fileNetwork topologyDirectory servicePoint (geometry)Control flowRootLogical constantTheoryGoodness of fitSoftware frameworkCASE <Informatik>Form (programming)PlanningContent (media)Chemical equationVolume (thermodynamics)Source codeXMLProgram flowchartComputer animation
23:23
UsabilityFluxMultiplication signInstance (computer science)Metropolitan area networkXMLComputer animationLecture/Conference
24:10
Computer-generated imagerySoftware testingSet (mathematics)Point (geometry)Execution unit1 (number)Presentation of a groupProfil (magazine)Medical imagingMultiplication signDefault (computer science)Process (computing)Computer fileJava appletComputer animation
25:31
Multiplication signSoftware testingMultiplicationSet (mathematics)Software frameworkInstance (computer science)Lecture/ConferenceMeeting/Interview
26:00
Product (business)Software frameworkVisualization (computer graphics)String (computer science)Content (media)Software testingWeightData miningComplex (psychology)Wave packetSet (mathematics)System callProcess (computing)Encapsulation (object-oriented programming)Point cloudIntegrated development environmentCartesian coordinate systemOrder (biology)VirtualizationMultiplication signGastropod shellPrice indexInstance (computer science)Meeting/Interview
27:34
Software testingMultiplication signUniformer RaumQuicksortSoftware frameworkACIDLecture/ConferenceMeeting/Interview
28:03
Lecture/Conference
Transcript: English(auto-generated)
00:05
Thank you all guys for coming. I wasn't expecting this many people, but that's a good sign, I guess. First, let me start a little bit by introducing myself. My name is Gonzalo Rafuls, and I do quality engineering for Red Hat
00:20
in Czech Republic for its virtualization solution, which is over on the upstream. So during this talk, I will try to go fast and make it light. It's been a long week. So to begin with, I'm gonna be talking a little bit about, I'm gonna actually show you a simple test scenario
00:42
for running pytest on Jenkins. And we're gonna have a little bit of a high overview on pytest custom plugins and how pytest loads custom plugins. And then we will look a little bit at the challenges faced at the time of trying to replicate this Jenkins job
01:02
on your local environment with Docker. And we will look also at Docker as a solution for doing this. And then if we have enough time, I'll make a little small live demo if it doesn't fail miserably, but yeah.
01:20
Okay, now I'm not gonna be talking. I'm not gonna be, I don't want to intend to sweet talk you into start using containers. You should be aware of the security implications that comes with containerizing your applications. And we're not gonna talk about containerizing Jenkins. That's already been done.
01:41
And it's ideal if you already have some high overview on Jenkins, pytest, and Docker. And also, most people would think that this is about the Dockerpy library, which is for managing your containers through Python code,
02:03
but that's not the case in this talk. Okay, now, so why did I decided to talk about this and how did we came up with this solution for containerizing Jenkins?
02:22
Now, when I first came to Red Hat, I was given the task of replicating my Jenkins job or testing framework on my local environment. And for doing so, I was given a six-pages document
02:42
that it actually took me approximately two weeks to complete, and then as when I realized how big of a fool's errand it was to try and do so. Plus, I also realized that most of the people
03:02
on my team was having issues when trying to replicate their environments as well. But at the same time, those environments that were successfully working at some point with the constant evolution of our testing framework, those setups were easily lost.
03:22
So there had to be a way to keep up with this and have it centralized and make sure that no one had to waste two weeks of their time to set it up. Now, at the time, we were not using virtual environments.
03:42
Therefore, you had to install all the requirements from the testing framework on your local machine and that could have potentially caused some dependency issues on the libraries that you already had installed on your machine. Now, also the biggest benefit of this
04:04
was to be able to put a Docker image on an internal registry that you can run on your organization so that whenever a newcomer started working just by pulling the image would be able to start debugging straight away.
04:21
Now, how many of you have been using byte tests on Jenkins already? Okay, that's a good number. So there's a couple of things that you need to take into consideration at the time of trying to replicate
04:41
this Jenkins environment or this job run. So first thing, if the job that you're trying to replicate is parametrized, you should look at the parametrized options on your job, on the configuration of your job.
05:01
And here you can have all sorts of things and these parameters can be used by Jenkins in many other places during the job execution. In the case that I will show you in this sample scenario, this small sample scenario, we only have a parameter that will be used on one of the build steps that I will show you
05:21
further on. Now, you might also have distributed tests and you could also have parameters for determining to which hosts you want or on which hosts you want the job to run. So that's one thing to take into consideration. Then, if the job itself has any Git integration
05:46
and if you're, actually this is the most common practice so you probably do so. So it's important to check that all those repositories are gonna be used for our local environment.
06:02
So this is really important and we will actually link all those repositories to volumes on our containers. Now, finally I would look at the build steps
06:20
on the configuration of the Jenkins job and it will depend on the complexity of your testing framework but you might have, in this case I just have one build step where it's actually a dynamic bash script that takes one of the parameters that I defined before
06:42
for choosing which tests to run and you might have additional build steps that could, for example, in our case, generate configuration files that are used by custom plugins and you need to make sure
07:00
that all those steps are covered before you decide to run Pytest on your container. Now, with Pytest, virtually, oh, I see a gun. Okay, with Pytest, virtually any Python module
07:21
can be registered as a plugin. So how Pytest does the plugin discovery, how it checks all the plugins that are on your testing framework, it first starts by loading all the built-in plugins that were installed with Pytest. Secondly, it actually loads all the plugins
07:43
that are registered through setup tools entry point. I will show you a small sample on the next slide where I am actually passing one plugin that is outside our testing framework through the setup tools.
08:05
Third thing that it does, it pre-scans the command line for the minus p name option and it loads the specified plugin before doing the actual command line parsing of Pytest.
08:21
In this case, you can also choose to deactivate already registered plugins by prefixing the minus p option. Actually, I didn't expect this to show like that, but it's minus p, no semicolon,
08:40
and the name of the plugin. So that's how you avoid loading plugins that perhaps are causing issues on your testing framework. Next thing that Pytest loads, it looks for all the conftest.py files as inferred by the command line invocation,
09:01
and if you don't have any test bytes, test paths specified, it uses the current directory as a path. Now, you need to note that Pytest does not find conftest.py files in deeper nested subdirectories at tool startup,
09:23
so it is usually a good idea to keep your conftest.py files in the top level of your root project. Now, next, it does also load recursively all the plugins specified by the, there is this Pytest underscore plugins variable
09:43
that you can specify inside your conftest, so that is actually the last place where Pytest looks for custom plugins. So now, if you want to make your plugin
10:00
externally available, you may define what is a so-called entry point for your distribution, so that Pytest finds your plugin module. Entry points are a feature that it's provided by setup tools, and Pytest looks specifically for
10:22
the Pytest 11, I cannot see now here, oh yeah, here, it looks for this entry point, and it actually looks for this entry point, and it loads all the plugins that you have defined under this entry point.
10:41
All right, so. Now, when I first started trying to containerize this environment, I started by creating a base image with Fedora and a couple of instructions that I already knew
11:01
that had to be there. For example, one of the first things that I had to do was at the time of linking my local repositories to volumes inside my container, I realized that I had to include all these
11:23
repositories into my Python path, so I managed to do that by including the env instruction on the Dockerfile for creating the image, and later on, I started trying to install
11:42
all the requirements that were defined on our testing framework requirements.txt file inside the root directory of the testing framework, and it was then when I realized that some of the requirements that I was trying to install
12:03
with pip had header files from RPMs that were missing or were not included on any place, on my testing framework, because all these dependencies were either already included in the operating system
12:25
that was being used for running these tests, or being provisioned by Foreman or other services that were invisible to my testing framework.
12:40
Also, how I managed to see which plugins were loaded by Pytest at every point while I was trying to containerize these was by using the trace config option from Pytest, which actually shows you all the plugins that are being recognized by Pytest at the time.
13:04
Now, as you can see there, I have this run statement that I included on my Dockerfile. Now, it is recommended that you minimize the number of run commands, because in that way, it makes for less overhead
13:25
at the time of actually running your container. Now, when you're leaking a volume on your container to a local repo directory on your machine,
13:46
if you have the SC Linux policy in force, then the container won't be able to write to this directory, and in the case that it might be possible that your tests are generating log files that are saved
14:02
on that same repository. And if you have a SC Linux in force, the container will fail with a permission deny or an IBC message in the host C's log. So to avoid that, you need to change the context of the directory, in this case, the repository directory,
14:24
to that tag in particular, as we sandboxed file T. Now, this has become much more easier since Docker 1.7, because, as you can see on the second piece of code here,
14:45
I am prepending the set at the end, and since Docker 1.7, this statement automatically changes the context of the directory
15:01
that you are trying to link to your container, therefore, the container will be able to write to that repository. Now, further on, at the very end of my Docker file, I define these two instructions, the entry point and the CMD. Now, the entry point, it virtually
15:23
makes your container executable. How so? Because when you run your container image, the first command that the container will run will be the entry point, and the CMD,
15:40
it's any additional options that you would like to run with your entry point. But why is it separated like this? It's because the CMD is overwritable. You can overwrite that by adding options to your run statement for when you want to run your container, and therefore,
16:01
it makes it more flexible. Now, at the time of debugging your pytests, you might want to, I'm not sure, I use IPDB, for example, but unless you use
16:21
PDB or pytests on PDB, you won't be able to debug because pytests is capturing the standard output, so you need to run pytests with the minus S option so that you are able to use any other debuggers apart from PDB or pytests PDB.
16:44
So, I went through that real fast. Now, I will show you a little bit of a live demo and a small, this small scenario that I made with Jenkins. You can find the testing framework,
17:01
testing framework that we're gonna be using on this live demo on this repository. And the second repository contains the Docker files and the Docker Compose file that I am using for instantiating this Jenkins environment.
17:21
So, I'm not sure if I'm able to see here, but... Wait, oh, yeah, cool. So, this is a Jenkins instance that I have running on a container, of course, in my local machine, and this job in particular is a parameterized job
17:44
that takes only one parameter, and based on the input that we put there, it will run one test or the other, as we saw on the bash script before. So, I'll run a bad test and a good test.
18:01
One should fail, the other one shouldn't. Most likely, both will fail, but... Okay, so, we got a failing test here. And the other one. It's a success. So, let's say I want to debug this failing test
18:21
that is running on my Jenkins instance. Now, I already created a container image that will replicate my Jenkins instance and a container that I can use for debugging my byte test locally.
18:40
Now, I have some scripts here. So, if I run this script for running this bad test, I can show you what does this file contain.
19:01
And it's basically a running instruction for Docker to run with the FC24 container that I created with this Docker file. And basically, I am telling straight away to byte test which test to run.
19:22
So, now in theory, if I... This is the testing framework. So, in theory, if I need my bad test to include this breakpoint, then I should be able now to run my container
19:46
and I should be able to hit that breakpoint. And yeah, there you see. So, and you can... Because that actually doesn't...
20:03
It actually... Oh, well, yeah, it actually hits the breakpoint when an exception is raised. So, it actually, it stops the execution whenever a trace back is raised. Now, you might have non-fuelling tests
20:22
that you will still want to debug, so. Okay. So, let me show you... I got another script here as well. Let me show you. I have defined on my conf test,
20:41
I have defined a small plug-in, a small custom plug-in that is in the form of a mark, and I am using that on my good file.
21:03
So, basically, this mark is for including an option name on the byte test run that will look for any tests that are tagged with whatever I introduce on my byte test run.
21:21
So, if I go here. Now, yeah. So, if I run the good file now,
21:41
this file, in fact, has... As you can see, I am passing the minus E option and the good tab that was defined on my marker for that test in particular.
22:01
So, if I try to run this container and I change the label to whatever, it will actually skip the test because it's not the label for good
22:21
that we had defined on that test. Now, additionally, we can see with the trace config, this is also just running a byte test with the trace config.
22:41
On the trace config, you will see all the planes that have been registered by byte test, as well as the conf tests that have been loaded by byte. So, if I do this, you can see...
23:01
See, in this case, it's showing me that it has loaded the conf test from the root directory, which is on the container itself because I am routing my volumes. I'm linking my volumes to the main root folder on my container.
23:20
So, that's pretty much it. Now, yeah, so that's pretty much it. Now, if you have any questions, I'm afraid we don't have any more time. Do we? Okay, yeah, sure.
23:40
Well, sorry, man. For this container in particular? Yeah, yeah. Uh-huh, okay.
24:02
Yeah, sure. So, this is the one for the Jenkins instance that it's actually picking up the base image
24:21
from Docker Hub, this is Jenkins latest, and I include some other instructions for what I needed at the time. Then you got the Jenkins slave Docker file, which is just the Java container, base container,
24:41
and there I am downloading the Java agent from Jenkins for executing the file then. So, you can find this on GitHub. There, you can find the Docker files there,
25:03
and it should be. As well, you have entry points and stuff. Where are they? Ah, well, for this talk in particular, I didn't use any entry points or CMD, but it's like that, it's just like, you can only use CMD once, and the entry point should be your default command
25:23
to be executed with Docker. Hello, thank you for the presentation. It was very nice setup. Sorry, I cannot hear you. Thank you for the presentation, that was a nice setup. I have two questions. First of all, this looks like one of the benefits of this might be able to run multiple tests
25:43
from Jenkins at the same time. Is this something that you've tried and it works properly with the setup? Yeah, yeah, yeah, you can run any test that you have on your testing framework if you already have it set up like that on your Jenkins instance. So, this Jenkins instance was just queuing them
26:00
because that's how Jenkins was set up. Yeah, yeah, this Jenkins instance was just to show the two different places like the Jenkins job that is running the pytest and my local pytest run that will be able to debug a test. Okay, cool. My next question, can you tell me what that shell
26:21
you had, it showed whether the git was dirty or clean. It looked very nice, I liked that. This is all my set SH with some tweaks, yeah. Okay, thank you. Hello, thank you for the talk, it's very, very important.
26:42
Well, my question is like this, when it comes, in my understanding personally, when it comes to deploying your application, I can understand the benefits of using Docker. There are a lot of advantages like encapsulating all of the big indices inside your container so you can ship it as is or make your application cloud native ready,
27:03
something like, stuff like this. But talking about testing only, what is the real advantage of using the containerized solution with respect to other available solutions that, in my opinion, might be more lightweight solutions like virtual environment? Yeah, well. So have you considered this other solution?
27:23
Yeah, yeah. And if not, what is your opinion, Legan Dix? As I said before, given the complexity of our testing framework, this was the best solution at the time. And as I said also, I don't wanna preach or sweet talk you into using Docker or containers
27:43
for replicating your Jenkins instance, but this is actually saving us a lot of time for newcomers and also it's maintaining some sort of uniformity on our testing framework.
28:04
Do we have any more questions? No, then thank you very much Gonzalo once more.