Python in system testing
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 110 | |
| Number of Parts | 119 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/19993 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Berlin |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 2014109 / 119
1
2
9
10
11
13
15
17
22
23
24
27
28
41
44
46
49
56
78
79
80
81
84
97
98
99
101
102
104
105
107
109
110
111
112
113
116
118
119
00:00
CodeSystem programmingSoftware testingComputer animation
00:21
Service (economics)Data managementKey (cryptography)Sound effectSoftware testingRevision controlCASE <Informatik>Level (video gaming)Software developerEnterprise architectureReal numberSoftwareVirtualizationExecution unitPhysical systemLecture/Conference
01:45
ArmSystem programmingMetropolitan area networkCovering spaceIntegrated development environmentPhysical system2 (number)Moment (mathematics)CASE <Informatik>Multiplication signCodeSoftware testingProduct (business)SoftwareFraction (mathematics)Virtual machineCartesian coordinate systemReal numberTouchscreen
04:05
Normal (geometry)Metropolitan area networkOperating systemMultiplication signFormal languageSoftware engineeringCodeInstallation artSoftware developerPhysical systemKey (cryptography)
04:50
Formal languageNatural languageJava appletLibrary (computing)
05:20
Maxima and minimaMetropolitan area networkSoftware testingDensity of statesSoftware testingLibrary (computing)Physical systemBootingStatistical hypothesis testingVirtual machineSource code1 (number)XML
05:57
Ordinary differential equationMetropolitan area networkGrand Unified TheoryData storage deviceLatent heatStatistical hypothesis testingVirtual machineDimensional analysisSoftware testingLogic gateSeries (mathematics)Installation artProduct (business)Real numberRepository (publishing)Configuration spaceXML
07:05
Metropolitan area networkReal numberSoftware testingState of matterStatistical hypothesis testingWebsiteCASE <Informatik>Category of beingEndliche ModelltheorieSoftware developerOpen setTime travelPhysical systemIntegrated development environmentSource codeLibrary (computing)Moment (mathematics)Virtual machineResultantData managementGraphical user interfaceMereologyRepresentational state transferJava appletXML
10:20
Metropolitan area networkValue-added networkDensity of statesSoftware testingLength of stayBefehlsprozessorDataflowElectronic mailing listStatistical hypothesis testingData managementOnline helpMathematical analysisField (computer science)Conjugacy classSoftware testingSet (mathematics)Quicksort1 (number)CodeFunctional (mathematics)Remote procedure callDirectory serviceSuite (music)Computer fileLibrary (computing)Unit testingXML
12:05
Metropolitan area networkConnected spaceSoftwareMultiplication signTable (information)Statistical hypothesis testingSimulationMathematical optimizationEndliche ModelltheorieComputer simulationMiniDiscStructural loadThread (computing)Heegaard splittingError message
12:35
Arithmetic meanEvent horizonLibrary (computing)Software testing
13:05
Metropolitan area networkDensity of statesSoftware testingTotal S.A.Set (mathematics)ResultantBlock (periodic table)MereologySource codeOrder (biology)Integrated development environmentMathematicsCASE <Informatik>Point (geometry)Software testingConfiguration spaceProduct (business)Service-oriented architectureInstallation artXML
13:57
Process (computing)Total S.A.Physical systemInstallation artProcess (computing)Configuration spaceLevel (video gaming)Plug-in (computing)Software testingProduct (business)State of matterObject (grammar)
14:50
PermanentMetropolitan area networkSpacetimeService-oriented architecture
15:07
Metropolitan area networkType theoryCondition numberShared memoryiSCSILevel (video gaming)
15:35
Metropolitan area networkMomentumMedical imagingSubsetSoftware testing
16:05
Metropolitan area networkDisintegrationSource codeState of matterRight anglePhysical systemDivision (mathematics)Revision controlEvent horizonIntegrated development environmentMereologyElement (mathematics)Software testingINTEGRALBacktrackingBootingProcess (computing)Software bug
17:50
Independence (probability theory)
18:25
MiniDiscServer (computing)Multiplication signRoboticsRule of inferenceBasis <Mathematik>Right angleComputer animation
19:13
NamespaceTable (information)FamilyUnit testingExecution unitShape (magazine)Series (mathematics)Open sourceView (database)Slide ruleState of matterINTEGRALHybrid computerDataflowSoftwarePoisson-KlammerSoftware frameworkDimensional analysisEvent horizonParticle systemFunction (mathematics)Condition numberOpen setSoftware testingCodeVirtual machineServer (computing)Remote procedure callLecture/Conference
23:04
Googol
Transcript: English(auto-generated)
00:15
Hello everyone, I'm Katarzyna, I work for Red Hat and I work in the QE department.
00:24
I'm not a software developer, I'm a QE person and I will talk about how we test our system and it's not about every Red Hat program, it's only about my team.
00:44
So it's only about Red Hat Enterprise Virtualization Manager, so this is a downstream version of Oviatt. So this is a tool which helps you to manage your virtualization. So on which level we are.
01:05
Imagine that you are a developer, you develop your application, it tests all your unit tests, it tests all your functional tests, it was already built, so you have RPM or in our case it's RPM,
01:22
maybe you have another packages, you want to distribute to user, but before you do that you want to test if everything is okay. So you want to test how it will work on real system, in real cases and so on. So you give it to us and what we do?
01:52
We develop your application on real environment, we try to do it on as production system as possible. So for example this is system for virtualization,
02:10
we will learn what we can on bare metal machines, not on virtual machines. We will use official APIs, we will not take a look on your code at all or almost at all.
02:30
We will simulate user, more or less we will do what your user will do and we will test both negative and positive cases but we will do
02:49
much more positive. It's like okay there's a problem if something is broken but it's much bigger problem if you cannot do some case.
03:02
So what are problems in such situation? First of all we touch environment, actually we break this environment and we do it on purpose. We want to test how the system will behave if we break network connection, if we break disk, can it recover?
03:28
So the problem is you have to maintain this environment and it's hard. What else? It takes time, you have to set up this environment, it takes time, it takes,
03:43
it's not a fraction of seconds, it's minutes, sometimes it's half an hour. Then you have to run your test, it is quite short usually and then you have to clean this environment and because this test may fail at any moment you have to have the screen up
04:05
quite well. You don't want to reinstall everything starting from the operating system every time. Okay so and of course we want to automate it. We may do it manually but imagine that you will have to do it every week the same, install everything, click on it,
04:25
click on the other thing. So why we want to use Python? First of all it's easy. Quite a lot of people in QE departments are not software engineering people. They have some
04:40
background in IT, quite often they are from admin departments, the only language they know is bash and you have to somehow force them to write code. You don't want them to, you don't want to teach them Java or C++ or anything like that. Python is easy, they can
05:04
understand it. On the other hand it's a normal mature language, you have libraries for almost everything and it's popular. So if you need people who really know this language, for example
05:21
you want people who will write tests and you want people who will write libraries and so on and for these people you want someone with coding experience and finding someone who knows Python is not that hard. So what one such test does, it must get resources.
05:47
You don't want to run your test on the same machine on which your system will be developed because quite often you will want to for example reboot this machine and test what will happen.
06:03
So you have to get machine on which you will run this test code, you have to get test definition, you usually keep it in a repository like git and you need the resources, the real resources
06:22
like bare-metal hosts or virtual machines or some kind of storages. Then you have to install product packages, you have to take them from somewhere, so you have to specify from which repository you want to take. Then you have to configure this product and your test may want
06:49
to change this configuration. Then there's what's specific for the test, this is a setup because for example when you simulate a user and test if you can add virtual machine
07:07
to your installation, you have to do everything which is required for the step. Then you have to run this test, then you need to prefer teardown and this part quite often is more
07:28
complicated than the test itself because these two parts, setup and running test,
07:42
they may fail at any moment and it may leave your environment in a really crazy state and you want to clean it up. Then of course you have to collect logs, you cannot okay you don't run this test manually, you run them from Jenkins overnight and
08:06
when you come to work in the morning you have results of these executions but you cannot assume that you will have access to these machines where you run this test. You have to copy every log which may be important and you have to copy it. You have to then
08:28
clean the resources because you will run next test and you have to report results. You keep your tests in some kind of test management system so you have to take execution of every
08:47
test and see that it passed on this release, failed on this release, failed because of something. Okay, so how we do it? We use tools which we found whenever we can. So for example
09:09
we use Jenkins, we use Fireman, we use Git. The problem is that we didn't find
09:20
anything which will help us for managing these resources and for leasing them to this test, then checking in which state they are and leasing to another one and we have our library we called it art and this is library for running this test.
09:50
Okay, so what this library can do? First of all we use official APIs, we use all of them. So for example in our case you have Python SDK, Java SDK, CLI, GUI and
10:11
REST API and you don't want to run the same test for every API. You want to run it once
10:21
and do your library, let your library to run it for every API. What else? We don't run the test collecting functionality by ourselves, we use
10:43
for it. So everything which you have in knows like tagging you can use here and when you take a look on this code one test is just a unit test and usually it's not that long because everything
11:01
what is in this test is only when we go back here it's only this setup and test teardown or sometimes even without setup and teardown when it's common for a lot of tests in one suit. Okay, we have a lot of helpers. We have helpers for
11:30
peer-open, pyemical, sometimes it's too hard for people when you want to run something quickly it's a little too complicated. We have tools for managing remote files directories and
11:48
things like that. You don't want your user to run im-af or anything. Some slight mistake, they will break something really really bad.
12:05
And of course helpers which are which we use in our tests, you don't want to run IP tables every time you want to simulate a network connection failure.
12:22
So we have just the upper split. Same for disk error simulation, same for load simulation and multi-threading. Multi-threading is hard. Multi-threading in tests usually means that you want to do something and in other fact you just want to wait for some event to AIs and
12:51
that's it. That pool executor is almost easy enough, sometimes it's not easy enough.
13:00
Okay, so yeah, this was this library which we use for running the test. So when we go back, yeah, so we have this part from setup environment to teardown. But still you have to install
13:23
product packages, install configure this product, collect logs, release resources and so on. So getting machine, getting test definition, we use Jenkins. Jenkins can do it for you.
13:42
We use also Jenkins for collecting logs. We use ours brokers for getting resources and then for cleaning them. I will go to this point later. But still we have these points, install product packages, install configure. So we have something which we call
14:09
Jopana and this is actually what is run by Jenkins. And we have something called jobs for every stage of the test. So this is installation versus running this test.
14:25
This is for example for upgrade because sometimes you want to test what will happen with your system during update. And each of such jobs defines its cleanup. So for example if you have job for installation, it defines that cleanup is remove all these packages.
14:47
Okay and it has plugins for getting resources because sometimes you need resources on this stage just for installation. Sometimes you need it within the test, sometimes you need it also here.
15:02
So how we, what we do for getting resources? We have, we call it resource brokers and we actually have two types of resources. Some of them you can just create on demand
15:21
to some level like VMs or NFS shares or iSCSI shares. Some of them you just have a pool, nothing else you can do. For example bare metal hosts. And of course when you have bare metal hosts, they are not, all of them are not the same. So your test has to specify that
15:46
it needs to host one of them with such setup, another one with another. Okay and of course when you have resources which you can create on demand,
16:03
when they are returned to the pool you just disturb them, you don't try to clean it. But when you have something you cannot create on demand like bare metal hosts, you want to check in which state they are returned. You try to clean it, if you cannot do it,
16:20
you just reinstall everything. This is why we use Ferment. You just mark this host for built-in Ferment and reboot it. So if we cannot fix it, we will not spend time on it.
16:40
Okay, so we have integration with external tools like backtrackers. If you know that there's a bug which is found by this test, sometimes you don't want to run this test. Because for example because of this bug, environment may be in such state that you cannot do anything.
17:01
It may be so broken that it doesn't make sense to run another test. Okay, version control system and so on. So, my main thing is do you know such libraries,
17:23
such tools? Because as you have seen, we use a lot of tools written by ourselves and at my previous job we were in the same state. We also use what was publicly available,
17:41
but then we still didn't have anything for getting resources. And we also without the same tools. And it's like I don't believe there are no tools for it. Because if it happens twice in two totally independent companies,
18:03
it must be a common problem. So if you know anything which can help, I will be very thankful for any suggestions. So thank you. Thank you, Katrina. So
18:28
we've got plenty of time for questions. Has anyone got a question they'd like to ask? Hands up. Okay, I'll bring the microphone. Hey, do you know of any solutions if you have
18:45
a bare metal server and you maybe need to push a button or, well, anything that wouldn't require a robot like taking out a disk or something like that? No. Okay,
19:02
we don't do things like that. Okay, thanks. Thank you. Any more questions? One at the back. I think we can hear you, yeah. So you ask the question,
19:20
then Katrina you repeat the question for the microphone.
19:46
Okay, so the question is if this package which enables to reproduce this flow and gives you integration with Jenkins and so on. I don't know about anything like that. If I knew,
20:06
I would use it and not, yeah, it's, again, something. If you know something, please say it. Okay, I hope one day we will open source it because it's not true that everything which is
20:28
in Red Hat is open source. Some of our tools is not open sourced. I hope it will be open source, but one thing is politics. Another thing is code state now.
20:43
I would rather not show this code publicly, at least parts of it. Any more questions? Are you doing all this stuff remotely? So you're using Paramiko to execute stuff on the
21:02
remote VMs or servers or whether you, like Jenkins runs the unit tests or unit tests and uses your framework and does SSH to the server or how does it work? Okay, so it's like in Jenkins you get a host on which you run this code and then
21:23
when you want to run something on remote machines because one test usually uses at least two machines, then you do it via SSH and this is painful. This is, for example,
21:44
Paramiko is not very good. It's not also very easy to use. Is there something better? No, unfortunately, we use Paramiko. We just have our wrappers for it. So it's like it's just you write, you specify what you want to run on this host
22:06
with such credentials, this comment, and you just get output and written code, of course. Anything else? Any other questions from the audience? One thing I was going to say was that I know that where I'm working we've set up, we've got
22:24
code to run IP tables and then tear them down again after the test is finished. I was going to ask you if you'd open sourced any of those helpers that you listed on your slide, but the answer is no. The other thing is that we've recently open sourced something to allow you to set up network namespaces in Linux.
22:45
We're running IP tables in namespaces which might be useful to you and that's called nomenclature. Okay, thank you. Okay, any other questions? Okay, well thank you again Katrina. I'm sorry for mispronouncing my name.