Running Unit Test on Top of Serverless Service
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 130 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/49922 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Software testingExecution unitService (economics)Point cloudExecution unitTouchscreenUnit testingParallel portMeeting/Interview
00:22
CodeSource codeSoftware testingTotal S.A.Physical systemExecution unitMobile WebGame theoryFrustrationSuite (music)Image resolutionDivision (mathematics)BefehlsprozessorVirtual machineMultiplication signSoftware testingBefehlsprozessorSingle-precision floating-point formatTheoryGame theoryUnit testingSuite (music)TwitterVirtual machine2 (number)ScalabilityCodeParallel portInformation securityCodeCloud computingMultiplicationExecution unitCore dumpProcess (computing)Front and back endsFeedbackLimit (category theory)Scaling (geometry)Endliche ModelltheorieElectronic data processingSurfaceWorkstation <Musikinstrument>TrailGoogolMobile appProduct (business)Line (geometry)Archaeological field surveyMathematicsReal numberSoftware developerDecision theoryNumberMehrprozessorsystemComplete metric spaceRemote procedure callGoogle App EngineComputer animation
07:30
Software testingPlug-in (computing)Process (computing)Configuration spaceBroadcast programmingKnapsack problemVertex (graph theory)Mathematical optimizationChemical equationParallel portCircleConstraint (mathematics)CodeModal logicVirtual machineLambda calculusPrice indexQueue (abstract data type)Plug-in (computing)Multiplication signCircleGoogolInheritance (object-oriented programming)Mobile appSoftware testingComputer configurationProcess (computing)Mobile WebFront and back endsFunctional (mathematics)Game theoryVirtual machineConfiguration spaceThread (computing)Asynchronous Transfer ModeNumberScheduling (computing)Group actionServer (computing)Heegaard splittingFocus (optics)Suite (music)Dynamical systemCASE <Informatik>Software maintenanceComputer fileMereologyInformation securityUnit testingDifferent (Kate Ryan album)LogicExterior algebraExecution unitKnapsack problemElectronic mailing listData managementLambda calculusResultantValidity (statistics)Single-precision floating-point formatSource codeComplete metric spacePhysical systemTelecommunicationSubject indexingSlide ruleCodeModal logicModule (mathematics)MehrprozessorsystemLocal ringDataflowWeightBinary codeShared memoryTotal S.A.Modulo (jargon)MathematicsCartesian coordinate systemCase moddingComputer animation
14:39
Function (mathematics)Source codeLambda calculusLimit (category theory)Computer fileBand matrixComputer networkExecution unitLink (knot theory)Binary fileSoftware testingAreaBuildingTotal S.A.Point (geometry)FrustrationSoftwareSolvable groupSoftware testingSource codeMultiplication signSuite (music)Student's t-testThomas BayesFunction (mathematics)Software developerGroup actionSoftware suiteUnit testingBranch (computer science)Point (geometry)Physical systemOperator (mathematics)FrustrationLambda calculusFunctional (mathematics)Different (Kate Ryan album)Band matrixAreaVirtual machineKnapsack problemLocal ringTheory2 (number)CASE <Informatik>Limit (category theory)Video gameCodeParsingRun time (program lifecycle phase)Computer fileImage resolutionInstance (computer science)Execution unitArithmetic meanWave packetBinary codeTraffic reportingComputer animation
21:47
Multiplication signBitIndependence (probability theory)Source codeExecution unitSet (mathematics)Knapsack problemCodePhysical systemDifferent (Kate Ryan album)Group actionRight angleLimit (category theory)Software developerUnit testingSuite (music)Lambda calculusSoftware testingGoogolPoint cloudMatrix (mathematics)TouchscreenMetric systemArithmetic meanCore dumpMeeting/Interview
Transcript: English(auto-generated)
00:06
We are going to have the next session now, which is going to be Adi Nata. I hope I pronounced the name correctly From pocket gems talking about how to utilize serverless technology to optimize and parallelize unit tests
00:22
Hi everyone. Welcome to my talk. I assume you can see the XKCD comic on your screen Please interrupt me if you don't This comic is a classic But it might not relevant with Python community, you know, because we don't usually compile our Python code
00:41
If you work at a larger company, you might encounter the pain of waiting your changes to be deployed It's pretty common in large company deployment can take more than 30 minutes Today I want to discuss a similar yet different pain
01:00
in my company Running full test suite can take more than 25 minutes Here is a screenshot of our unit tests when being run locally using those tests It run with four processes if you look at the user time It almost 6,000 seconds or 100 minutes
01:24
roughly four times of the 25 minutes running time This unit test being run in parallel with four processes which make it four times faster There is a great talk from Jin Hui Yong yesterday in paratrack that explain about speeding up data processing with
01:44
multi-processes you can watch the talk to understand more how the multi-processing work and the thing is The unit has already run on top of the line MacBook Pro I'm going to share how we decrease 85% of time to fully run the unit test
02:07
From 25 minutes to be under 5 minutes My name is Adi Nata. I've been writing code professionally for six years I work at startups in real estate gaming company and finance
02:23
Most of my works are in the backend or DevOps I can do some front-end development, but my design skill is not up to par with my coding skill Please say hi to me in my Twitter Give feedback about my talk, cross me, send anything to me. Your feedback is very valuable
02:43
My talk is based on my work while I was working at Pocket Gems Pocket Gems is a game company We use Python for our backend and the backend run on top of Google App Engine to serve our players It's highly scalable and you can focus on developing features rather than scaling the infrastructure
03:06
The company has been around for more than 10 years and has accumulated more than 4,000 unit tests in NOS tests Previously, I was explaining this talk only using NOS tests, but it's overwhelmingly underfunded
03:25
So I added PyTest material. Thanks to everyone who participated in my survey Our topic today is how Pocket Gems run the unit tests in serverless infrastructure I will share with you about our decision-making
03:42
How did we solve this issue and how we were able to complete the full test suite in four minutes I know this comic is an anecdote, but How do you really feel when you have to wait half an hour before you can do your job?
04:06
For me, it's very frustrating Apparently it also frustrates others So we said to fix the problem Let's try to understand what the problem is Running full unit tests is slow
04:24
Is it worth solving? If we make the unit tests complete faster, we improve developer productivity and happiness If developers are happy, they can produce more failures
04:44
We always run full unit test suite before deployment If we make deployment speed faster, we can deliver failures to our players faster It sounds like this is a good problem to solve So we said to fix it
05:01
How do we make the test suite complete faster? If we have fewer things to run, we will need less time We can even skip running unit tests and it will be blazingly fast Of course, this is not scalable Unsafe, please don't think I'm serious. I am joking
05:24
We need to go through all unit tests The only way to go faster is to define and conquer Each unit test is independent We can split the unit tests and run them individually
05:40
Instead of a single process running the whole test suite, we can have multiple processes running How do we parallelize test executions? We already had a glimpse at the beginning that we can run unit tests in parallel in a single machine with multiple CPUs
06:05
The limitation is it limits you to how many CPU cores your workstation has It can also parallelize your ability to work because of the CPU consumption while running the test suite
06:24
The second approach is running it in multiple remote machines With the era of cloud computing you can have an unlimited number of CPUs You are still limited by money But in theory, it's infinite
06:41
the drawback is You have to maintain those machines You need to do health checks, security check, monitoring, alerting, auto scaling There is also a warm-up time associated with auto scaling that can Significantly add more time to complete
07:02
In the last approach we can try to run it in a serverless infrastructure If we can do this, we will also have an unlimited CPU but without the need to maintain the work or not And because the pricing model is only paying what we use It will be cheaper
07:22
compared to using remote machines Let's go into detail for each approach local runner Test runner usually have the options to configure the number of processes to run the test suite for example
07:41
Pytest have the XDS plugin which accepts test end options and knows test with the test processes options The way it work is first the parent process will spawn processes based on the given configuration
08:03
The worker processes collect all the tests and the manager then validate the collected tests Manager then schedule tests to each worker For Pytest all the communication between worker and manager happens via the exact net channel
08:22
Those tests use multiprocessing.q to communicate Next multi-machine Pytest also have the capability to run in remote net with the same plugin XDS or probably cross this
08:41
The flow is mostly the same the only difference is It are sinks the necessary source to remote machine There is also ruby alternative called knapsack pro The basic idea is very similar
09:00
You have a queue each processes in this aci machine send list to the knapsack pro and the manager And knapsack pro validate it and schedule the test worker worker letter send the result into the server there is also fallback mode which Will run the test by splitting test by file name in case of the api timeout and failure in reconnect
09:24
It also use some kind of dynamic queue to ensure the time of a completion around the same between all the nodes One main difference is Each worker can start their part without rsync
09:43
Usually triggered with git commit or git push This is helpful if your ci solutions Manage the worker for you And you cannot use ssh to the remote machine Which is what the pytest cross this depends on
10:02
Example of such ci solution is circle ci And also github action Now, let's try to design our solution As a mobile game company
10:22
Pocket gems focus on delivering value to the players and creating new innovative games It means we won't have engineering resources to manage the worker full-time If we use a worker machines We will have more things to do which can be a full-time job for a team
10:43
From monitoring health check security dependency To ensure the cicd work continuously Of course we can invest in the beginning to make the system more robust require less hand-ons less maintenance
11:02
If we look into how puppetchamps backend works, we can see opportunities First Because we run on google app engine All the tests don't have external features And also the backend code is 100 python code and aren't using many binary dependencies
11:27
The situations raise Opportunities for us to consider using serverless So we decide to use AWS lambda To remove the necessity of managing worker machines
11:44
Here is how we design it First we upload the code package or deploy it We create a lambda function that can run the whole unit test It includes all the package dependency including the binary dependencies
12:05
Then we decide how many workers we want. Let's say 10 workers In the local machines, we chart the unit test based on the number of workers If we have 500 unit tests and 10 worker
12:21
Each worker will execute 50 unit tests Let me show you the next slide Here is my best illustration to show you what I mean In this case, we have 12 tests numbered from 0 to 11
12:44
It's a simple module functions module by number of shards And if your test id is the same with your shard id We take that test as part of the groups
13:03
We created a nose test plugin that has a custom selector And here is the logic where Which unit test will be run for a given node and the node total
13:22
We then create multiple threads that send HTTP requests That invoke the AWS lambda functions For each HTTP request, it will communicate with a single functions or container
13:42
Because we are using serverless each container doesn't know what is worker id Hence, we send the thread index And the number of shards as part of the request Those HTTP requests are asynchronous so we can send all of the requests without waiting for them to complete
14:08
Lambda functions Will then run the unit test Selected based on the thread index and the number of shards And meanwhile our local machines will be blocking to wait for the result to complete
14:29
It's cheap to run hundreds or thousands of threads at the same time in this case Because all the thread is doing
14:40
Is waiting for IO When the lambda function completed The HTTP request return the test output To our local machines And when all the HTTP request complete The local runner parse the output
15:02
Show it to the developer This approach works well for our use case Of course with some gotcha Here are what we have learned When developing we encounter the code limit of 50 megabytes for AWS lambda
15:27
So the local runner when uploading the source code need to exclude everything in git folder exclude all the pyc files documentation files dependencies
15:44
Dependency tests, I mean and yeah, basically making sure that we only upload what is necessary for the test to run In the system design, we deploy the code base as a lambda function
16:03
Now imagine two different machines two different user Running the test at the same time, which means we upload the code at around the same time How will the unit test behave? while one unit test suits
16:21
While one of the unit tests with running someone else uploading a different code from a different branch It probably won't work and probably won't be safe So we handle this by letting each machine have its own name functions or deployment
16:43
So each user have different deployment to different lambda function The third is it depends on also depends on your network bandwidth Is your network fast enough to upload 50 megabyte?
17:04
If you are currently traveling and have a slow network, this might be less useful It's because for every test execution we upload the whole source code to AWS lambda The four minute times that I showed you before
17:23
One minute and 30 minutes one minute 30 seconds was used to upload the source code So it's actually only required two minutes 30 seconds to run the whole test suite In theory we can use 1000 parallel executions
17:46
But the battle donut changed into the longest running unit test If you have one test that is particularly slow Then we will need to wait for the test to complete before the whole test suite completed
18:06
Serverless is excellent, but with its limitation You cannot log in to compute instance you have limited time out You also cannot customize the runtime
18:21
If you depend on binary dependencies, you cannot install it beforehand into your runtimes That means you need to upload your binary dependencies together with your source Nowadays, I think this is a solvable problems
18:43
for example We can use github actions The knapsack pro that I showed you before actually support running on top of github actions We can create a docker container that will be used to execute the test runner
19:04
There is also several learning that we encounter Our intern had developed it for roughly four weeks And we have been using it for the last two years
19:24
Running the test suite not completed in four minutes That's 21 minutes being saved on every execution And we have more than a thousand executions each month
19:44
If we translate that into engineering hours the 21 minutes before multiplied by one thousand Defied by 60 minutes We have more than 350 Engineering hours each month being safe
20:06
So assuming when engineering hour costs 50 In bay area we are Saving 17 500 each month
20:21
Now let's talk about the cost of operation In average the system costs around 120 each month based on the AWS report So this is very cheap compared to the engineering hours being safe
20:45
And if we look at the initial investment It was four engineering weeks That is less than 10 000 The break-even point is less than a month
21:02
And we have more than 70 engineer happier using the tools If we go back to the beginning We are just frustrated If we look around our daily life, we might be able to see stuff that made us frustrated
21:25
Others might have the same frustration with you. Are you able to proactively figure out how to make it less frustrating to you? You might solve everyone's problem That is the end of my sessions. I think I talk faster than what I trained if you have any question. Feel free to ask
21:48
Indeed thank you very much. So let me play the applause just a second
22:03
So thank you very much for your talk that was you know, very impressive really I mean uh you It's not only I mean, it's not only money that you saved It's it's actually I think it's you know lifetime that you saved the developer's lifetime, right?
22:22
Exactly sitting in front of the screen and waiting for stuff, which is often very frustrating It would I mean, you know when I think about it Python development has this problem as well It takes ages to run the test suite For the for the python source code, so and it's basically running unit tests as well
22:47
And I believe that most of those unit tests are actually Independent of each other so it would be easy to parallelize everything By just you know Uploading everything x times and then you know scaling up that way
23:05
so That's actually something that we might want to consider Yeah, I think it's also with the github actions. It was very easy to parallelize things a lot if the knapsack The snaps the knapsack pro that I showed you before actually allow
23:25
Us to use github actions to parallelize the unit test executions Right. So it's it I mean, what do you have to have you have to have a github action? So it's had the code has to be on github, I suppose, right?
23:42
So, I mean that's for for python code development that's a given we're using actions as well So the only thing that Is missing is setting up the aws I guess right If you use github actions, you don't actually need AWS because github action actually run in a container so you can run the unit test in that container
24:08
But then how do you okay, so you would have to parallelize the github actions Yes in the knapsack pro. I haven't tried this by myself, but in the knapsack pro It used the move the matrix. I believe the matrix to parallelize it so you can have the same
24:26
System for example linux, devian to be run to run five or ten different Executions Right, that's very interesting
24:44
One question about the your use of aws lambdas. So What you're doing with aws is essentially for every single Let's say unit tests or set of unit tests you upload the whole Source code ball right to aws lambda
25:01
And then you run into these issues with the uh with the code Code size limitation would it help to put the code on s3 for example, and then load it from there or is this a You know, is it a let's say a container limitation that aws lambda has It's the aws lambda limitations
25:21
At the time of the development the github actions, I believe doesn't exist before so we try to use aws lambda and google doesn't even though our infrastructure run on google Google at that time doesn't have python as the serverless
25:40
So we use lambda because it's already support python. I think google already support python now for the serverless I haven't used google cloud in I don't know a long time I know that aws supports python And has been for I don't know how many years so i've used that quite a bit so i'm familiar with it
26:06
Right, so yeah