Fixture factories for faster end-to-end tests

Video thumbnail (Frame 0) Video thumbnail (Frame 11793) Video thumbnail (Frame 23374) Video thumbnail (Frame 34955) Video thumbnail (Frame 35938) Video thumbnail (Frame 37682) Video thumbnail (Frame 38484) Video thumbnail (Frame 40239) Video thumbnail (Frame 41797) Video thumbnail (Frame 42527) Video thumbnail (Frame 44281) Video thumbnail (Frame 44903)
Video in TIB AV-Portal: Fixture factories for faster end-to-end tests

Formal Metadata

Fixture factories for faster end-to-end tests
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Fixture factories for faster end-to-end tests [EuroPython 2017 - Talk - 2017-07-13 - Anfiteatro 2] [Rimini, Italy] When developing and maintaining many different services, unit testing is not enough to make sure your code works in production. By now, many teams doing SOA (service-oriented architectures) have a set of end-to-end tests that cover critical workflows to make sure these work. For these tests, all of the utilized services need to have the proper test fixture data in their datastores. This often leads to developers having to deal with raw datastore data (like JSON or SQL) for these tests, making the authoring of those tests very slow, tedious, and error-prone. This talk is going to discuss several approaches we tried at Yelp to generating these fixture data in a quicker, developer-friendly and more correct way. The main part of the talk will be a deep-dive into what fixture factories are, how to implement them and how to integrate them with pytest, the leading Python testing framework. I'll show you several other benefits this approach has over writing raw fixture data and how this leads to more maintainable and easier to adapt code. We'll also explore how you can then run your tests in parallel, cutting down runtime drastically
NP-hard Sensitivity analysis Intel Metric system State of matter INTEGRAL Code Multiplication sign Execution unit Set (mathematics) Solid geometry Parameter (computer programming) Mereology Stack (abstract data type) Subset Web 2.0 Medical imaging Web service Software framework Endliche Modelltheorie Abstraction Physical system Social class Service (economics) Theory of relativity Data storage device Bit Flow separation Statistical hypothesis testing Data model Repository (publishing) Architecture Software testing Endliche Modelltheorie Quicksort Figurate number Point (geometry) Functional (mathematics) Service (economics) Computer file Connectivity (graph theory) Disintegration Letterpress printing Online help Rule of inference Scalability Statistical hypothesis testing Product (business) Element (mathematics) Twitter Number Goodness of fit Universal product code Software Factory (trading post) Energy level Software testing Statement (computer science) Task (computing) Default (computer science) Default (computer science) Multiplication Demon Scaling (geometry) Graph (mathematics) State of matter Code Database Cartesian coordinate system Integrated development environment Software Personal digital assistant Web service Function (mathematics) Factory (trading post) Object (grammar) Table (information) Abstraction Library (computing)
Group action Run time (program lifecycle phase) Code State of matter Texture mapping Multiplication sign Set (mathematics) Water vapor Frustration Parameter (computer programming) Mereology Subset Mathematics Web service Computer configuration Algebraic closure Special functions Endliche Modelltheorie Error message Stability theory Theory of relativity Software developer Token ring Fitness function Sampling (statistics) Electronic mailing list Data storage device Bit Determinism Instance (computer science) Flow separation Statistical hypothesis testing Message passing Process (computing) Repository (publishing) Internet service provider Order (biology) Natural number Software testing Data conversion Pattern language Right angle Quicksort Resultant Sinc function Point (geometry) Software engineering Slide rule Functional (mathematics) Server (computing) Service (economics) Overhead (computing) Robot Virtual machine Letterpress printing Statistical hypothesis testing Product (business) Number Twitter Universal product code Term (mathematics) Robotics Business model Integer Software testing Contrast (vision) Gamma function Boolean algebra Default (computer science) Addition Distribution (mathematics) Dependent and independent variables Scaling (geometry) Code Planning Core dump Database Ultraviolet photoelectron spectroscopy Line (geometry) System call Software Personal digital assistant Factory (trading post) Blog Statement (computer science) Social class Codec Object (grammar) Table (information) Library (computing)
Mobile Web Service (economics) Process (computing) Integrated development environment Data storage device Configuration space Right angle Continuous integration
Human migration Medical imaging Presentation of a group Mathematics Software developer Order (biology) Normal (geometry) Set (mathematics) Database Right angle Product (business) Statistical hypothesis testing
Key (cryptography) Personal digital assistant Multiplication sign Database Software testing Insertion loss Information security Statistical hypothesis testing
Service (economics) INTEGRAL Database Mereology Statistical hypothesis testing Human migration Web service Integrated development environment Hypermedia Repository (publishing) Factory (trading post) Core dump Endliche Modelltheorie Data structure
Logic Cartesian coordinate system Table (information) Statistical hypothesis testing
Area Personal digital assistant State of matter Translation (relic) Statistical hypothesis testing
thank you all things my having
having before start of a little bit about my employer that thankfully allows me to be here with you from yellow has a website and add to help you find the great little businesses and we get over 100 20 million user reviews and we're active in over 32 countries so that help you make sure you find to correct because this does what you want to do and have a good experience solid step right into it and what will be talking about 1st of all I'm just the that specify what I mean with trend tests and then discuss the problem that we have with setting up and print test from see whole fixture data factories help solve the problem and just to clarify because of some people ask me what I mean with fixtures fixtures father tasks you need to do to set up some the environments with test can run them in our case that's mostly from inserting data into data stores but it can also be all sorts of other tasks and lastly how can that help us make the test pass so end-to-end test some people call them system tests the acceptance tests and have seen people say on the models I integration tests yes the integration testing something where you might just has like 1 big strong component where is contained tests that are defined as I see them as in an environment where you try to replicate as much of your production environment as possible so you spin up as many internal services as possible and you try to be as close to reduction as possible and then you run your test on that test infrastructure this means the sensitivity the slowest test run and they are also the most expensive tests and in that sense and you know so we want to make sure their effective because if you have a distributed architecture from those tests are crucial in making sure that you know everything works in production and works together and obvious unit as a cilia foundation of everything but they don't tell you that everything fits into the elements so just come in the wake child to the technologies we use in our technology stack and that are relevant to this talk so I'll be mentioning these and we use pyramid to web framework that we use letter splashed open API to communicate between services ended in the example here I'm going to be using my as with a skill alchemy from again we also use a lot of other software like for data stores the most notably Cassandra Elasticsearch several hundreds it doesn't matter for this talk so what i mean with a distributed architecture I'm sure like many of you are familiar with this and you have services here in this images of Michael so this doesn't have to be the point is that the services are not necessarily running on the same host are separated by network layer and in our case the cold measles also separated we don't have a big model a some like some companies these are in separate git repositories uh and just like in production also in testing these services us on on separately and the doctor containers and then communicate from all over the network and so the problem that the hard part about tests is that you have to create the correct state for your test and not only your own service in the armed services for but also in all dependent so downstream services that you are calling from and so how did we do that the well um I'm not really happy to say that yes we basically wrote a bunch of this scale for my my SQL so you know that these files and when you talk a container is created for your downstream services then we just 1 is still uncertain data and then use that data and test this means you have very tight coupling to the downstream services and yet it's hard to write it was right role as well it's hard to get right might forget stuff you might do it differently than on the production code would have created that data and it's hard to maintain so what are some of the possible solutions if we take a look at what the jangled us so they have all their own or and you can use it really nicely and test to import your model and you have a nice iPhone API to creating the data you need you do that in a set of function and it actually even cleans up after the test is 1 and so this is pretty nice but we'd like that for downstream services as well and without code application so without having to do that like in n multiple some services that call 1 and the same service of thank we don't want to read yourselves so persons we don't use Django internally I mentioned that we use pyramid and S. Cohen committee what's there can we do not know this is what my talk is about and we are creating a yes we have these factory libraries and that basically generated data from that you need for your test and with a nice the Python EPI and they contain the STL walls you need for that class of creation functions that provide a slightly higher level of abstraction and they take care of things like default values you sometimes have to specify all of that if it really doesn't matter for your test and they make sure your data as politically correct and so making sure that if you create a a certain object certain role in a table you also create all the other rules that should be created at the same time and this is especially important since we like some other companies that randomize skill scale a disabled foreign key checks for performance reasons for the scalability the so we can't even rely on the database telling us that we forgot to create a relation and lastly them if you have these packages all of a sudden you document which services are using which other services and data for PPI for the testing so all of a sudden what you do know beforehand I couldn't figure out reliably now you can actually figure out manually and automatically creating a knife dependency graph and tell you know which pieces of code you need to look at if you change something in the data model so let's take a look at such an example and and test how it previously on yeah it's basically we used I test for testing and so I'll talk a little bit about that later the important part here is we do a GET request so it's really not that complicated I would say and in the URL which has 2 parameters them to test this endpoint and it's the 4 dimensional it's like the rest of your eyes the with some basically of import many people are using and so it has a business ID and the question ID the and how do we do that we pass them as hard coded it's so there's an actual number and the test and that number that corresponds to the Aidid um
we specified in those scale statements I showed you before on and then we use the mirror and that both for the business at the and the question and you and you know we have some other parameters doesn't really matter in this case and then we assert on response so how do we want to that test will cut wanna change and how does it look now 1st of all so you you see that could see that it's basically just 3 lines that change on the 1st line is the top line of where we has some pi test fixtures this is why I'm using data factories now for so those libraries we used to create the data because of pi test uses the same term fixtures for setting up from your test and when it does that and providing special functions that you Marcus fixtures and then if you function name as an argument to be a test function it executes the fixture function and returns or provide in that argument whatever come and that function returns we're going to see an example in a 2nd but I hope that makes it here but the point is that like question and this is a the is not something that gets done during set up and provided here as arguments that we can use so the only 2 other lines to change a now where we pass the business idea and we don't specify and hard-coded integer anymore but we actually passed whatever the idea was generated for the business and we do the same for the questions and so those other code changes in the test itself what we get to do now in addition is delete those ugly Estelle lines used to set the test set up test previous but not impossible question is how do we actually generate such a business and the whole of the business data on well the fixture itself is very short because the delegates all the work to that of factory library I mentioned earlier the remarkable the decorators so Cervantes knows it's a fixture and then it calls for the treatment of on the business of factory to create the business with a frankly like all the default values we don't care about what specific businesses for our test so how does that create method look like this is so we get the option of passing all sorts of values to to customize the business if we want but really all that does is used a business model which is in a spell out in the model and which is not important for this specific examples and and and creates some an instance of them all in the database and returns the this provides reasonable defaults that we don't have to specify for each test and it prevents the deer why so we don't need to review so but might last mean why do we need to use the functions and we just use the models directly as I mentioned so we don't have a foreign key checking and even if we don't like that everything can be on call a foreign key checking and sometimes the production called only generates the data entries together and you want to do the same in your test so in this case this for the business owner of this user we call internally and you can see there's like but this is a private table that that should always have an entry for any entry that is in the misuse of table and there is another relation with the business of business table that might have 1 or more entries that connects the business users could business on you know so and what do we do here and we want our code to mirror like as close as possible as what or production code does so let's take a look at the fixture and what we do here we can see it's relatively the same to the business fixture in the beginning it creates an abuse user entry the with the Pascal alchemy model but then the here could also always creates an entry into this is a private table so you can't do not do that which is exactly what we want and it also since most of the time you want to associate a business user with the business but also there's this here as a convenience and with the 2nd method that it calls for you if a provider business idea that you can still use a new test separately and in case you needed for your test so there's another case of maybe is a well this is all nice but they actually do have some services that provide me with the data creation API can I use that yes that actually integrates very well with this approach that also in this case we got a service that's called question-and-answer and we used the swagger so a library called bottle to communicate with that service where we make a request to the server of parameters and we get as a result from a question object and this is also quite test fixture and that we can then use and I did in the example before we just specify question and this piece of code gets executed and returns the data not home mothers as possible and this is indeed a taken out of an internal repository you should be aware that in contrast to using models and those olfactory methods for functions rather than in this has the disadvantage that it kind of assumes that you can actually communicate with that service and that uh the specific endpoint will work and return data so in case it doesn't for whatever reasons of getting a nice test failure that the that we pointed to the issue you get an error term that fixture function execution which is a little bit of harder to depart so just be aware of the now let's discuss the below that the pros and cons Why I like this approach I think it's a natural fit the 4 pi test fixtures integrates very well it also provides a much easier decoration from if you like if you look at this you could use those a test fixture functions to secure or as well and you would get some of the same benefits but you would still have to write as well would still have this extremely tight coupling them again and that it just this provides a nice privately come also it makes sure people create separate data entries for each test automatically sense once you have written those high test fixtures of most of the time all people need to do is just use those arguments in the test functions and and it will automatically create new entries for each test which has happened several advantages that I'll get into just in a 2nd and you don't have to come convert all of the test immediately you can actually just for example right new with this pattern and not to existing test or migrant and slowly from and this will continue to work 1 thing you do need to do is you need to maintain those of
libraries of with those factories it's a little bit of overhead and in our excuse is actually not that much work but it is another repository with more cold so be aware of the and some of it can be potentially slower especially if you use to share some like these fixtures these data between test them because you know how well 3 new entries for each test every time you run it and that that can be sold which is not great sense that I used to work faster than in my title so let's take a look at what we can do to make them faster so what does that really really easy use all the cost a machine has more than your test circuit or and don't run more tests in parallel there's 2 issues with end-to-end trend tests in particular um that might make this a problem since it's it's about writing data so 1st of all many tests are not repeatable so you can run them more than once with others that in your data stores this is not a problem here for just running them once the hidden Rosetta data about 0 that's given in mind for later and the 2nd 1 is that if you share data between test you now all of a sudden depend on execution and this is something and you'll see in a 2nd we've encountered that with our test so you write these tests and they use the same business and 1 test changes something about the business and the 2nd test plans and everything is fine because it knows that this data was changed but if you paralyze tests you can't guarantee test execution water so depending on which test runs first one of them or even both of them might start to fail so but how do we actually can run tests in parallel what this is it for pi test it's actually really using you install something you provide the parameter and with the number of workers so processes you want the tightest to use and that's it the well since we did that before converting any of our test just to see some if that would work well it turns out and is just a sample list of issues we had and you can see we get a bunch of test failure so this is not something you can simply do and this is why he is a fixture factories or data factories have because once to isolate protests and you have separate data entries for each test some you fix this issue since you at that point what 1 test us the other test doesn't really care about and the so on all action 1 thing and that I don't want to mention even if you just randomized fast execution order you will get most of these failures so paralyzing tests just something to make it faster but in my experience especially if using something like high test which has a a deterministic test execution order by default if you randomise test execution which is basically what we do with paralyzing you get these failures so there you can see here the latest issue of the last issue and list was let's enable parallel test execution for all of our acceptance as we call them acceptance as internally so we should be fine right it turns out that we were find that were just the issues that we only discovered when more people were running the tests in parallel but simply because even if they do run in parallel with high test doesn't really always randomized the test will run them and completely different order so the more people use your test more issues you will discover and again this resulted in basically us switching more and more of those and tests to the use of tools data factories and I provided the number 4 in our example or halt execute or how many processes to use I can tell you that at least for us so using more than 4 processes didn't provide a significant speedup from bot it did cut down the execution time by roughly 60 per cent and as you know from and and tests are like the slowest so this this picture significant speedup on plus it provides you with you know I the i it's easier to write those tests and then since we also solve the problem of test repeatability it means that you can actually like the when you do developed the tests you can extract from the multiple times without having to represent the data store state across the downstream services because it since always preview data entries you never get a problem with like and I don't know you want to like to reply to a message and the 1st time it works in the 2nd habits as well you actually already replied to that message but if every time you run the test it creates a new message and test if you can reply to that message the test will just continue to work but so so what are the main takeaways of my talk well 1st of all if you use this kind of pattern so you will get faster developments among only test execution but um people will be able to write tests faster and and less frustration writing these complex and print test and you get more caloric test data we've had multiple instances where we discovered that the test set of we did using a raw SQL and was not the correct 1 like it didn't represent all the data would look like in production convert tests for test isolation repeatability this is basically what all those and the durations I showed you were about and if you do that then you can also actually make them you test faster by for example running them in parallel and will just get come much better tests of set up anyway since you we don't want to depend on a test or execution from yeah and as I mentioned is also easier to iterate over them the over the test and after the test part right like it's not such a pain to development and test the developers are more likely to actually write and test which will yield to robots and warm and stable software I also wanted to call out the essence of now at the end of my talk and talks by my colleagues that already happened yesterday and the day before yesterday please check the mountain were really good in my opinion solves most prized smarter than me and they will be online as I assumed that yeah but that they also showed this slide 1 thing I wanna call out here is the engineering block the way we have in my opinion be an interesting blog post again written but people smarter than me and yeah and that is my slides are online and it obviously like just a few typos only to fix it also contains uh some of the speaker not capture all of the speaker notes to smoke that there's so many and reversible so go check out and thank you mn
it but we have time of about 5 minutes of questions of and the 1 in this this HCI on how are you orchestrating mobility services before running and and up idea 1 in this that like honestly I mind right and how are you or the micro-services before
in OK so it's actually really hard to hear from frontier only saw the speakers struggling with that so 1st it was about this CI environment right and so while you can execute them locally with then use Jenkins for continuous integration and it also spins of everything and what yes always how we set up on in the past we used the proper compose for doing that actually in the process of switching to and an internal solution called Yelp compose which Canada's will talk of posters does what is more tailored to our environment so basically yeah you have something that defines the dependencies and the configuration more than the data stores they need and and it builds and spends all that on yeah I think so I wish to use of the the right away so you why
don't you of your database for all these images migrations youth from normal in the last in the beginning of the presentation you show that you right away skill in all in order to generate new the wider at the base of the yeah and it's a good question
well um 1st of all production and databases this way to make it so we we couldn't use that and busy yeah canoes and test so basically we just want to create a set of data we need more of running on our tests and if you depend on production data you would also again have the same problem of depending on the fact that the data doesn't change or you know like you can just have a fixed set of data that you then always reuse even if you use production and would say well there's a business with the ID 100 and I'm going to modify it and then yes you at least Omega Test refutability right like because once you have modified it you need to reserve your data and we're anyhow and we also need to make sure that for example all of the test is all different businesses from your production data which again is actually not that easy to coordinate amongst many teams of developers so it's more sometimes it's easier to created the I to get so I have a question had you avoided that
this 1 running out of yes from frankly in the case of my as well we just let the database security and so on we tell it to create and the data like an entry and it returns a primary key and we used a primary
key and frankly the database just make sure that if there's like 10 inserts coming at the same time that also still works it's really like I can actually spend on and on the problem is really with some tests using the same data so as long as you have row-level locking you'd you never run into any issues as long as every test uses different roles and and as long as you do that you find the it's holds a lot of these kinds of issues and yeah and the and at the beginning
of we show that these have model packages have with the factories and models in them and then he said that it was also an advantage that you can see the dependency structure of these models did you mean that you have different micro-services using integration databases using the same models or today mission is yes so like media like a
specific example we have some sessions managed in a specific service and and so obviously like since this is the very core part of things we have like multiple other services using that session service so if you have like a common package that is 1 that creates sessions just as an example you can reuse that everywhere so at 1st you don't repeat yourself and 2nd of all you can see like you can write a small script and find out which are the repositories use that data creation package right so all of a sudden you know who is actually creating the test data for your session service so if you plan on doing the backwards compatible data migration or schema migration at least you know now who you need to watch out for for the test them environments right the but 1 over the question you said that you know
any test uses different data because they are severely Pietro's but is that like know what of your application logic that for example like these account of every something in the table yes
that is a very good question and I
was waiting for that obviously it's not solely all of the problems that it's true if you had such a test that would be a problem to be defrosted when they take account of all businesses and but there are some cases they can give you an example we do have some things that are specific for countries right so on and then if you have test that modify the state of something for a country and then you have another test that creates this is another country but all of a sudden like 1 of the test influences the other you don't have test translation anymore and you might get test areas so yes there are still cases that wall we sold by this it still requires you to manually designed test that way by for example using a country we don't normally use another test signal stuff like that but I guess you're right it but I would say it actually helps in the vast majority of the cases we encounter not of a and that yes maybe is the comment was maybe there's a way to actually specified transformed but and yeah so I suggesting continued in the whole of the that the and new than this but here it few