Understanding Celery & CeleryBeat

Video in TIB AV-Portal: Understanding Celery & CeleryBeat

Formal Metadata

Understanding Celery & CeleryBeat
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Understanding Celery & CeleryBeat [EuroPython 2017 - Talk - 2017-07-10 - Arengo] [Rimini, Italy] Celery is a distributed task queue for Python. Although it is most popular in the web development ecosystem, it has a wide area of usage from system management to IoT devices. With Celery, transforming a function into a task is quite easy and can add great performance & usability to the applications that we build. This talk aims to give attendants a general overview on Celery and its uses. We will walk through the core Celery architecture by introducing key components with the help of various real-world examples. This will also lead to an understanding of the task queue systems in general. Attendants will also gain knowledge about Celerybeat; a tool that focuses on scheduling tasks. We will be looking for the answers to the following questions: What is a distributed task queue? What are the main elements of Celery? When should we use Celery tasks? How do we use Celery Beat? Attendants should have a basic knowledge of Python, and a minor development experience
Intel Presentation of a group Video game Software testing Object (grammar)
Domain name Code Software developer View (database) Projective plane Planning Lattice (order) Line (geometry) Cartesian coordinate system Product (business) Web 2.0 Different (Kate Ryan album) Ontology Calculation Website Software framework Library (computing)
Web page Dependent and independent variables Functional (mathematics) Email Gateway (telecommunications) Run time (program lifecycle phase) Computer file Inheritance (object-oriented programming) Multiplication sign Web browser 2 (number) Connected space Message passing Process (computing) Personal digital assistant Error message
Email Functional (mathematics) Email Group action Computer-generated imagery Dependent and independent variables INTEGRAL Image resolution Multiplication sign Computer-generated imagery Data storage device Online help Coroutine Mereology Demoscene Personal digital assistant Term (mathematics) Synchronization Software testing Analytic continuation Task (computing) Backup Resultant Queueing theory
Slide rule Implementation Presentation of a group Functional (mathematics) Multiplication sign Gender Modal logic Projective plane Set (mathematics) Parameter (computer programming) Cartesian coordinate system Mereology Element (mathematics) Computer architecture Computer architecture Message passing Personal digital assistant Internet service provider Queue (abstract data type) Message passing Task (computing) Resultant Queueing theory
Frequency Functional (mathematics) Projective plane Software testing Parameter (computer programming) Task (computing) Computer programming Social class Task (computing) Queueing theory
Data management Message passing Latent heat Military base Term (mathematics) Website Set (mathematics) Software testing Database Cartesian coordinate system Form (programming)
Email Functional (mathematics) Computer file Multiplication sign Range (statistics) Virtual machine Set (mathematics) Numbering scheme Database Parameter (computer programming) Mereology Product (business) Different (Kate Ryan album) Touch typing Queue (abstract data type) Electronic visual display Energy level Software testing Service-oriented architecture Message passing Form (programming) Task (computing) Information Surface Keyboard shortcut Physicalism Basis <Mathematik> Instance (computer science) Cartesian coordinate system Computer architecture Type theory Message passing Uniform resource locator Process (computing) Order (biology) Right angle Task (computing) Sinc function Queueing theory
Slide rule Functional (mathematics) Server (computing) Service (economics) INTEGRAL Multiplication sign ACID Mereology Bit rate Atomic number Electronic visual display Software testing Task (computing) Area Email Electric generator Projective plane Electronic mailing list Content (media) Line (geometry) Price index Message passing Process (computing) Order (biology) Website Task (computing)
Computer architecture Beat (acoustics) Type theory Message passing Queue (abstract data type) Software testing Cartesian coordinate system Message passing Task (computing) Queueing theory
Beat (acoustics) Scheduling (computing) Game controller Personal digital assistant Multiplication sign Moment (mathematics) Time zone Set (mathematics) Solid geometry Software testing Position operator 2 (number)
Dataflow Addition Multiplication sign Software testing
Logical constant Scheduling (computing) Context awareness Run time (program lifecycle phase) State of matter Decision theory Multiplication sign 1 (number) Set (mathematics) Parameter (computer programming) Inverse element Mereology Endliche Modelltheorie Series (mathematics) Monster group Physical system Covering space Email Data storage device Sound effect Bit Instance (computer science) Flow separation Demoscene Connected space Message passing Order (biology) Normal (geometry) Moving average Quicksort Task (computing) Point (geometry) Implementation Game controller Computer file Distance Event horizon Rule of inference Product (business) Revision control Frequency Computer architecture Latent heat Operator (mathematics) Queue (abstract data type) Software testing Computer architecture Task (computing) Window Pairwise comparison Standard deviation Prisoner's dilemma Forcing (mathematics) Projective plane Expression Cartesian coordinate system Uniform resource locator Object (grammar) Musical ensemble
the higher and today I'm going to talk about
city that is to be intellectually uh presentation so if you use it in your daily life at all familiar with it this presentation will that many things to but if you don't know anything about so that it or if you find test use object in general this presentation might be useful for you so my name is he fired from you
stumble 30 uh this will be my 1st Europe widened ball so we should look and I have a member of widened stumble community the young may change in the the not the meetings and the planning to organize it Pike ontology the hope for the next year so we should consider a Iot working people it's a company where review products so uh companies and start-ups commonsense their ideas and the transform them into applications and the use city heavily in of protecting so before starting to i want
to talk in detail about the beauty of the In my opinion the beauty of fighting comes from its diverse views people from many different backgrounds use fights in every day from the development to astronomical calculations fighting is being used everywhere so it's a great opportunity for developers to take a great advantage from the huge knowledge off the academic aside in projects for example if you're making a website you can easily integrate uh and uh in which the Commission library it just by adding some lines of code but the same applies for the people from other domains as well imagine that you're limited edition and working on an orbital you can easily transform your forward transform your project to and that project while importing get small micro framework and makes it however the Web itself has a problem it's it's patient with have you ever
seen this kind of errors many this is mostly happens when uh the function of the function it become gets its small and cannot provide a response in the given time so you will get into in detail later so I created a and
imaginary but page left people uh gets some Hall and you get the DNA files and otherwise the Viennese and generate a PDF file and send them that PDF file as an e-mail and then show it thank you response many we'll get to a runtime so each functions you our imagine scenario getting the user takes only 50 ms which is possible because probably it's just about the base connection analyzing the DNA let's imagine that it takes 5 minutes which is fighting OK in In these times and sending the e-mail you takes about 2 seconds the parents yeah that redisplay the thank you Misses so let's imagine that you are the visit of this page and for the 5 minutes and 2 seconds the only thing that you see is just a white page this is definitely not the best user experience into puzzle 17 and most probably you would never see this message because most of the browsers just lose their hopes on the pages that are slower than 2 minutes so most likely your users you just see a wife page and then a page and then they will try again and and after 15 minutes they will receive 3 e-mails with the same PDF files so what if these 2 have functions could be all of our function but that would be a way to use these 2 functions to someone else to another process sometime later and continue to team and displayed a thank you message so in that case so he will get it off these 2 heavy functions and just give the thank you falls to the users in 15 ms and this is very
serious this is the exact use case also value you have a function that you want to also say said it comes to your help using
setting you can assign a test to some workers and continuity you will get into the details later about what the work produced by the test is but generally you can put everything that can be put a lot of the questions falls sigh so what these terms anything for example sending e-mail sending an e-mail can take up to 3 seconds of or maybe more or sending push notifications imagine that your you need to send 100 push notifications 2 different people just as a result of an often action residing in editing images of always attain especially value due to its high resolution images and 3rd about uh the data storage solutions like history and they out the most commonly seasonal FIFO 2 letters in my opinion and there are also some tests that take time like taking bake-offs the normalization and that the scene tissues read the top part integration this is an example of
a set of pictures so here we see that we have an application we have a message queue and we have 1 or more of workers so this is you will see in the slide that the mean time putting this presentation because this is the basic of architectures you don't uh I care for the result of the bosonic think this is the simplest possible implementation so let's look at each element and all eyes what they do the
application is our main Application I mean in this case it's about you call to the function so this is the part of where do you want to make it faster and this is the part that wants to also some of its best to someone else but it gender gender size on any kind of project so it has a great support for applications uh because an application just need to give the function name that uh it from city to execute and provide necessary arguments Drexel the application should give the function name take the arguments set it transform them into a pickle juice and according to the said it has a
built-in support for jingle it uh because once it was a genome project now it's open to all white projects it has support for class bottle period and tornado and many others frameworks it even has support for PHP so what if you're writing a PHP uh future program you can also use it because all you have to know is the name of the function that you should do you want to run and the necessary arguments so you can even serialized Jason to each the project and the test used this was the main
huge can you see in the middle and this is very important because so it needs to the court these fast for the base to a specialized publications and then processed 1 by 1 that I'm inevitable uh task use a very
tense you are the only if you heard of it but it's a very popular for a message to the database uh it's it's the most uh supported formed by city but this is also being heavily used it so that it because you can use the in other uh issues as well for example you can use the same settings this form the caching offer website and the test management and support for all other that the bases for Koch mom would be along this curious if you don't want to deal with them in the
and at workers user specific term for so you can imagine this as I have another application that is written by so guys and it will just work and hold but the base if there is any test so it always ask you have we haven't do you have any there might and vendors in the 1st to mark that test as assigned and processes so when we look
back you see that the application compares the past data which is the name of the function and the arguments sends them to the message queue which is a specialized base and as all although the basis through Monday put something on there that up on this basis for which they did an idea so it gives back yard so that the application can track the surface of the test is a processing is a process that's is that in the on 1 and there might be involved multiple workers but you should have at least 1 workers to process this tests in the queue and they can interact they can be in the same instance they can be in different uh physical machines difficult different locations so let's move on here because
to say you remember of example what you want is to get the use of the information on the belly PDF file and e-mail the PDF file to the use before being that the 1st thing that we should do is to set up reproductive this is the biggest dependency setting and for this example I decided to use this uh because it's very commonly used and also we should install said true that it's the easiest part and then that's would what you are going to trust that you are going to take these 2 functions and good give it certain so you create a new file in the test of time and the creator function is so what we do here is basically you have a function that causes these 2 functions so when I call this function beginning file any of it so 1st of all is it and then send e-mail it did touch in In order to make this as a set test mission import and define a set applications while offering range so that application of course attended many details want as in production is used so you just give a name and the book Europe so that so they can connect to this this is not the base a while creating tests and consuming that's but this is not enough you should be fine that this function don't facility so we had this the decorative on top of it and thanks to this the so that you know that it has such a test so violent dating up to workers if you look for the test and youthful puparium fast that it's not and it lakes acute when the right kind of tone and they go back to our of function and the get the had the G the DNA file usage so then the executive dysfunction in this form what you will see you not surprise us you will see 5 minutes of white scheme and then a message me because you just call that function we didn't wanted to that city the Due to the cost of this facility test so what we have to do is that a delay at the end of the function this is a short time the short cut functions there are other ways to do that we have more information but when we add this delay at the end of the function the Python about run this function but what is the defeat since package you create the task by the functions may function and the 2 arguments sense and according to the police and as soon as it it gets reported to the police and gets the ID if you continue to its routine and display the thank you message so as read this and other that that is also quite fast maybe in a couple of milliseconds it will pass the part and display the users a nice message so the users not great for the test it's nice the use of a set and so the message but here obviously the message you hear it fighting its dominance and that have been in the marks why because there's no that could decoded into the but the race we know that we have to execute this task but there is no 1 to execute so you should they got the most this is the easiest way to write a makeup of workers this there I opened this small levels to but if is you look visualization and if the type function and we have to install this is this to be what this by the
way insanity if you don't see make red lines this means that you're out of luck everything is OK it connected to the editing is moved and here you can see the you see this is a list of test so the citation there is known by accident so if said the such a fast if you just go executive and as soon as the on the city workers you will see this slide because the all data recorded a new task in the that's the problem and it will receive the test and start working on and after about 5 minutes you will see the competition message what is that yeah this is just 5 minutes because I just 3 functions inside yeah
and 1 of the nice part about it is that as you know some there's some functions that can create problems especially for quot indications value make integration then e-mail servers also and integration with the push-notification service there's a great chance that they can they can have some technical problems and if you do this in your give quantile functions assuming that your website you uh you crashed because of the top spot and if you put a everything inside that phi functions it will not give areas to users but it too much make the job so 1 of the biggest advantages of using 30 we is that you can just be try test for example you can set up a test there to try and for example you can say we tried to status 3 times by rating 1 minute before each trials this displays you can improve the chance of executing to yours the contents also certain tasks can create new tests so if you buy today it always thousands of users and send them e-mail so you don't need to do this in once you can just have a function that generates other city functions acidity test this way that your IT to lose and generate thousands of new tests the workers who long 1 by 1 all the status and execute them and if 1 of them failed only that the what you'll be executed again this is a huge advantage especially if you're dealing with e-mails because the 1st thing that can happen to that will push is not to shut down the service but syndrome humans because if you shut some server all the people who walk on the side you see that you did some bad things but if you send wrong e-mails area evidence see those unions or push-notification status same so so is quite useful on handling this kind of atoms institutions and this so that it is also a great uh tool that's the frequently use yeah it as In my opinion anyway size or any project so although it has been required to do the test In order to run the test you have a tool named so they beat it comes inside the city package and it's quite
easy to set up however many people call fuse city because they think that if they have workers alive and if they define the city you test they're eating do work smoothly but in fact that's not true because it was just accept test also they cannot they invoke the new me of tests 1 themselves so you can't you
can't considered as a separate application that sense to message queue so you should imagine just like your application so you have to keep your application a lot using provides the or something like that and you have to keep your workers alike also you have to keep the city poses a lot there are 3 types of skin
schedules the time that this reduces polished the easiest 1 you just use the time interval and the test just throngs this is dependent to the stuff final a set of beats so if you start so that it for 2 seconds after the test will be initiated . 2 seconds after the test will be created yeah so and so forth if you had the if you need to have more control on when the test will be fired you can use crawled up stages and you can say centuries these do this thing at this time for example in this case sending he made by just every Friday at 5 30 PM and it's that easy to implement this the only thing that you should take attention is the title of course because this will be find by the times of the solution so please yeah
and there are soulless schedules as well if you want to fire tests by the position off the and so on you know you can use this 1 for example if you want to fighting HIV tests then it's also that stumble you can just use solar cell said and according to so the stumble and be to send the new test just at the right moment then it's the sunset but the from
there are different types so that our dust to dust multicode does customer to so if you have time you can check all of these and find the best 1 there are great
additions there is the 1 that I want to call will be flawless because I don't have too much time flow is the tool to monitor your so that workers all they do is they are they processing the tests so I strongly suggest you to use flow which if you so it add some
sort of it's a great and simple tool therefore time-consuming tests final that there are many people who hates city fighting they have a clock heaters but I think this is mostly because all versions of the new version especially the version for quite stable and if you had bad experiences with city I suggest you to a trader you will you will be surprised you should you should never forget that argument to be serialized and then say som never posting models as an argument because they will be serialised as a stink and if you do that you will have problems the update the problems and everything and so there is sometimes is to use it cover architecture problems so imagine that you have a faulty architectures and you just being aware it can't uh or you make too much the normalizations and that instead of fixing this thing and removing them the normalizations you can easily get into the trap of using say to uh the pedals generalizations a little bit about decision so please and with that so I think I'm fine and think you a few
months of the thing I hope too many questions you will need it was the 1st 1 yeah it OK thank you for a top and I have several questions that have probably most interesting 1 is like how would you compare series to more like workflow operation tools lack of Luigi or air fall and like a when task can depend assume each other at the top like dust can produce other dust this most interesting and I have like 3 minor questions you should ask the right person I use goes the other ones so all I know yeah OK and and the question for q it all that Europe roll back and supported and that still try to normalize of some guarantees for cues because like mentioned scarce and SKOS doesn't during order men also doesn't guarantee exactly once rule for prosecuting grew and like a what if if you use as the best as is there is a chance that were thought to workers workers will receive 1 message use the expression in the problems with this curious and for 6 months to use their scarce in many projects but you have many problems with city I don't know why about they just didn't all the problems that you mentioned happened the same test uh this scene by co-workers so the the events back to pretend you need to obtain it's quite easy to set up and you feel that it's the it's storage along that is you can force so if you have the q-th and if you have a context your operations by strongly suggest to have retained few not this not as uh but Georgia assume met specific message queue the system OK and like the question is about contacts Kedua music your example was its involvement prisons in normal Kron at was a start standard chronic you're issues with this scandals like last Sunday of monster the the explicit problem I'm using force so like he's based crime or like it's a super complementation of constant or you might as well as I know it's a separate implementation but because problem it's chrome is that when you change 1 of the schedules as it calls to a text file the less uh runtime and it creates problems so it might be possible that it is being fired before you want so what I do generally at the beginning of each set of tests that will be fired by protest is to add a point 40 and at something to reduce some expire time and make sure that it will never and overall before time especially this is crucial in in in May because for example if you want to send a mailing Friday and you change the time Monday you it just gets fired so in Viet control looks at the beginning of the test of hand final question Our about or you have an application object of celery at in work time this is situated in working part time as have side effects of like grating connection 1 just you import what the great instance or if lazy it I don't think this is a problem it like because like when you start test and if you have inverse side effects in in Britain effects and you generally bad practice yeah but why the distance in the painting from and you think you and the next question is that something for diversity it I think even talk 1st of all and now the question is what is the advantage of the salary if I'm using it only for the background workers from in comparison to you riskiest pool of for instance in my opinion the biggest advantage is that so you can use it from so you don't need to uh you don't need to operate it 1 is single thing and you can send test from various locations you can use by the public that supports for example periodic test so if you have a so the installed in your architecture you can just use it in in many tests as possible and also available if they have the trial months they have I didn't know OK so I think 1 more question and I think he's still over there so I think you can ask them all the questions I'm pretty sure you at OK so think if another helped soul using using these having production we have any takes on how you operates a efficiently what kind of message the using mentioned before you must make a switch from the Amazon Services the rabbit and the is that what using production for the existence and yeah uh you tried to do this and looking through the book for its mirror what Lucht inches of course but they're the only problem that I see when you have told me because I mean like millions of dollars I have some friends here that that dealing with some uh e-commerce business and yeah there might be problems where you have to watch the test you should there to take attention to the overall performance of the system because that might be object dependencies the same objects might be used by different tests and you should want that as much as possible because it gets the called complicated it gets the debugging complicated so you shouldn't too much test especially compared to OK and what happens if you had asked that as the mean time the beaches to write intention would use the DuPont say exactly like our that analyzing in a and and then you say Sandy manifested that mean that you find time around this only just write you make tacos there's anything inefficient you waste 5 hours of compute time for the 2nd year so that's why you can you can divide into subtasks stands for the generating recursively grace of from an idea 90 Ben's I scheduler thank you yes and if OK thank you OK so let's spend he again MIT and