Tasks: you gotta know how to run 'em, you gotta know how to safe' em

Video in TIB AV-Portal: Tasks: you gotta know how to run 'em, you gotta know how to safe' em

Formal Metadata

Tasks: you gotta know how to run 'em, you gotta know how to safe' em
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Web developers often find themselves in situations where server processing takes longer than a user would accept. One very common situation is when sending emails. Although simple and relatively quick task, it requires the communication with an external service. In this situation, it’s not possible to foresee how long that service will take to answer. Not to mention the many unexpected situations that can arise, such as errors and bugs. The solution to this problem is to delegate long lasting tasks while responding quickly to the user. This is the point where we need async tasks. There are some tools available that can assist in this job. In this talk, you will learn about the concepts, caveats and best practices for when developing async tasks. For this, I will use Python’s most popular tool for the task: Celery. Rundown: Setting the context (~3 min) The architecture (~3 min) Brokers Workers Use cases (~2 min) External calls Long computations Data caching Tools available (~1 min) Celery (~16 min) Callbacks Canvas Logging Retrying Monitoring Tests and debugging.
Computer animation Task (computing) Task (computing)
Service (economics) Software developer Expression Twitter
Wrapper (data mining) Information Open source Boilerplate (text) Projective plane Order (biology)
Context awareness Game controller System call Service (economics) Multiplication sign Real number Database Disk read-and-write head Quilt Subject indexing Goodness of fit Process (computing) Befehlsprozessor Cache (computing) Computer animation Insertion loss Personal digital assistant Task (computing) Task (computing)
Web 2.0 Component-based software engineering Computer animation Connectivity (graph theory) Software testing Service-oriented architecture Sinc function
Computer file Connectivity (graph theory) Multiplication sign Execution unit Set (mathematics) Peer-to-peer Radical (chemistry) Process (computing) Computer animation Software Musical ensemble Service-oriented architecture Descriptive statistics Physical system
Dependent and independent variables Code Bit Letterpress printing Event horizon Number Web 2.0 Process (computing) Event horizon Object (grammar) Right angle Object (grammar) Endliche Modelltheorie Task (computing) Resultant
Area Complex (psychology) Multiplication sign Complex (psychology) Parameter (computer programming) Parameter (computer programming) Revision control Web 2.0 Computer animation Personal digital assistant Computer configuration Object (grammar) Software testing Object (grammar) Lie group Service-oriented architecture Task (computing) Sinc function Address space Resultant Computer architecture
Operations research Presentation of a group Multiplication sign Computer Software bug Mathematics Category of being Mathematics Error message Computer animation Operator (mathematics) Computer science Category of being Error message Resultant Task (computing)
Area Operations research Dependent and independent variables Context awareness Idempotent Multiplication sign Bit Cartesian coordinate system Perspective (visual) Number Mathematics Event horizon Series (mathematics) Operator (mathematics) Core dump Reduction of order Series (mathematics) Resultant
Email Group action State of matter Code Multiplication sign Set (mathematics) Parameter (computer programming) Mereology Newsletter Cartesian coordinate system Facebook Medical imaging Bit rate Hypermedia Cuboid Social class Physical system Exception handling Email File format Sampling (statistics) Fitness function Uniform convergence Facebook Computer configuration Phase transition Right angle Task (computing) Spacetime Row (database) Random number Overhead (computing) Divisor Limit (category theory) 2 (number) Operator (mathematics) Software testing Task (computing) Alpha (investment) Consistency Code Exponential function Line (geometry) System call Software Atomic number Statement (computer science) Exception handling Buffer overflow
Web page User interface Information File format Control flow Login Login Cartesian coordinate system Rule of inference Tracing (software) Demoscene Software bug Computer animation Personal digital assistant Network topology Video game Software testing Information Error message Information security Task (computing) Task (computing)
Trail Beat (acoustics) Multiplication sign Resonator Cartesian coordinate system Product (business) Centralizer and normalizer Process (computing) Lecture/Conference Software testing Quicksort Error message Position operator
Duality (mathematics) Graphical user interface Computer animation Confidence interval Interface (computing) Adaptive behavior Limit (category theory) Product (business)
Area Uniform resource locator Computer animation Source code Set (mathematics) Software testing Software testing Task (computing) Task (computing)
Point (geometry) Complex (psychology) Information Consistency Data recovery Complex (psychology) Shared memory Parameter (computer programming) Parameter (computer programming) Computer animation Object (grammar) Video game Software testing Right angle Software testing Object (grammar) Extension (kinesiology) Information security Task (computing) Physical system Task (computing)
Slide rule Open source Link (knot theory) Information Water vapor Insertion loss Checklist Newsletter Twitter Checklist Newsletter Process (computing) output Compilation album Task (computing)
and of the and and
and and the thank you all for coming so and I'll be talking to you about us gross tasks and most specific specifically about sorry on so
you have an innovation and as a she was my nickname and you can find it on Twitter as she not I'm from receive a there's a sitting Brazil nor fist and n yes and no work at
faintest so 1 of the partners there and we are a service common from a 0 on most of our Pinter from the US and Europe and services who mentally and we have a team of 11 express developers and work with genuine react and this is the the sharply
playbook II we can't we compiled there are a lot of information about how we work the things we do so few features that were there we also
do a lot of open source so year some project so you might be interested there's and ah plates on it's a jungle react what re-equilibrate on we have generous permission but joke and there's a few orders of only help so feel free to random
yeah but still establish some context In brought so the reason we use
ICT tasks is because we want to answer quickly to our users are some reasons
on some situations where we might want to use this index that's the simplest case will be when we want to delegate long last CPU jobs but the most popular reason people use it seems that astrologers acute external get bagels whenever independent of service you no longer have control over how long things will take to be ready the it may also be the case that we will never be you will never be ready this is rely on my be down or broken Algren good reason to some users see this syntax is to prepare for it to prepare in cash about values on and you can also use them to spread local that evisceration over time this can help you not to the DOS your own that leads grow jobs are also another good example things you can do with them let's have a little there about how they work the problem of running a seat that can be easily mapped to the producer consumer problem produces place jobs in real consumers then check for the head of and the head of the quilt for waiting jobs being the 1st 1 is a acute there are
many tools available to manage isn't there some by act gruesome civilian how efficiently but sorry is the all-time champion so far so I
several summary 1st of all let's introduce
department your components From now on producers would be Web nodes Our who will be called broker and consumers of new workers since workers can also place new test in a pool they can also behave as producers now it the mother have on the
basic components we can dig a little deeper the concept of a broker is very simple but having England isn't peer system there are
many ways to do this 1 of the simplest will reduce the text file and thus can hold the segments of job description to music unit therefore we do good is then as a broker of all systems the problem with text files is that they're not made for 2 and we obligation problems such as network and computes an axis because of that we need something more robust support of based their hand are capable of running a network and dealing with concurrent access the problem with them is there on choose low no other based on the other hand are quite fast but many times the like regarded so I knew it includes we should use fast and reliable computer enable onto a set such as radical rightist West
sorry has full support for had an equal in red although ask innkeeper also will the are offered with clinical cavities
let's not talk about web in Working Notes on the left we have a code that should run at a single that goes and where model work on the right who had called the code that place the request for a job to run this no legal and had no In this example you have no place in that job in which for the result to be available response is ready there isn't sprint now I have a gendered simple had you have no request that the Web nodes request and the number of these of the event to be of a bit of a single only notice we are passing the event objects to test to the to the that don't do it all etc.
lies in storing the broker they're in this so as before best to test Besson complex objects such as moral since as the parameter can have a few problems 1st of all known versions of cellaring because use it to do as a full-service solution method as some you know people as secured that nobody's well by a lot complex objects you increase in the chance of getting exposes litters versions of this of the sorry address this by using Jason as the false so method another issue on is that that these objects that you pass my beach might have changed between the time you place the the desk in the time together persecuted the In that case you'll be working often with an updated version of the object what you want to do is about the idea of the
object and fetch a fresh copy for we've been calling on delays and get together all the time with that 2 separate things delay place the testing in acute by worker In returns a promise they can be used to monitor the status of the status and get the result when it's ready going get on in that promise will block this ecution onto the is of debatable the act as has has just started result some out somewhere and then act and then and whenever it finishes it will be accessible onto the prostitute attributed test the test this means that we need some piece of the all of the architectures it besides the web broker in Working Notes non compliance there is also results back and the results back and will be used to store the best results in practice you can use the same reasons you're using for the broker to also store results the there are all a technology decides the supportive broker options that you can use due on in their results they can but there are some difference depend on 1 use the Bosporus for example they get metal with appalling to check the result of what 1 at wins the results is ready but another stration searches of those 4 areas designed by but server listen how as a kid
tasks in house so works but the real on hard problem I wasn't says that is handling errors how to prepare for bugs and failures we're not going to roll over 2 concepts are essential for that matter if you are to learn something from this presentation land the
1st 1 is the idea put potency and what is it is a property of something of operations in mathematics and computer science that can be applied to which both times they can get away with what times without changing the result beyond the initial obligation with my way
0 and the defined by 1 i 2 examples of a whopping operations was that by an animal 0 no matter how many times you do it again the result we always 0 is the same form with by my 1 no matter how many times with quite a number where 1 the result will never change let's the for and it should be which of these methods are highly potent getting there in resource should not produce changes in it so get is an important both is used to create resource but at least that is what we used to do before or after all on so every post requested is I change this to of occasion before it's not only it what we cannot expect subsequent to of course request you keep applications stayed the same was a great example of what an operation the 1st could request to make up for this change in the state but if you keep repeating the same request no changes are expected did is that it did it is a bit of a gray area from the applications they perspective it is an important from response response respective the 1st column I return to 0 4 0 the next 1 might return a 404 so depending on the
2nd common concept is optimistic at the cooperation is indivisible and reducible series of inspirations such that they're all core on on acquire despite has being commonly assumed that 2 these operations the concept of of optimistic can also be applied in other contexts
the the 1st books we have a test that sets this you start to appear since then makes a request Facebook and only then updates the user name is is not only if the request FAO's you're going to have inconsistent statement of these do I want you to do it is to 1st make the request and then update the user status in user name at the same time another way to improve item is the is rights for tasks In the 1st example we are entering and sending e-mails to all users in a single if 1 of them fails is that best will stop and part of the URI users received it up the male and the other part will not In the 2nd example we don't over of again with the ligand descending to or that before the fail you offer the only we will will affect only 1 uses the other advantage is that it's easy to rerun at that and your it is an overhead on creating Shiites text that's make them too fine-grained may harm performance so to keep in mind when you're design the so just tell you about and how you can hit right that's that they found a common situation for test affair directly if external such systems they're not available so provides a read-write method that can be called and said this Xu make it try again to kid that and sample where fetching the user likes on Facebook if it fails we rate 10 seconds and we tried if your tests and important anatomic should have no problems going to try as many times as needed and to the test seeds if they're not retry neprilysin con consistent states or conducts by media users it there are some Monica gets through trying the previous example the request Facebook might be fading because there is phase holds under attack so it might be a good idea to back off and give it some space recover you shouldn't take a male the ground I'm not a good idea is to make the interval between tricycle explanation this agrees the chances of things getting back to normal before you next right also for another a random factor imagine having 100 dust fitting in metritis in time the overflow might actually be a reason why the system you're interacting with is not the 1st place all right this is something action learned about on the 1st day of the Conference Palace touching with David Bohm gold and so if user have so for you can actually go down the lines of code by pressing the UltraDrive 4 parameters you can do the same thing I should for just using the of it right for invested on the classes the and their classes I also learned that they discover request approval fiery Tribeca of parameters the next so it really is moving images we try back off along with alpha try for parameters and explanation back off of the box so thank you Dave ch our common
issues are test that they too long to the kid because they are that is there's a book there of we have a problem of network latency to present to prevent this format from harming the performance of the obligation you should you consider a best buy new salary will put the test on sorry were there of the test if it takes longer than the time he said occasionally to some per recording before the that's interrupted also said tests of time when they're runs of salaries soft than exception and you can do some operation before the skills the late setting is of those of you should know about but if 0 sorry 1st makes the call and that makes that remarks at best as ran and then persecuted this prevents us from run twice and this is often unexpected shutdown having more than in at the desk will give you the ability to turn on actually it's settings by doing so automatically rerun the endurance test was the course it we start a
lot of a lot about how to actually have to errors but the rule know the books on that the inevitable it but is not levitation journey is just what if something breaks your user rubella 500 page and they will find a way to let you know that something went wrong for them for a scene that is not always the case you manage ministrations will be dealing with things that do not on directly affect the user experience this means that you should be extra careful with monitoring tasks morning tree will help you to capture errors early and not when they're already too late Starting from the basics of life on
make sure you log in as much as possible this will help you traces what went wrong when bugs arise as usual be careful not to expose sensitive information the logs is a security threat log is a general advice for any kind of application with important for tests as they had no user interfaces and then the harder make
sure people get not firemen things fail those sort such as central cannot be it can be is integrated John salary and will help you monitoring errors you can also integrate then we select so get notified every time if something goes wrong make sure to find their what promised applications too many false positives and your team will stop lately Bayesian and the actual resonances I have always find process to do if those eras mission including backlog Paris ties in accordance
4 is a tool for like wondering sided tests allows you to inspect which texts are running and keep trace out to keep track of these occurrence it says that obligation and after the war for using in bigger products general sorry beats
status is something that just came out we didn't talk not talk much about ask desk with the and not to go to monitor and test it the crontab API is sometimes confusing and my lead to mistakes and and this is also product from finger general surrogates status will
adaptation to begin with an interface and it UPA which seen all scheduled tasks along with their necks ETA this will give you a limit to what we do is give you a little more confidence not elections right
in the learning tasks to be harder than with what we used to know whether locations but there are few things you can do to mitigate this best
always eager is assessing the comes very handy for testing in about this goes from this area source the if you had that that's always here in the 2nd set to true whenever called delay or a plastic you run the tests ghrelin delegated will simplified about you know we environments and facilitate all my best the last thing
on is actually more of a fun fact so turns out there is a pity for salary RGB were lawyer took down at your breaking point in life devoted To be honest I'm never use it cannot militia share you use it but I'm sure someone will find handy right so pantry captain
on the 1st floor values complex objects and by the and best parameters you will help you avoid security and consistent issues this 1 consistency issues right I wanted the ultimate test you want to be able to freely rewrite your tests that golf when you try or systems may maybe some space recover make us extensive use of modern monitoring tools thus the heritable the book collect as much information as you can get notified when something fails the but this could be careful and drawn-out fight when something portend happens and last use task always eager for working and the last thing I want to
show you something that I've given process of making this that the stock so on best answer does checklist has a compilation of the things we talk about and you can actually keep for the diet items as you verify the water missing your tasks and it's also open source so you are very welcome to con to would improve it
that's it thank you few from through input but some good you the to the slides are there I so it's the election mean 2017 of there is loss of the Lead user this checklists my contact information and of we have a newsletter if you want you sign up to this it's the links the thank you thank you so much Philippe if you have questions land definitely talent and after its origin that we have been working on I think you and were about half closings and hang out closing hands you the fact but this few
and SHS and are