But, why is the admin slow?

Video in TIB AV-Portal: But, why is the admin slow?

Formal Metadata

But, why is the admin slow?
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
This is the general outline I'm working from so far. I think this could change slightly as I develop the talk, but this outline conveys the general theme. Introduction and display of basic django-debug-toolbar usage (2 min) Things the admin does well (3 min) Makes development very fast For many use cases, it "does the right thing" automatically. For example, modifying the HTML in a callable won't cause new queries. What can sneak up on you (5 min) Having lots of related items visible in the list view Using list select related Overriding queryset for additional select related and prefetch related options What to avoid in callables (3 min) Queries that will be executed on every row The default widgets for many-to-many and foreign key fields (3 min) What widgets to use to replace the defaults based on how many options you have in your database Custom aggregates in the list view (i.e. custom querysets) (3 min) When this is a good idea When this is too slow and you need other options More general performance improvements through caching (3-5 min) Django's caching framework Caching with third-party packages / tools Custom caching with Redis Questions (Remaining time)
Computer animation Multiplication sign
Multiplication sign System administrator Electronic mailing list Formal grammar Arithmetic progression Reading (process) God
Web page Mathematics Computer animation Code Line (geometry)
Web page Functional (mathematics) Meeting/Interview Code Query language System administrator Damping
Boss Corporation Meeting/Interview Multiplication sign Debugger Client (computing)
Authorization Cuboid Endliche Modelltheorie Library (computing)
Web page Slide rule Default (computer science) Greatest element Server (computing) Mobile app Information Code Multiplication sign Gender Software developer Set (mathematics) Directory service Field (computer science) Template (C++) Computer animation Query language Personal digital assistant Authorization Figurate number Endliche Modelltheorie Library (computing)
Web page Default (computer science) Server (computing) Information Demo (music) State of matter Gender Multiplication sign Debugger Virtual machine Electronic mailing list Set (mathematics) Product (business) Integrated development environment Query language Local ring Physical system Identity management
Overhead (computing) Information View (database) Image resolution Electronic mailing list Instance (computer science) Connected space Revision control Computer animation Software Query language Object (grammar) Local ring Physical system Row (database) Library (computing)
Mobile app Server (computing) Overhead (computing) Sequel Code View (database) Execution unit Function (mathematics) Product (business) Frequency Different (Kate Ryan album) Electronic visual display Endliche Modelltheorie Error message Key (cryptography) Information Gender Electronic mailing list Mathematical analysis Bit Database Line (geometry) Connected space Software Personal digital assistant
Default (computer science) Computer animation Key (cryptography) Execution unit Electronic mailing list Electronic visual display Bit
Web page Computer animation Key (cryptography) Electronic mailing list Electronic visual display Row (database)
Default (computer science) Computer animation Query language Gender Endliche Modelltheorie Mereology Field (computer science)
Inheritance (object-oriented programming) System administrator Content (media) Set (mathematics) Database Disk read-and-write head Field (computer science) Computer animation Semiconductor memory Personal digital assistant Query language Core dump Endliche Modelltheorie Object (grammar) Spacetime Exception handling
Functional (mathematics) Standard deviation Computer animation Key (cryptography) Query language Multiplication sign Right angle Line (geometry) Object (grammar) Row (database)
Optical disc drive Computer animation Query language Personal digital assistant Multiplication sign Electronic mailing list Authorization Bit Right angle Field (computer science)
Computer animation Key (cryptography) Multiplication sign System administrator Direction (geometry) Authorization Field (computer science)
Word Computer animation Personal digital assistant System administrator Authorization Set (mathematics) Exception handling
Information System administrator Gender View (database) Electronic mailing list Counting Neuroinformatik Computer animation Core dump Authorization Right angle Physical system Library (computing)
Functional (mathematics) Mathematics Computer animation Key (cryptography) Query language Gender Counting Bit Quicksort Field (computer science) Row (database)
Mathematics Computer animation System administrator Core dump Authorization Counting Database Water vapor Endliche Modelltheorie Field (computer science)
Web page Filter <Stochastik> Randomization Greatest element Presentation of a group Digital electronics Sequel Code Multiplication sign View (database) Execution unit 1 (number) Control flow Machine vision Theory Number Product (business) Revision control Meeting/Interview Term (mathematics) Core dump Touch typing Electronic visual display Cuboid Endliche Modelltheorie Mathematical optimization Default (computer science) Arm Key (cryptography) Gender Software developer Debugger Electronic mailing list Counting Bit System call Word Angle Query language Order (biology) Pattern language Object (grammar) Figurate number Resultant Row (database)
1 but that would be the time and it will 1 who the of the 2nd half of the time and the and and cello of thank you and said my name is just initially this is my
2nd time speaking Agenda Cont I'm very excited to be here with you guys have all been having a great time so far I've been using Django for a little over 4 years and I've given a couple of tutorials on the admin I love reading and growing but haven't had time to grow much recently and I found in the tutorials that I gave that performance improvements were always a topic of interest and in particular there were a few really easy things to miss and a few really easy things to change particularly around the list you on that people always seems to find interesting so I decided to develop a top grammar the basic progression is that
you just started using the admins you're relatively new Django and people then tell you it's a skillet feature the 1st you feel like this I am a God I only
wrote 1 line of code and I have this so web page now people start using an
they start asking you to make feature changes it's really easy you change 1 line of code and they have what they want you say they say can you show this column absolutely can you order by this column absolutely all of these tiny little changes that are normally somewhat difficult if you're doing them by hand and Django has provided all of this to you for free but there's always about
them you haven't changed any code but all of a sudden
you're users start talking to you about how this page seems to be loading really really slowly you did this is ground so you might suspect that it's something to do with queries but you're not really sure you're in the Django you're not really sure how to figure out what's going on here of and now is thinking of and have to scrap the admin it's going to be a month or 2 months 3 months of work to rebuild all of this functionality from scratch in years starting to tell
your team needed client your boss how much time is going to cost how many features you I have to put up doing seek rebuild the admin and they're like no
in it so when we do In this talk we were going to go through by example exactly this case and we're going to use on a debug tool that is very common that only very commonly used in the Django ecosystem and if you haven't heard about it yet this talk is probably worth it for just tiny you acting that developed to lock the
example that we're going to be using as a library In this
library we have users we have authors and we have box people can check out books and that is represented by the loan folks model which is a many to many relationship between a book in the user the to can see this here on down toward
the bottom we have our books relationship which is many-to-many for a library everything else is basically just default user fields we have an author model which is incredibly simple we just have the 1st name and last name of the author we have model which has a many to many relationship with authors because many books can be you know about can have many authors and author can have many books thus it keeps an ancillary information that's useful to library like the title and how much of would charge you every day if he returned your booklet finally we have the actual through model so when you have a long book you have a relationship between the patron who check the book out the book itself when it's due to whether the finds have been paid a not how many times has been renewed and things like that original code for this has more comments in it but they basically just say what I told you right here so I remove them so they make the slides Figure the next gender debug
toolbox it's on get the installation is very simple just added to install apps for a local developments of using 1 server typically or I have to do is add this to installed apps and you get this nice um Bora sidebar when you click on each of the fields will tell you additional information so you can see how long it's taking the page to load what your settings are on what really concentrating on is how many queries you're running by some other useful things in particular templates here had a case where the template isn't loading and you're not sure what directories being looked at at this will give you that information if you click on that kind of sub field the a quick
check if your local development environment is a virtual machine but there is an internal IP setting which gender but to a lot of what this is is a white list of I'd he's coming to the server that you want to show that sidebar to because for example if you have a dead server that sort of accessible by all of your dad's stands might be public state facing you don't necessarily want the tool are showing up or demo server for example I'm where it's safe to have the debug tool bar but not some but you don't want it to be shown to everyone visiting you can restricted by internal IP is also if using local BM but it sometimes wants what it perceives the VM of your local system to be and generally had never use Django debug tool bar on a production server unless you're absolutely certain that it is not facing the world and because it exposes a crap kind of information about the system that you don't when exposed the whole world so obligatory warnings
out of the way the 1st thing that a users access us to do is to display a list of all of the books that are out on long so in here but we see the loan book are unique to a definition had the name of the user who checked out the book and the title of the book itself is all randomly generated by gender by default will show 100 rows and as you can see here we're executing 204 Siegel queries every time we look this page this this seems excessive the so what can we do about that well you could buy identity but to
allow allows you to see what queries are actually being executed in in more recent versions it hopefully includes information about which queries are actually being duplicated so the resolution this is a little poor but what's happening here is for every row in that list view it's requesting the objects and the library user object as well as the book object for every single row and this has a lot of overhead especially if you're not using a local system there's network latency there's overhead to actually set up the connection and zoomed then you
can see the specific instances happening so book and then you'll hear over and over and over again so to recap
from were seeing is this was our unit code definition it's the join of self that patron self book so the user in the book now because it's in a unit code method GenGO kept going to see that the the foreign keys and should be using select related so we have a couple of different solutions that we can use here solution 1 is to use with display and explicitly include those foreign keys because if you use a foreign key in enlist displayed gender was smart enough to know to use select related in that case it doesn't know to use it if the foreign keys that you're accessing are inside of a callable only if you use them in list display solution to if we wanted to keep it in a callable and not useless display we can also set list select related to true what this does is for all the foreign keys on your model but it will fetch all the information for all the foreign keys again you can reach from them all the solution to be you can actually be a little bit more explicit as a Django 1 . 6 plus with list select related and specify which foreign keys you want so if in the last few you want to i so you're using only 1 of those foreign keys in a list view but not all of them and you don't want into the disco randomly fetch all the foreign keys that can possibly reach which it will do on and which can also be costly performance-wise I you can specify specifically the foreign keys that you're using analysis to so say we apply solution 1 this is the output that will get
we now have 2 columns 1 for patron that shows the name of the user checked out the book and 1 for book which shows the title and we've gone from 204 quarries to for periods and more importantly we've 20 x speedup from around 28 to 30 ms to around 1 and a half this is using sequel light all of connection if you had a real production set up where your database server is located on a different server from your app server network latency and connection overhead and post press on but this would be even more significant so 28 speed up with 1 line of code error
so coupled notes because we can dig a little bit deeper honest of what if we had used the 2nd solution and if you remember I said that you can specify which foreign keys to use of it and we we left our list display went back to using the default unit and and just specified patron what happens is we end up with
104 poories because for each role in the
column you're going through and it knows enough to you select related to get all the users but you didn't tell it that you still needed books so it's going through and it's executing a separate Corey for each row to get the title of the book so that's that
the reason this is important is
sometimes you're requirements will change and where previously you only wanted to show 2 of the 3 foreign keys in your list you users now ask for you to show the 3rd foreign key don't forget to update it in your list display or you might have complaints about how old 7 pages learning slowly
right so full solution for to be some something what worries
of but 1 thing that is interesting to note and that is another part of gender default behavior that a lot of but novices a beginner's guide I don't know some or don't realize or don't realize the implications of this is that for every query by jingo gets all the fields on your model no matter how many of those fields there are and no matter how you doing a whether it's to the island of 3 or and if you are getting a model jingle will cache all the fields by default unless you tell it not to so we were only using book title use 1st name and use a last name but we're getting all these extra fields let's
see what we can do about that because in some cases you'll have fields that are very large and Django is fetching them from the database even though you never use them to take a lot space memory for slow what you can do is use of custom Corey set and this is 1 of the most powerful features of the Django admin in my opinion on in something that I actually use relatively often so and I could have alternately titled this talk custom presets for fun and profit so what we're doing here is the or em has a method called only which allows you to specify specifically which fields from your model you want to that from the database so in this case what we're doing is we're overwriting to get reset method we're taking the query set from the i from the parents and then adding only patron 1st name hatred last name and booked daily find yes but daily finds mistake that some purposeful will get to that you can modify the core set to your heart's content as long as it's I actually returns a quarry set at the end because I if you forget to return a chorus that you get a very uninformative there that says that your database is not properly configured on my mere may not have right across that aspect my head for a few minutes when I was working through the examples in the stock the but there's also a corresponding method called defer that does the opposite of only so if you use for a set of objects that differ In agrees said what that will do is it will load everything except for the fields that specify so if you only have 1 really large field that you want avoid loading because you're not using it that worries that you can use differ for that instead now I did make a mistake here a you previously we may have shown the daily fine ends that there were 2 great but I'm supposed to be showing the title but what happens if if I leave this mistaken
and it with 104 queries again something about it
so what's happening here is the of the price that is doing a lot but in that 1st query that's listed were getting 1st name last name and the primary keys and daily fine and that's it now what you must do on every role because you did not catch the title that you needed is every row it now goes fetches the title because it realizes that the last 2nd that it needed that time this is this is an efficient it's very easy to
fix just put the correct of the correct reference in their booked under Title instead of book under standard daily fine but I just want to point that out because oftentimes if you're using only if you really wanna check your quarries after that to make sure that you're not using something 10 lines down the function that you forgot to get in the only earlier because then you're data you're binging a will have to go and fetch that for every object in the court right so once it's
fixed about 4 queries great I we have a minus the out from our previous speed odds about . 3 ms in this case which is a lot because we don't have any large fields and were not saving a lot by leaving out the other fields but forget about fields this would be useful but let's try something else a little bit more
complicated 1 of the users has come along and ask well this is great but in the books I would like to see a list of all the authors and books the the I all the authors on the right is the the we we add callable called authors display so we add the authors display callable to list the spike and because we learned a lesson from last time they were using was select related were setting it the true what could possibly go wrong well 104 queries
again the reason for this is
because authors is a many to many fields not a foreign key select related as for foreign keys prefetch related is what you
want if you're going in the other direction is a duplicated 100 times and just getting off every time so for going in the opposite direction of your many relationships were you doing a reverse foreign key look up you what prefetch related and unfortunately there is no switch from nests in the Django admin
where there is for select related problem because it would be more difficult but there it's something yeah someone will look into some day I instead what we do is another custom for a set of said then where do the same thing except in this case we take our book admin get the course at the Django is using prefetch related authors and now 5 words this
is the actual core that it's running but gender computes in advance all of the primary keys and get a prefect related and off the rest so this is useful you can put more information in your admin without sacrificing performance and you can keep the admin around for longer which means you can develop features and get here system up and running and off to the races are right last example that we're gonna go through is
counting so now uses a combatant like alright and we've done so far great we want to see how many books the author has written that we have in our library in the list view a right we can do this article self that works that count add the list and book count to the list despite and we have 104
queries again because we have 1 count worry for every row this 1 this one's a little trickier red by and how do you deal with this there's there's no foreign key here there's there's no adverse foreign key relationships this let's related professional late enough and work or you have to do a custom caching when we what we do here well the answer
is on we use changes aggregate functionality so 1 of the interesting things that you can do gets is annotated um and what that does is it takes the aggregate actually adds it as a field sort of on the agenda knows how to deal with this so this 1 is probably a little bit more complicated than beginners in the room and solves that threat but we still have counting was displaying but in our query set what we did is we used in and we said that book count should be the name of the pseudo field on that we're using and were doing account of works so now you can actually reference that in the in the civil and gender knows how to refer to that and pull that out so now we have this yeah we
are back to for quarries once again and you can see here the quarries that are being
executed where are you have that has book account that allows changes to refer to this within the core itself you can do everything in the database in a single query and as a side note
on you can In the author model where he defines this callable you can't remove the callable bond in a kind of freaks out even though I don't think it should be but you can set the count . admin water fields to be book count and then you can actually sort by how many books in author has written and you can order by about which is pretty nasty of a few random notes before it so that's the
end of all the examples of few random notes if you have an issue and you just can't get things working fast Nevins you what's really quick hack these solutions on you can decrease list per page which will just decrease the number of rows the you have a single page and it maybe give you a little bit of extra time on to figure things out before like all hell breaks loose so by default angle 100 you could reduce that to like 25 and then you're quarries will be decreased by 4 x are to you actually figure out a better way to do things I show full result count is if you have filters on your heart shows the count how many objects you have total that's if you have a really large number of rows on that Korean itself can be slow show full result count was introduced in 1 8 new removes that but it doesn't remove the default full pagination at the bottom the arm which also does that worry so you need to remove pagination if you're doing that and so I don't recommend any of these really as a long-term solution but they're just a quick facts if someone is breathing down your neck and you need a way to it time to actually fixed problem the cinema presentation thank you all for listening
without and I think we have maybe a couple of minutes for questions if anyone has hasn't the the the it and I like it so you chose about stuff yet how much of this could be like gender core and to you need to be aware of Is there actually were from progenitors of course you have to what's so that's that's an interesting question I think a lot of it would have to be introspecting call ables because that's where you end up with some of the problems of I don't know how I'm sure that you could build that optimization I the don't know what the priority of that would be compared to like other features that people really want because basically we have to do is like introspects Dundar unit code methods to see if people using foreign keys and actually look at all the calls that people are putting in was display to see if they include things that could be automatically select related prefetch related Hamilton that enough to know if it could be baked and it it might be possible to do something clever with that ends include hedging to detect that automatically yeah my thanks for the great talk I use gender devoted to a lot to sort through queries of uh you know it's a custom with it but I never thought at it's a circuit and vision and wondering if you have any advice on you know sorting through the list of best will theories because it can be you know expensive like you're saying 100 and 4 or upwards and how can you still that down to you know exactly where is it calling all of these queries with your code I've had some trouble with that and wondered if you have any advice for that so I think within the I it's usually easier to figure out then if you're on a custom web page on because it's a lot to say it especially with like the list view of the core result and be in order if you have issues it's generally because you're executing queries on every row and and debug tool bar will in the most recent version the mentioned hopefully indicate which ones are being duplicated and shorter that the it the way that I typically approach it is to look for patterns on where queries are being executed on things that I think should already have been cashed in the code and there there's probably an opportunity here for someone to write a tool to actually go through any analyze some of these quarries and make recommendations specific to Django about you know how you can use the orange figure this out but it's right now there's there's not much there it is they can do beyond being familiar with the models and familiar enough with how queries are executed by the or em to be able to see the patterns 1 thing that you can do and that I found helpful is to purposefully by right queries and look at the sequel that's generated on just as a a learning experience and because that will help you get a sense for the relationship between things that you're writing in Python and so the number of queries are actually generated this if you're using a debugger can you like Paul's halfway through it and look dreamed of the book to words like shown in half of the requests and don't think they do that with gender they don't toss models might know but I do know that on if I'm using the debugger and I have long been turned on so it's actually logging Rockeries as you step through you can see which queries are being generated at each step the thank you so in terms of what your view of boxes perhaps like query can morning like India but more than these that you during over 50 queries just you know pronounce no pages 300 where's not into the so that idea 1 sprinter coming up in I also just obliterate note of the company or product on demand is is hiring and gender developer itself if anyone's interested in touch there are no more questions thank you all intervals if what thank that so if you don't mind paying the fee