Django Rest Framework, one year after: tips, tools, tricks and pitfalls.
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33822 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 20173 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
Software frameworkSoftware developerDirection (geometry)Element (mathematics)
00:23
Transportation theory (mathematics)Core dumpLine (geometry)RoboticsGraph coloringOperator (mathematics)Context awarenessGoodness of fitEndliche ModelltheorieCartesian coordinate systemStudent's t-testSoftware frameworkComputing platformSheaf (mathematics)Point (geometry)Core dumpMultiplication signLogistic distributionSuite (music)Decision theoryComputer configurationSystem callBitWeb pageSerial port
02:46
Software frameworkMoment (mathematics)Area
03:03
SummierbarkeitAttribute grammarSerial portQuery languageMultiplication sign
03:31
Query languageSpecial unitary groupMobile WebInheritance (object-oriented programming)MathematicsDynamical systemUniform resource locatorMaxima and minimaExecution unitWechselseitige InformationTerm (mathematics)AuthenticationMoment (mathematics)Point (geometry)Logic gateField (computer science)Endliche ModelltheorieCASE <Informatik>Expected valueFiber bundleTracing (software)FlickrNetwork topologyBeat (acoustics)Student's t-testPrice indexSemantics (computer science)Mobile appStaff (military)Software frameworkNumberExecution unitLine (geometry)WordType theoryMusical ensembleFormal languageWater vaporEmailRepresentational state transferUniform resource locatorMobile WebLink (knot theory)Query languageSerial portAuthenticationParameter (computer programming)Attribute grammarValidity (statistics)Data dictionaryGroup actionDefault (computer science)Subset
08:59
CircleSquare numberFinite-state machineSoftware frameworkMaizeSoftware frameworkPermanentMultiplication signData managementStudent's t-testCoordinate systemValuation (algebra)Execution unitState of matterBitEvent horizonPresentation of a groupPrice indexTheory of relativityBeat (acoustics)Compilation albumLibrary (computing)Level (video gaming)Table (information)SimulationWater vaporTrailMoment (mathematics)InternetworkingCuboidView (database)MereologyProper mapConcurrency (computer science)AuthenticationMatching (graph theory)Field (computer science)Endliche ModelltheorieSerial portData structureCalculationFinite-state machineValidity (statistics)Profil (magazine)UsabilityComputer animation
14:26
Raw image formatEmailSoftware frameworkDressing (medical)Group actionFilm editingComputer animationLecture/Conference
15:17
Lecture/Conference
15:36
Software frameworkMathematicsComputer virusForcing (mathematics)Lecture/Conference
16:05
Different (Kate Ryan album)outputCASE <Informatik>Software frameworkOrder (biology)Curve
16:44
MathematicsDynamical systemAuthenticationSquare numberCircleEmailBasis <Mathematik>CASE <Informatik>Staff (military)WeightOrder of magnitudeQuicksortSimulationComplete metric spaceMultiplication signOverhead (computing)Software framework
17:39
Software frameworkEndliche ModelltheorieAngleMathematicsFlash memoryOverhead (computing)Function (mathematics)Lecture/Conference
19:25
Computer virusDifferent (Kate Ryan album)Condition numberLecture/Conference
19:41
HoaxBitOrder (biology)Context awarenessSign (mathematics)WordEndliche ModelltheorieTheoryInformationDifferent (Kate Ryan album)Wage labourLimit (category theory)Rule of inferenceLevel (video gaming)Structural loadLogic
Transcript: English(auto-generated)
00:05
Hello, thanks all of you for being here. First, I will start with a little introduction about myself. I'm currently the Redeveloper at OnTrack. I've been programming for over 20 years.
00:21
15 of those professionally and the last 5 plus of those using mainly Python. And just for the context of what I'm going to tell you, what's OnTrack. We are a startup, less than 2 years old, set in Madrid.
00:43
And we are trying to put tech into the logistics industry. Which is a very funny industry because it has both very technological stuff, like people doing self-driving trucks or fully automated robot sorting facilities and stuff like that.
01:06
But then on the day-to-day operations, it still works, mostly as it worked 30, 50 years ago. So somebody calls and asks for quotes from drivers and so on. So that's a bit the problem we are trying to solve.
01:22
So something like a year and a half ago, we started building the OnTrack platform. And our CTO proposed us to use Django Rest Framework. Since he had used it before with success, we analyzed it and we decided to go with it.
01:41
It's a decision we have come to both rejoice and regret over the last year and a half. So lots of good things and bad things from there. So I will just try to show you some interesting stuff that happened to us during that time. So first, a very simple, naive assumption we did. So we started and we had our main core application.
02:04
And we had the views, the serializers and the models. And we started very small, very lean. And we said, okay, we will split them when we reach that point. Of course, that point came a little late. So this was all quite big.
02:23
So more naive assumptions, one does. So we said, yeah, you know the pagination from the Django Rest Framework? It doesn't suit us. We would like to have this option of setting a page size of zero and getting everything when we use that as a page size.
02:45
So we went ahead and did it. Let me tell you people a very bad idea. There is a very good reason the Django Rest Framework doesn't usually let you do that. And it's because when you are getting some thousands of heavy models,
03:04
the serialization goes completely crazy in heaviness and time. So one of the things you hit when you do that is the famous M plus one query problem. Main way of solving that using prefetching and so on.
03:24
So yeah, some of our prefetchings look like this. Yes. And one of the things I want to mention, the prefetching into an attribute that is a very, very useful thing to do.
03:44
So basically as you see some of them, we put it to a private attribute in the model and then we do some magic in the models. So we always have to fill that. So thanks to stuff like this and using SICK,
04:04
we managed to solve for M plus one queries problem, or more like 50 M plus one queries problem. And we got some of those endpoints from over three minutes to just some seconds, which was very nice.
04:20
Then next thing that happens is you have your very nice REST API and you are very conscious about it. And then of course the mobile applications come and tell you, yeah, you know what, fuck auto-discoverability. I want to reduce the number of requests I have to do. I don't want to have requests depending on other requests.
04:43
So you start doing some different endpoints for the mobile apps. And then you might run into something like this. And you end up, since the mobile endpoint is, let's say, a subset of the standard endpoint,
05:00
you might think of extending the mobile endpoint into the standard endpoint. Believe me, that's the wrong way of doing that and you will regret it later. So please, when you run into something like this, don't do this. It's better to reduce your standard endpoint and have it the other way around.
05:23
And talking about in-editing, let's mention a little thing. So when you are trying to inherit the fields and the extra keywords in the serializers, remember that you need to copy the dictionaries because it's very easy to forget
05:41
and it can be a pain in the ass when you start adding random arguments to your original model. Then another thing that happens is when mobile tells you, yeah, what about giving me different data for the list
06:02
than for the detailed view? Okay, you could do two different views, but it's only just having a subset. So let's go ahead and just change it dynamically, what serializer we use. You end up with something like this. This self-detection, it's very undocumented in the Django REST framework,
06:24
but it's very helpful for this kind of things. So basically, when you have your view, you can use, that action point is auto-populated in it, so you can easily use it, but it's very nice.
06:41
And going with small things that can go wrong, which happens everywhere, there is this very funny point in the documentation of Django REST framework. I don't know how many of you are native English speakers, but for us it was a very funny moment when we realized that what it says there
07:03
is not that the lookup URL work is defaulting to PK as the lookup field, but it actually defaults to whatever value you put in the lookup field. That was some curious phrasing that drove us crazy for a while.
07:41
Then of course, once you've dealt with separating your endpoints and doing all that crazy stuff, then you have to deal with strange requirements that come from the business side of things. So for example, for us one of the craziest things we had to handle
08:04
was that they decided at the beginning that they didn't want users and that kind of stuff. So basically they wanted to give a link to the people and make sure that whenever somebody would go through that link they would be logged in and be a proper user and whatnot.
08:23
So we ended up creating this crazy authentication method that was just based on getting some parameter on the header and then would actually even create a user if the user was not already there because nobody was creating those users.
08:42
They were just simply giving some random URLs to the clients. Believe me, this kind of thing tried to avoid it also. Then you might end up with strange validations
09:02
like let's make sure we do not change the past. Okay, seems reasonable. Of course, seems reasonable. And then a week later business comes and tells you hey, by the way, we need to actually be able to change the past in this, that and these other circumstances.
09:21
So you end up having to build your own way of going over your validation. And then you might end up having strange ideas like okay, this field, this thing is not so well defined.
09:41
Let's not structure it so much. So let's just go ahead and put it in a JSON field. And then we will just give it a structure using the serializer. And you end up with something like this. That's a lot of horrible calculations at the serializer level
10:06
just to get some data that fits in a JSON field. This, for example, has been not only a pain to maintain but especially it was quite painful to then undo
10:21
because that was some quite valid data that we had there and we ended up having to put in a proper table with proper structure and so on. And yeah.
10:40
So that's a bit what some of the funny and interesting stuff we've gone through over this year. I also would like to mention some of the tools we found helpful during this year of learning and finding out the stuff. So for example, we have this Don't Repeat Yourself permissions framework.
11:05
That works quite well. It has a strange thing that is that you define the permissions for your views in your models instead of in the view itself, which feels weird. And at least I feel that I end up repeating myself quite a bit, but it works quite well.
11:25
Django push notifications is a very nice and properly maintained library for handling push notifications. It integrates quite well with Django rest framework.
11:41
Then the Django finiter state matching, also nice and useful to have. Gives you a very nice out-of-the-box finiter state matching and even keeps track of stuff like the concurrent requests and stuff like that. Then probably the library I've ever used with the longest name,
12:04
that is the Django rest framework Sophia Loauth2. That gives you basically everything you might ever need related with authentication and user creation and all that stuff.
12:21
And SICK, that I mentioned before, it's a really, really good profiler. It's very nice to use. It's very helpful. It's quite deep and it's quite pleasurable to use. So basically what I would mostly like you to get out of this presentation
12:43
is that you need to use your framework. So a lot of things, a lot of times I've seen a lot of people that take a framework and try to use it however they want to use it. You need to learn your framework, you need to use the different parts
13:03
in the way they are supposed to be used in general because if you are fighting your framework then it will get on the way and then it's not worth the thought of using a framework. So basically try to find one that adapts to your circumstances
13:21
and learn to use it properly for what it does well instead of trying to fight it over. Because there is a good reason why frameworks are out there. They give you a lot of nice things, a lot of speed, a lot of ease of use.
13:43
In particular, my experience with Jangora's framework, we've talked about getting rid of it quite sometimes and for the time being the advantages have always surpassed the problems it gives us. And of course it has limitations, it has parts that could work better.
14:05
For example, as I said before, when you are serializing big models it goes into quite heavy consumption of resources but in general it's worth the effort.
14:21
So yeah, basically try to use hammers for your nails and screwdrivers for your screws otherwise you would be screwed basically. So yeah, that's wrapping up. That was me one year ago.
14:40
This is me after a year with Jangora's framework. And yes, both are company pictures. So please feel free to come and visit us. We have a stand downstairs. We have Gummy Bears, we have challenges, we have prizes and we have some very nice pink dress people to talk with.
15:03
Thanks everyone for listening. Any questions you can ask now or you can also email me at castillo.com.
15:21
Thank you very much for sharing your experiences. Are there some questions? Any questions?
15:43
Hello. Thank you for your talk. Thanks. My question is, do you feel there are changes in how you use the framework and how you adapted to your circumstances today as opposed to a couple of years ago?
16:02
So first to clarify, a couple of years ago I hadn't really used the framework yet. So yeah, there's a lot of difference on how we are doing our stuff right now and how we were doing it a year and a half ago when we started. So there's a learning curve whenever you try to use a new framework.
16:27
And in this case I think it's been quite interesting the findings we've done. So for example, when I was talking before about the order of inheritance,
16:40
there's a very good reason. For example, this, we were in editing in the wrong direction, so a specific case, the mobile, was the base for the standard case, that was the serializer, I mean, sorry, the main Jornidle. And then when we changed this in the mobile one,
17:02
we were not counting with the other one in editing from this and it was completely absurd and something that everything broke, suddenly and so on. So yeah, you learn stuff and you learn from your mistakes. It's important to iterate and make mistakes,
17:20
otherwise you are not really learning so much. And yeah, it changes how you use stuff over time. Yeah, hi, thanks for your talk, it was quite informative. There's somewhat of an overhead of using Django REST framework
17:45
just for making the models and stuff like that. How do you feel if, would you use it for a small needs, for example, just needing a JSON output from a model for reading it or something like that?
18:00
I myself have just kind of written a small method for the Django model to give me the JSON itself. Okay, so if I understood correctly, basically you are asking about a smaller thing and if, yeah.
18:21
So probably for a smaller thing I wouldn't go with Django REST framework unless you already know it. I mean, the thing is if you already, I have been working before with quite a lot of plain Django and adding the layer of Django REST framework on top was not so hard, so that's why we decided also to go with it.
18:44
But probably for a small thing it's not worth the thought of learning all these layers, because even Django itself it's not a simple framework, so for something simple probably you are better off using Flask or something like that.
19:02
I think there are even some linear frameworks out there. So yeah, probably for something small, unless you already are familiar with the Django and Django REST framework, I wouldn't go with it.
19:25
Hello. Hello. I want to ask you about permissions. Did you use permissions on different methods of the endpoints? Yes. So what permissions? And how did you implement them?
19:41
So, we use quite some different ways of having permissions. As I said, we use that Don't Repeat Yourself permissions library, which basically hooks up your models to allow you to define their permissions.
20:02
So basically you do them at model level so you can play a lot with them. So for example we have for the journeys, as I said we work in some relationships between zippers and drivers and so on. And the journey is what a driver actually does,
20:23
so it's what loads he gets and so on, just to give a little bit of context. And for example there the permissions are quite complicated, because a journey that is not a sign can be seen by any driver, but if it's a sign then only its own driver can see it,
20:43
but then also the zippers involved in that journey can see that journey also. And of course all the super users can see the journeys. And even we have some rules about when it was a sign so that you can see the last journeys that were available and stuff like that.
21:07
All of that is very easy to handle with this DRI permissions framework. It allows you to create very simple rules in just pure Python logic.
21:22
I hope I answered your question. Any more questions? So if this is not the case, thank you very much again Alejandro. Thanks everyone.