Fast product development using Django Rest Framework. #lessonslearned
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33796 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Product (business)Software frameworkIntelComa BerenicesSoftwareRaw image formatPoint cloudBitGoogle App EngineProduct (business)Exception handlingSoftware developerSoftware frameworkStack (abstract data type)Type theoryWeb 2.0BuildingComputing platformScripting languageMultiplication signClassical physicsParticle systemGradient descentSystem callUltraviolet photoelectron spectroscopyBoss CorporationVideo gameCASE <Informatik>Lecture/Conference
01:22
Execution unitCharge carrierProduct (business)Raw image formatPrototypeScalabilitySoftware frameworkIterationProduct (business)TrailDevice driverPrototypeProcess (computing)Software frameworkPoint (geometry)CodeStatement (computer science)Goodness of fitProgrammer (hardware)Stack (abstract data type)Matching (graph theory)Logistic distributionType theoryBitMultiplication signNumberSoftware testingAxiom of choiceAuthorizationLibrary (computing)WeightFigurate numberCASE <Informatik>Term (mathematics)Fitness functionFamilyDescriptive statisticsSampling (statistics)Software developerIRIS-TScaling (geometry)Perspective (visual)Video gamePower (physics)Formal language
05:55
Video gameFocus (optics)Web pageIntegrated development environmentContext awareness10 (number)Physical systemCartesian coordinate systemFront and back endsSystem administratorOperator (mathematics)AuthorizationLoginDimensional analysis1 (number)Type theoryField (computer science)Point (geometry)Software frameworkScaling (geometry)Web 2.0Element (mathematics)MassMultiplication signCASE <Informatik>WordSierpinski triangleLink (knot theory)Order (biology)WebsiteSelf-organizationComputer architectureDigital electronicsInformation privacyProduct (business)Uniqueness quantificationUniform resource locatorPresentation of a groupKeyboard shortcutWeb pageInterface (computing)HierarchyLecture/ConferenceComputer animation
09:41
Product (business)Serial portStrutPhysical lawDecimalField (computer science)Grass (card game)MassOffice suiteQuicksortProduct (business)Physical systemPasswordKeyboard shortcutNumberMultiplication signLogic gateReal numberProcess (computing)Virtual machineInstance (computer science)BitField (computer science)System administratorTrailComputer animation
11:19
Moving averageView (database)Device driverQuery languageVector potentialHuman migrationProduct (business)Computer programmingGame controllerCuboidBlock (periodic table)Condition numberObject (grammar)Level (video gaming)Complex (psychology)BitSoftware framework
12:08
View (database)Query languageDevice driverProduct (business)Attribute grammarFilter <Stochastik>Object (grammar)Software frameworkDevice driverInstance (computer science)Moment (mathematics)PlotterCASE <Informatik>Computer animation
12:50
View (database)Device driverQuery languageInterior (topology)Scale (map)Game controllerProduct (business)MultiplicationSerial portLevel (video gaming)Latent heatPattern languageSet (mathematics)Multiplication signCASE <Informatik>Execution unitPoint (geometry)
13:15
Object-relational mappingElectronic meeting systemNumberDecision theorySoftware frameworkGoodness of fitWave packetCodeWeb pageBitQuery languageFigurate numberPoint (geometry)VarianceCategory of beingFunctional (mathematics)State of matterCondition numberComputer animation
14:58
Finite-state machineTrailTwin primeElectric generatorFile formatElectric currentDefault (computer science)SummierbarkeitCore dumpGroup actionState of matterFinite-state machineLibrary (computing)MathematicsBitDifferent (Kate Ryan album)Level (video gaming)System administratorMultiplication signEuler anglesInterface (computing)Closed setAbstractionPattern languageBuildingComputer architectureInstance (computer science)Normal (geometry)Sheaf (mathematics)Software frameworkCondition numberFunctional (mathematics)Subject indexingMathematical analysisDecision theoryType theoryComputer animation
16:57
WindowCore dumpMenu (computing)Process (computing)Open sourceCartesian coordinate systemCoefficient of determinationCNNBuildingDatabase transactionSoftware developerCASE <Informatik>DampingDifferent (Kate Ryan album)Line (geometry)Parameter (computer programming)Template (C++)TelecommunicationPhase transitionFront and back endsPlug-in (computing)Open setLimit (category theory)Variable (mathematics)Order (biology)
18:38
Focus (optics)Product (business)Process (computing)MeasurementBuildingControl flowDecision theoryPhase transitionTime zoneTrailPoint (geometry)Process (computing)Order (biology)System callService (economics)MetreDifferent (Kate Ryan album)Software maintenanceDemosceneDesign by contract
19:51
Software frameworkEvent horizonNormal (geometry)Shared memoryEmail
20:26
Connected spaceLatent heatClient (computing)Home pageBitImplementationComputing platformOpen setWebsiteInteractive televisionSoftware frameworkPoint (geometry)Multiplication signInstance (computer science)Cartesian coordinate systemINTEGRALServer (computing)CountingMereologyMobile appWeb pageFront and back endsTemplate (C++)VolumenvisualisierungProjective planeComplete metric spaceElectric generatorFlash memorySingle-precision floating-point formatProbability density functionMeeting/Interview
23:06
Software testingModule (mathematics)Software frameworkMultiplication signSet (mathematics)Point (geometry)Unit testingExecution unitPerspective (visual)Mixed realityOffice suiteProduct (business)Presentation of a groupObservational study
24:08
Core dumpRow (database)WindowSoftware frameworkObject (grammar)MereologyGroup actionFinite-state machineInstance (computer science)CASE <Informatik>Meeting/Interview
24:32
Set (mathematics)Different (Kate Ryan album)Field (computer science)Serial portInternet service providerKeyboard shortcutView (database)Similarity (geometry)Software frameworkElectronic mailing listCore dumpBeat (acoustics)Frame problemMeeting/Interview
Transcript: English(auto-generated)
00:05
Hello everybody, just for me to adapt a bit what I say to who is attending the talk. How many of you do, or you consider you do fast product development in your day-to-day work? Okay, not that many.
00:21
The rest is probably in research, hopefully. And how many of you have you been using Django REST Framework before? Okay, perfect. Then a bit about me and how I end up using this stack. I started programming some 15 years ago, building web platforms with Bash scripts and CGI. Luckily, around 2009, I entered a startup called F24,
00:45
where they were using the Python stack. It was mostly soap at the time, and Django started their life around that time. I've been involved in a couple of startups, mostly from that time. And in many cases, I've been using the Django stack.
01:04
I say the Django stack because Python has many flavors, and mostly for me, Django performed well in the past. The only exception I think I've been having is when I was doing valve fleet and tried out the Google App Engine type of approach.
01:23
Now what is OnTrack doing? I will give a very brief discussion, just new thing to follow the small code examples we will be discussing afterwards. So OnTrack is a B2B marketplace. We just basically match pallets or industrial flight with available track drivers. Normally, track drivers have either two, three tracks
01:43
or a very small company. It's hard for them to find jobs. For the customer, it's hard to find the available track driver. They need to call many people. They use email, they use faxes back and forth, and we just simplify the standard marketplace ways. What do we mean by fast product development?
02:01
Usually, when you are building a startup, and this might be completely different if you are in research or if you are in a large company, but when you are in a startup, you normally tend to be in one of two phases. Either you are looking for product market fit, and then you need to iterate very fast. And really, nothing matters except finding product market fit,
02:24
because no matter how good your product is, no matter how automated everything is, no matter how well you dockerize everything, 100% test coverage, really nice code disruptions, if you don't find product market fit, it will all end up in the bin. Now, one day, you are lucky, and if you are lucky,
02:43
and something makes click, and then it's the day when you start scaling and you start regretting all the choices you did to get there. In our case, this is a bit the revenue growth from OnTrack, so this is six months trying to figure out what the hell we are doing in this logistics market
03:01
that we have no clue about. And then six months saying, okay, maybe we got lucky, maybe we figured something out. Now, how do we grow very, very fast? Luckily, we were, from the beginning, using Python, using Django, using Django Rest Framework, which is a stack that I value a lot for this type of problem, even because it lets you perform well in both stages.
03:25
You can go very fast in the prototype phase, and we will figure out, and I will show afterwards a couple of examples of how extreme we want to get even a bit faster, to put our product out there even a couple of weeks earlier.
03:40
And once you get to the point where you satisfactorily figure something out about your market, the tools are there for you to scale. And it's not only scaling from the technology perspective, it's also scaling from the team size perspective. In our current team, there are like seven people who touch the backend, only three people
04:01
have previous Python experience, so it's very important for a stack, and it's something I really love about the Python work, that you can easily jump in, coming from other languages, coming from other frameworks. Things that I really like about the stack is that it's very well documented, you have an awesome community behind it,
04:20
it's very easy to read the code that your colleges wrote before you arrive, and especially it's very opinionated. I give a statement here that I think many people will not agree, especially everybody who is very expert on something, and I have seen many good Python programmers saying like,
04:41
wow, yeah, but Django, you know, there is a point that is really, really complicated, it's really missing, it makes too many opinions. I prefer to go with my Flask thing, I put a couple of libraries, I know them by heart, and I have so much more freedom. And this works very well when it's you or two, three other very experienced people with that particular stack. When you need to quickly, never underestimate
05:05
the amount of time you will lose in discussing and re-discussing everything. If your framework has a couple of opinions, even if you end up in 80% of the quality of all the possibilities you could get to, the fact that you do that three times faster, because you save a ton of research
05:22
or a ton of discussions, it's something that actually comes to be an advantage. And you will have time later on to go for very specialized solutions on things that you can discuss a bit further. So what did we do or which lessons we learned in iterating fast?
05:41
Lesson number one is regret nothing, no? Go for it and try to put something in front of your customer as soon as possible. The founder of LinkedIn used to say, if you are not the same of your product when you launch it, you launch it too late. And this is something that I agree in this particular environment. Like I said, it might not apply to other contexts, no?
06:00
But it definitely applies when you are trying to build a startup very fast. This is the architecture principle that we tried to follow. We didn't use Django, and this is something that for me has been a lesson from past companies. If you use Django, the MVC approach to the max, you will very quickly be having too many points of entry into your backend, and it will get messy.
06:23
So one thing we did was to focus on being API first and just allow for API entry to the backend. And then delegate the frontend, delegate the presentation layers to the more appropriate tools. Also, that gives an advantage because the JavaScript wall is not stable, unfortunately,
06:40
not like Python or Django. So it will change for us. It was React now. We actually started with Angular, and it cost us a couple of weeks to move to React. It will hopefully cost us not so much time when the new JavaScript framework comes out, probably in autumn or whatever. A bonus point for having API first is one day your CEO decides they got a customer,
07:02
and the customer doesn't want to enter your beautiful website. They want that you integrate with the SAP system they have where they spend tens of millions for having it, and he doesn't want to train his people, no? And fortunately, and this is something that Django REST Framework gives you almost for free, is that you can have professional API
07:22
ready for that situation. If you are not using Django admin, which is one of the main advantages of Django, how can you get an admin page when you have zero orders? Like, how can you invest the least amount of money in it? Well, fortunately, Django REST Framework lets you manipulate your data,
07:42
and this is intended not as an admin page, of course. It's a disaster for that, but it's very cheap because nothing. This has two advantages. One is zero time until you have an admin interface because that's the API Explorer, and another thing is you align the vocabulary with the people operating the system because they really know and understand
08:02
how is the system behind because they actually need to use the same fields. This doesn't scale, of course, and it's a disaster past a couple of weeks, but you can launch two, three weeks earlier. If you need real-time notifications, also, there are many ways that are faster
08:21
and better than WebSockets. In our case, we quickly integrated with Slack so people would know, okay, something arrived at the system, assume then we put there a link to the API wrapper, and they could very quickly operate with almost zero time invested in our case, so we had more time to try to figure out what our customers want instead of developing internal back-office applications.
08:43
Now, to do some code, this is a problem that you'd normally have when you have a very clever UX product guys that they come and say the basic auth system is very boring, and we don't know how our customers want to use it. You might sell your application to one guy,
09:00
and then there might be another operator using it, or they might sell the login details, and how are you gonna map all this hierarchy of the people using your system? So one shortcut that we took was to say, okay, let's just figure out a login way where we just give people unique URLs.
09:22
Again, this doesn't scale very well. There might be some privacy concerns there, but this saves you a couple of weeks, and it lets you understand how your customer, how your customer interacts with your application, and then by way of dialogue, you would discuss with him how he's sharing login details with other colleges, and so on.
09:40
This saves a lot of time for them in putting the password in a post-it and sharing it around in their office. Another shortcut that we found useful when modeling our system was to start modeling complicated things with the JSON fields in Postgres. So we were having this problem.
10:00
We had shipments, and then our business guys said, we need to add some extras. Okay, so what's an extra? So for instance, if the thing needs to be picked up after 10 p.m., then we should charge a bit more money. Okay, then we put a rule, and then we put, okay, what does it mean more money? 10%, okay, 10% on top, fantastic.
10:22
And then if the track stays for a long time, then we need to charge more money. And it's like, okay, how much more money? Well, it should be a fixed amount that we would decide later. Okay, who decides that and when? Oh, this is the admin guy. Okay, fantastic. And then you go like, yeah, and we need another extra, like what?
10:40
Yeah, some tracks need lift gate. Okay, what the hell is a lift gate? Yeah, it's this thing you put behind a track that lets you put easier the pallets up if you don't have like a specific machine to do that. Okay, fantastic, what's the price for that? Is this a percentage or is this an absolute number? Yeah, it's a bit more complicated because it affects the number of vehicle.
11:02
If you have many pallets, it should be included in the price, but otherwise, it depends on this, like, okay, fantastic, how the hell do we model that? And then you go to the engineering team and they are like, are these all the number of extras or will they add more? Like, if I know how this works, they will add more.
11:23
So we started with a simple hack, which is we use Postgres SQL JSON. We don't need to model a lot. And we take advantage of Django framework serializers, so we just connect it directly, taking it out of complex logism, potential migration problems.
11:42
Another problem we were facing very early with our modeling was that the permissions and the access control layer were not as easy as we figured out. So usually, and we were using don't repeat yourself framework, you discuss something like, okay, I want the user to access some models,
12:01
and from those models, I want the user to access some objects. That's kind of easy, it's easy to map. We had a bit more intense problems where people said, yeah, but some users should not see some attributes of the objects even if they can access the object. They're like, awesome. And then in some cases,
12:21
you should also not be able to see any kind of attribute. For instance, a truck driver should not see some attributes related with the pricing in a specific shipment. So we were messing around a lot, and we had some solutions. I'm not pretty happy with it, and if you have some experience on the topic, it would be really nice to discuss afterwards
12:41
all the approaches. So we ended up using don't repeat yourself for advanced permissions on the models. Then we used the filters of the angle-rest frameworks to add an extra level of permissions or control. And then we ended up, in some cases, with having dedicated endpoints or specific serialized. So we have this kind of pattern of multiple serializers
13:02
over one model that we were also seeing the other day on a similar chart. So that's for how to try to put something out there very fast. Now what happens when you enter, when you get successful and start growing? Which kind of lessons were we learning here?
13:23
Lesson number one, the newcomers to the stack will always get in trouble with the ORM. And again, there is not a simple solution. We were trying to train them, showing them some past EuroPython talks, and doing a lot of code review, but it's very hard to train somebody new in Django.
13:40
I mean, and documentation is, I wouldn't say lacking, but it's a bit hard to follow down, no? Because it's this N plus one queries where you end up putting the wrong property, and then your queries go from 20 to 8,000, and you cannot figure out where the problem is. Then you teach them how to use professions on, and then you end up figuring out that,
14:01
of course, prefetch works only if the query that uses it is very similar to the query that the prefetch does. Otherwise, you need to map it in more specific ways so that you can reuse it afterwards. Again, if you have some good experiences in training people that are new to Django
14:21
into this particular aspect of the problem, it's the only thing that we have found complicated and that gets people really, really complaining about the whole framework because of the magic or because of the decisions on there. I would be very happy to share experiences or even figure out ways of documenting this better.
14:40
So there is a webpage where you say, okay, after some playing around with the framework and some experience, go start read this page where you have a lot of documentation talks, whatever, that show you how to avoid the typical traps.
15:00
Another lesson that we learned early on is if you have state transitions, use a finite state machine. There are very nice Python libraries, very easy to use, and it will help you map your abstractions better. While doing it, we found an interesting pattern
15:22
coming from our mobile guys. We said if we have status changes and mostly mobile, we want to go forward to advance. Why don't we, instead of meeting the state you should go to to avoid race condition, I tell you the state is a bit connected with what we heard before about declarative attitude.
15:40
I will tell you in which state I am and I will tell you that I want to go forward. And then we use the state machine in that way for the mobile phone, which is, they just tell us in which state they think they are and they want to go forward. We check if they are in that state, fantastic. We give them back the new state. If they were not in that state, we tell them, sorry, somebody with higher privileges
16:02
moved this state before. This is the new state, but there was something wrong so you can alert your customer that it was not a normal transaction, a transition, sorry. This allows you to control the states in a different way, for instance, from an admin interface where you might need to go back and forth and from a mobile phone.
16:24
And now on the closing part, a couple of thoughts, not so much related with Django Rest Framework, but a bit more higher level. One thing that I've seen that gets people to go very slow is to try to build everything themselves. This has been said many, many times. But as soon as you start growing something,
16:40
try to get it asynchronous. You have fantastic salary if you are in this stack or something else in others for building queues or other types of architecture. Another thing that I observe many times when I talk to other startups is they build their own CRM, like lead generation, lead tracking engines,
17:01
and they build their own billing because their case was very particular. Funny enough, billing is very complicated. All cases are particular, and you can rely on tools like Odoo, which is Python-based, open source, lots of plugins from the community to do billing in many different countries, many different situations, and you can just slightly touch it instead of building from scratch.
17:24
And another case where I found that we were doing templating, so kind of like front-end was for all these marketing applications. And in marketing, people come with like, oh, your transaction limit is okay, congratulations for your order. Fine, we will design one, we will put it out there,
17:40
we fill in the variables. And then they come and say, you know what, if it's the first order, we should put something different, like awesome. I will do an if, I will put something different. Yeah, and if it's a VIP customer, we should put something else, like fantastic, and you end up modifying something that there are tools that do it for yourself. In our case, we use CustomerIO, there are others,
18:02
and we just send the parameters of the order, and then the marketing team can do both templating there, and they can also do the segmentation. So for all these internal or pseudo front-ends that you end up building, like communication to your customer, you can also get it done outside.
18:20
One final thought is if you really want to go fast, and that was interesting because I had different thoughts in EuroPython in other talks, I don't believe that engineers or the developers should focus on the build phase alone and completely. I really disagree. I think this is a very waterfully approach.
18:41
And going and saying, yeah, we just like to build, and then somebody else deal with the customers. I think that this ends up in having the wrong solutions because you will just build what either you build before or what you have read online. And I think it's a much more valuable exercise when you get to be involved from early on in the decision process. Like if you can get involved in research
19:02
and understand why are we doing what we do, and get into a diverge phase where you're like, can we do something different? Can we do something crazy? Can we do something stupid? Can we do something that is maybe, something we should have never thought if we would have been in our own room and seen the tickets,
19:21
know that all the spec that come from somebody else. So for me, those were the main learnings I've been taking over the last year on track. And one final thought is like, if you feel that you are a bit out of the comfort zone
19:40
because everything moves too fast, you can never get it completely under control, that's probably the point. Where do you want to be? That's the sweet spot. And yeah, if you have been dealing with the Django Rest Framework permissions, if you have been trying to teach other people about the M plus one problem,
20:01
or if you just care about logistics, whatever, feel free to reach out to us. I would be super happy to have some discussion and talk, share some experiences. We have a stand downstairs, we will be the social event, and of course, all the normal online channels like LinkedIn, Twitter, email us. Definitely looking forward.
20:20
Thank you very much. Okay, thank you, Samuel. We have 10 minutes left, so whoever has a question has time to ask it. So who has a question? No one? Well.
20:49
Do you have a Swagger specification of your API? If you do, how do you generate it? Excuse me? Swagger specification, or open API specification
21:00
of your API, do you have it? No, we don't have it yet. Thank you. So the way we work is, the only integrations we do is with very, let's say, slow moving partners, like the typical partner that would do an implementation for you connected with a SAP, or like a very large company.
21:21
For them, they prefer to see the documentation via PDF and so on. And then what we do is we send a PDF because for them it's important so they know it's fixed in the point of time, and then we all commit to the API being that way. And then we spawn a new instance with a fixture data and we just give them this very same from them.
21:42
Okay. I was wondering, are you building only the API or you somehow have some connections to the client side rendering, and are you doing, because I was in the talk in the next room a bit earlier, and they are talking about client side rendering and speeding things up. So I was wondering, are you doing something similar?
22:03
Can you repeat? Didn't get the whole point. Do you render HTML and stuff on the backend or you just serve the API and that's all? So on the backend we just do an API. We are lucky in a way, compared with my past projects where we had to do a server side rendering,
22:21
so we took advantage of something that is usually not normal in the industry. We just do SEO for the homepage. So we don't do SEO for the main platform. And the homepage we have not built with Django and Django REST Framework. The homepage is a small flash application where we can do a lot of templates and that we do server side rendering completely.
22:43
And then as soon as the customer wants to play with anything, we move them to our single page app which is a React application and there it's all client side because we don't need to do SEO on all that interaction. Lucky, otherwise we will need.
23:08
Hi, thank you for the presentation. How do you manage your tests or how do you write your tests using Django REST Framework's class-based views? So we have a set of tests against the API.
23:23
So we use Pytest and we try to test everything from the point of an API consumer. Yeah, so you act like a, not like a unit test, but like a. More functional tests, I would say. Okay, thank you. Some of our tests are unit tests
23:40
but they are written normally from that perspective. So we have a mix of tests actually, but from there. So that's how we code it. So we don't have a specific test within some of the modules. Yes. That's what you ask. Thank you. So any more questions? Then I will ask one question if we have time
24:02
for somebody else to answer. How do you train people in the N plus one problem if you have been doing it? Or alternatively, how do you do permissions, advanced permissions with Django REST Framework? There were many Django REST Framework people. Like when you cannot just give an object but you need to give parts of an object
24:21
or when you have people that can do certain transitions of your finite state machine, for instance, but not others. In my case, we're doing something that's not REST. It's not really permission related but you can bind it to the permission. We have different set of fields for the list view
24:44
of the resources in REST Framework and different set for the details. So practically, I'm hacking the core of the serializer which is a bit ugly. But I can hack the fields and provide different set of fields. Maybe you can use something similar and bind it to the request and the user permissions.
25:02
Thank you. Any other questions or answers? Well, if not so, then thank you and thank you.