Learnt lessons in a big Django Project
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 48 | |
| Number of Parts | 173 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/20228 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Bilbao, Euskadi, Spain |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
BlogRepetitionMaxima and minimaMetropolitan area networkFront and back endsHeat transferTranslation (relic)Point (geometry)Projective planeWeb applicationStrategy gameMereologyContext awarenessOpen sourceTwitterWeb-DesignerWeb 2.0Decision theoryProduct (business)Level (video gaming)MassCore dumpWebsiteSoftware developerPoint (geometry)Content (media)Stack (abstract data type)Multiplication signTranslation (relic)CodecData structureRoutingComputer architectureHeat transferState observerData managementFrequencyControl flowAdditionComplete metric spaceRootPhysical systemSearch engine (computing)Cartesian coordinate systemParameter (computer programming)Associative propertyTheoryInternet forumObservational studyInstance (computer science)Lecture/ConferenceJSONXMLUML
04:05
INTEGRALMobile WebRevision controlMobile appCartesian coordinate systemFlow separationLevel (video gaming)Lecture/Conference
04:32
Flow separationClient (computing)Different (Kate Ryan album)Module (mathematics)Product (business)Cycle (graph theory)Video gameProjective planeLevel (video gaming)Instance (computer science)Cartesian coordinate system
05:15
Module (mathematics)Turtle graphicsINTEGRALMultiplication signAuthorizationMathematicsVideo gameLevel (video gaming)Internet service providerLecture/Conference
05:53
AbstractionMeasurementCodeStudent's t-testMultiplication signFunctional (mathematics)MereologyInstance (computer science)Physical lawRule of inferenceThermodynamicsString (computer science)JSONComputer animation
06:48
Archaeological field surveyThermodynamicsPlastikkartePhysical lawSoftware bugLecture/Conference
07:08
Software bugArchaeological field surveySoftware testingKnotThermodynamicsWeb 2.0Execution unitLibrary (computing)CodeGame controllerSoftware testingProjective planeMathematicsControl flowUtility softwarePoint (geometry)Cartesian coordinate systemMereologyRevision controlWeb-DesignerCondition numberState observerOpen sourceHypermediaSystem callWeb crawlerInstance (computer science)Query languageAreaData managementSocial classVideo gameAngleCASE <Informatik>SummierbarkeitSource codeComputer animation
10:51
CodeProcedural programmingSoftware testingSystem callNumberMultiplication signOverhead (computing)Branch (computer science)Software bugSoftware developerLecture/ConferenceJSONXMLUML
12:23
Metropolitan area networkInternetworkingRepository (publishing)Dependent and independent variablesDecision theorySoftware developerInstance (computer science)Sign (mathematics)Lecture/Conference
13:14
CAN busOpen setAbstractionSoftware testingFlow separationPort scannerLibrary (computing)Physical systemDesign by contractFunctional (mathematics)Open setOpen sourceMereologyMultiplication signLevel (video gaming)Software testingRepository (publishing)Product (business)Projective planeFocus (optics)Revision controlRight angleSource codeVideo gameCodeJSONXML
16:29
Dimensional analysisSoftware testingStaff (military)Parameter (computer programming)Information securityFlow separationImplementationDesign by contractVideo gameForm (programming)ResultantMereologySign (mathematics)Open sourceCASE <Informatik>Multiplication signDifferent (Kate Ryan album)Computer networkDecision theoryPhysical systemFunctional (mathematics)Software frameworkProjective planeComputer architectureBitTelecommunicationRight angleDegree (graph theory)CodeProof theoryCodeFront and back endsDemo (music)Lecture/Conference
Transcript: English(auto-generated)
00:01
Hello to everyone and thanks for being here in my talk about 10 lessons that I learned during a big Django project. Introduce myself, I'm Yamila Moreno. These are my Twitter and my GitHub account. Two years ago, I founded with another Pythonist I've seen in Spain, the Spanish Python Association,
00:22
and we organized the first PyConnes in Madrid and we also, I also co-founded PyLadies Spain. I'm a software developer in Caledos. It's a company that I co-founded four years ago and we work only, mostly, 99% with open source
00:44
and it's important and we'll see later. In Caledos, we work with, we make web applications, big web applications usually for, with Django as well as other technologies as Python and Django are part of our stack.
01:03
So, this talk is about the good and the bad decisions that we made during a big Django project. It's especially, the project was the complete renovation of our roadmap, of our website in Spain, quite famous, and it's meant for all of you who are starting
01:22
with web development or maybe facing for the first time a big project. The website, this project is the Guiarepsol. In Spain, it's quite famous. Repsol is a massive oil company and the Guiarepsol is part of the marketing strategy.
01:42
It's a roadmap with, you can search for routes and it also has many advice of tourism, gastronomy, and accommodation. So, it's a baby. With, in this project, we needed to change completely the brand and give a new style,
02:02
but also we needed to keep the original essence and the creative team focused on five main features which were tourist and gastronomic cards, search engine, a powerful map, a planner to organize the trips, and looked after the digital content. I said it was big, how big?
02:20
So, we had 80,000 touristic cards, more than 500 articles and growing. This amount of data maybe is not so impressive with all the data, but it was quite difficult to show all of them in a map at a time and very quickly with filters and everything. The first release took about 18 months
02:41
and we spent another six months with a transfer to a new team and adding new features. In the present time, the project is being evolved. The original team were two UX, one designer, one front-end, six back-ends, the product owner, and as well as the core team,
03:01
we had the content managers, the translation team, the IT team from Repsol, the Academia de Gastronomia, because we had lots of people giving their opinion to build the team, the project, sorry. We work within the agile methodology, so we use these small iterations, which are sprints, two-week sprints.
03:21
We made 42 sprints, 600 user stories, and more than 1,300 points. So, this was the project. This is what I mean with a big project. With this context in mind, let's talk about the decisions. One of the first things that you have to face when you are making a project, no matter if it's big or it's small,
03:42
is the architecture or the structure. If the project is small, you can keep on the proposed NBC of Django with the back and the front coupled, but a bigger project will face bigger and harder challenges, and you need stronger foundations. In the Repsol, for instance,
04:00
we started developing this website, and six months later, we were asked to expose an API for third-party applications, for mobile applications, a mobile version of the map, and several other integrations, and we weren't prepared. We really, we had some API modules, but it wasn't really an API from one side,
04:25
and we struggled to update to the new requirements. So, the lesson that we learned the hard way was completely separate API and front projects, because API is usually stable or maybe more stable, and the front projects change every day,
04:40
and you can add lots of new clients or new applications consuming the API, so different life cycles completely separate. As well as being prepared on the big picture, it's also necessary to be prepared in smaller modules. For instance, we knew we had to make a roadmap,
05:01
but we didn't know the map provider, because the customer hadn't made its mind. We thought it was Google Maps, but it wasn't sure. So, we prepared an isolated module which was talking to our inner code, and this module was the only in charge
05:20
to talk to the uncertain map provider. Finally, we integrate with Google Maps, and we didn't change, but Google Maps changed the API two months later after the integration, because they can do this, and we only needed to change in this isolated module
05:41
to be, like, with very few effort, we were prepared to update the requirements of Google API. So, be ready for changes, but don't over-engineer. It's difficult to know when are being prepared and where are we engineering, and some rules may be useful,
06:02
and the rule that is the lesson three is refactor an abstract the second time you have the chance. That means that if you are working within a method, for instance, and you have to parse a string, you can parse the string within the method, but the second time that another part of the code you need to parse a string, remember that code that you did,
06:22
and abstract this, there's this new functionality in a utils, isolated, and called the new function, the new method for everywhere in the code. That way, you will refactor the second time, and not before you need it.
06:41
Okay, so, before continuing, let's carry out a very little survey. Please raise your hands, those of you who have heard about the laws of thermodynamics. Okay, smart people, of course you are right in my talk. Now, raise your hand if you know by heart the fifth law of thermodynamics.
07:02
Yeah, I was expecting that. It's pretty common to forget it, and it's a very important law which says sooner rather than later you will make a bug. It's not me, it's thermodynamics, it must be true, but being that true, the next lesson is quite easy, because you have to test, test, test. If you have a big application, a big project,
07:22
you need to intensively test the code, because you will need to be sure that your new code won't break the behavior of the application, and you will need to have your colleagues to add new features without breaking your code,
07:42
your precious code. So you need to test whenever you are adding code, and in the other tool, for instance, even if all of us knew that you have to test, we wrote tests, but not many, and we struggled, because some parts of the code made us fear, and we didn't want to touch it.
08:01
Only the person who originally wrote that code was in charge to change it, because we didn't trust ourselves, which is normal, and in Python and Django you have lots of utils and libraries and tools to test. PyTest, the unit test, and other libraries to mock, stub, and spike. You need to know about testing,
08:20
and really, really test your application. In Django, when you are developing with Django, and this is specifically for Django and Python, you need to know about the strongest point, which is the community. If you go to GitHub, you are going to find thousands of projects with Python or Django tags,
08:40
so when you need something for your application, you have to think that maybe, probably, someone has solved before you, and probably that person has done tests and documentation and has done better than you are going to do, and you don't need to put effort that.
09:01
That community helps us to meet one important point in the web development, which is don't reinvent the wheel, but sometimes the library that you find doesn't exactly do what you want, or maybe it's too big and it's a very big dependency to add into your project, and you just need a very little thing, or maybe you need the whole control of what's happening,
09:24
so in that case, the advice that I can give to you is that you have to know the internals, and don't be afraid to do it yourself. In Python and Django, it's open source, so you can hack almost everything,
09:42
and you can add new features, and you can improve and extend as you want, so know Python, know Django, and don't be afraid to do it yourself. As well as these big headers, there are other good practices,
10:00
maybe they are not so technical, but they are also very, very important. For instance, if you have been working for a day in web development, you will know this lesson, but nevertheless, it's important to say it, so use a distributed source control manager.
10:20
You have Git and Mercurial. Mercurial is made with Python, very cool, you can hack it. Subversion is not distributed, Subversion is from the past, don't use that, Subversion, Git, or Mercurial, or other, I really don't know others, but maybe. So with these kind of tools, you will ensure your code to be readable,
10:44
you can go through the history of the changes, you can undo changes if you don't want to, and when many people are committing into the same code base, you need to use something like this. In Git Hubs All, we knew this, but we introduced a new procedure,
11:04
which is like the lesson eight, use pull request as the basic procedure to have features to master. That means that when we were starting a new feature, we created a new branch from master,
11:22
and then we developed the code, we developed the test, and the documentation, and when we were comfortable with the result, we made a pull request, and another colleague was in charge of reviewing this code, and testing it, and maybe suggesting some improvements.
11:41
That way, all the code which was entering in master was reviewed by two people, the original developer and the reviewer, and at first we thought that it would make like time overhead, we were losing our time, and it was all the contrary, we were making less bugs from the very beginning, it was a very, very good new procedure
12:03
to add into our team, and it's very recommended. One often overlooked issue is documentation. Maybe we think that we code so, so well that we don't need to document our code. Well, no, it's not that.
12:20
You need to document your code as much as you can, and do it okay, do it well. In GRF Sol, for instance, we had two documentations, one for the internal team, concerning the reasons of the some decisions of design, concerning the versions, the deployment,
12:41
concerning the code, and we had a different repository for the API documentation, which was meant for those developers who were consuming the API, and they only, for them, only responses and status matter, and everything else they don't mind to. So we made those two documentations,
13:02
and for the new features concerning the API, we made also pull requests, and we used this kind of documentation-driven development, which meant that for any new feature concerning the API, we made first a pull request into their API documentation repository,
13:21
and the colleague in charge to review it was someone of the front team, because they were the first people using the API. So if they, when we agree with this API documentation, the back team could start developing, and it's like the design by contract,
13:41
but focus on a higher level of functionality. For us, it really worked very, very good. Finally, and it's very important, when you are working with Django, with Python, or maybe with other open source tools, you are using, you are taking advantage of the open source ecosystem.
14:00
So those tons of libraries that make your code better and your life happier are part of our ecosystem, and it's important that you should give something back to the system, so when you are developing, make as much open source as you can. There are many, many ways to contribute.
14:20
It's not necessary to be a top-notch engineer. You can report a bug, you can fix a bug, maybe create a tutorial, you can create a workshop, a fantastic workshop of Django Girls, and you can, of course, release a library if you have done this. In the other episode, we have, sorry, in Kaleidos,
14:43
we have contracts with our customers, and in the contract, they agreed to release non-critical code, so whenever we are working and we need to, we discover some part that would be useful for others, we can make that to be a library, release the library,
15:01
and then we use this code as an external library, but it's in GitHub, and it's available for anyone who wants to. And also in Kaleidos, we have developed our own product, so we are on our customers, and we have released it from the very beginning. It's called Taiga.io, and it's made with Django
15:24
and with Angular, and we are very happy and very proud to share it with the community. But if you want to share it, if you want to give into the wild, and if you want the community to contribute with you, you need to follow those steps and maybe other steps
15:41
to make it easy and clear how to contribute. You need to make tests, documentation, and you need to, like a beautiful API, you have to put some effort in making things good, not only for performance, but also to share with the community. These are the 10 lessons that I can give to you.
16:02
I learned during two-year project. Most of them apply to any big project, but some of them apply specifically for Django because you need to know which are the good maintainers, which are the obsolete libraries, and it's important to know.
16:21
So I think that that's it, and do you have any questions? Any questions?
16:48
So you mentioned that you agreed the documentation for the API upfront before you started coding. Did you then write the tests against that agreed API before you then started coding up the API,
17:01
or was that done in parallel, or how did that work? Because it would be really nice to be able to agree the API, and then you know what the test should be, I suppose. So you are asking if we wrote tests on the documentation? Yeah, so once you've agreed in the documentation
17:20
what the API's going to be, did you then write tests that then obviously failed because you haven't written that thing yet, and then write the actual code? About documentation, we have tried several things, but in the end what we made is just writing the documentation by hand, because we didn't use any specific tool.
17:42
The tests were about the implementation, so when we started implementing, we made the test, maybe TDD for that specific part, but synchronizing the API documentation with the implementation is a matter of the human effort.
18:01
Yeah, sure, okay, thanks. A question I have myself. You said that your customer agreed for you to open source parts of the project. Was that something they were happy to do, or did they need convincing? If they are happy? Yeah, were they happy when you suggested open sourcing?
18:24
They are happy, because at first time, maybe they didn't know us, but when we are signing the contract, they know us, and we have convinced them that it's very cool to be part of that, and of course it's mandatory for the first thing, so it's not a decision that we share with the customer,
18:43
but they are not forced. They feel comfortable with the decision, and they feel very happy, because we have shown them that I am not implementing my own API. I am using Django REST framework, so if they don't want to release, I will write my own API, which is crazy,
19:04
so they are happy, and sometimes they go saying with other colleagues like, oh, we are in this open source technology and so on, so they are quite happy. Any other questions?
19:26
Thanks for that. So you said you're the people behind Taiga IO? Yeah. Just a question I haven't checked in a while, but last time I seen it, there was no live demo. Do you have live demo already? Live demo of Taiga?
19:40
Yeah. We have Taiga in Taiga IO. Taiga.io, you can sign up, and you can play with it, and you can start it on premise, so be welcome. Thanks. And if you want a specific demo, we can talk later.
20:03
I think there was something in that part. You mentioned separation of front-end API, so how do you separate it, and at the same time,
20:23
how do you make sure that you are actually using that API, or do you writing a different access methods and different code paths for your front-end? In our case, the front team and the back team,
20:40
we are together, so we agree, even the architecture from the very beginning, but nevertheless, in the project, we have those mockups, which are the end of the sign, and the customer agrees the functionality, so as long as we are exposing the data, and the front is consuming the data,
21:02
and the customer is agreeing with the result, we are like, it's the only proof that we can take is the approval of the customer. I think, is that what you're asking? Okay.
21:20
Any more questions? Nope, okay, thanks very much, Yamuna.