Standing on the Shoulders of Giants: The Kotti Web Application Framework
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 50 | |
| Number of Parts | 173 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/20076 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Bilbao, Euskadi, Spain |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 201550 / 173
5
6
7
9
21
27
30
32
36
37
41
43
44
45
47
51
52
54
55
58
63
66
67
68
69
72
74
75
77
79
82
89
92
93
96
97
98
99
101
104
108
111
112
119
121
122
123
131
134
137
138
139
150
160
165
167
173
00:00
ExistenceWeb 2.0Software frameworkQuicksortLecture/Conference
00:47
Software frameworkWorld Wide Web ConsortiumContent management systemInstallable File SystemObject (grammar)Information securityGroup actionAxiom of choiceFermat's Last TheoremComplex (psychology)Level (video gaming)CloningNetwork topologyObject (grammar)File systemMultiplication signPhysical systemClassical physicsCartesian coordinate systemCASE <Informatik>Content (media)CodeData managementGroup actionUtility softwareSoftware frameworkSystem callLevel (video gaming)DatabaseInformation securityEndliche ModelltheorieCuboidAdaptive behaviorContent management systemEnterprise architectureVotingSoftware developerClosed setSoftwareCore dumpStack (abstract data type)Computer fileAbsolute valueSoftware maintenanceData storage deviceComputer animation
03:49
AuthenticationAuthorizationSource codeCore dumpSoftware frameworkLevel (video gaming)Decision theoryForm (programming)Software frameworkAuthenticationDecision theoryAxiom of choiceMultiplication signFraction (mathematics)Perfect groupAuthorizationRepresentation (politics)Web applicationComputer animation
04:29
Uniform resource locatorForm (programming)AuthenticationSource codeAuthorizationAxiom of choiceAssociative propertyElectronic mailing listDatabase transactionDisintegrationObject-relational mappingAssociative propertyDatabaseArithmetic meanRelational databaseCategory of beingAuthenticationoutputAuthorizationSource codeObject (grammar)Template (C++)Cartesian coordinate systemTheory of relativityHybrid computerForm (programming)Database transactionFigurate numberVideo gameProxy serverValidity (statistics)Object-relational mappingData storage deviceAxiom of choiceXML
05:21
Social classElectronic mailing listPattern languageNetwork topologyRootCommunications protocolInheritance (object-oriented programming)Uniform resource locatorObject (grammar)Polymorphism (materials science)Query languageTable (information)HierarchyLocal ringAttribute grammarEvent horizonHuman migrationIntegrated development environmentComponent-based software engineeringData structureForm (programming)String (computer science)VolumenvisualisierungPhysical systemContent (media)Texture mappingWorld Wide Web ConsortiumEuler anglesData modelContext awarenessCodeRootObject (grammar)Communications protocolFlash memorySocial classOperator (mathematics)Multiplication signSequenceHuman migrationMereologyContent (media)MultiplicationPhysical systemData dictionaryBoolean algebraInstance (computer science)Complex numberAuthorizationUser interfaceSequelDisk read-and-write headMassScripting languageLevel (video gaming)Integrated development environmentINTEGRALTable (information)Formal languageDefault (computer science)Bootstrap aggregatingDatabaseSystem callVapor barrierCuboidEvent horizonData structureInheritance (object-oriented programming)Electronic mailing listNetwork topologyMoment (mathematics)Template (C++)Database transactionHierarchyFigurate numberCycle (graph theory)Pattern languageGame controllerSoftware developerForm (programming)Validity (statistics)Library (computing)Query languageState of matterString (computer science)Group actionAxiom of choiceMetadataOrder (biology)Student's t-testArithmetic meanType theoryCASE <Informatik>Revision controlCategory of beingElement (mathematics)File systemCovering spaceComputer fileAttribute grammarTraverse (surveying)Computer animation
12:22
Social classRevision control
12:44
Revision controlEmulationCodeFluid staticsMathematical analysisContinuous integrationSoftware testingMaxima and minimaSoftware engineeringComputer configurationType theoryContent (media)Electronic mailing listFunction (mathematics)RootDefault (computer science)Factory (trading post)Template (C++)AuthenticationEuclidean vectorAuthorizationGroup actionPrincipal idealComa BerenicesState of matterComputer iconArtificial neural networkSpecial unitary groupPointer (computer programming)Metropolitan area networkArmData typeConfiguration spaceClefArc (geometry)RippingPhysical lawVector space modelProgrammable read-only memoryExecution unitGrundlagen der MathematikInformation managementDialectInterior (topology)Continuous integration1 (number)Inheritance (object-oriented programming)View (database)Process (computing)Validity (statistics)AdditionComplete metric spaceExtension (kinesiology)Content (media)DatabaseSoftware testingInformationTable (information)Internationalization and localizationExterior algebraGroup actionSuite (music)Condition numberCodeConnectivity (graph theory)Default (computer science)Attribute grammarType theoryForm (programming)Online helpServer (computing)Social classIntegrated development environmentCodeArtistic renderingAuthenticationHuman migrationSingle-precision floating-point formatMathematical analysisComputer configurationBitFluid staticsComputer fileSource codeKey (cryptography)State of matterProjective planeLine (geometry)Stability theoryMultiplication signData structureInformation securityMereologyNumbering schemeRenewal theoryAssociative propertyFilm editingCartesian coordinate systemAuthorizationUser interfaceSolomon (pianist)TwitterSerial portMusical ensembleDrop (liquid)NeuroinformatikFunctional (mathematics)Physical systemComputer animation
18:03
Artistic renderingFormal languageCumulantXML
18:36
Core dumpComputer fileLibrary (computing)MassArithmetic meanAnalytic continuationComputer animation
19:35
HypermediaContent management systemDifferent (Kate Ryan album)Software frameworkCanadian Mathematical SocietyBitRelational databaseHierarchyDatabaseCartesian coordinate systemValidity (statistics)MereologyInheritance (object-oriented programming)CloningContent (media)Physical systemMultiplication signResultantFiber bundleFunction (mathematics)Object-relational mappingCombinational logicType theoryRight angleDialectWebsitePrimitive (album)Data managementMetric systemShared memoryEndliche ModelltheorieIncidence algebraMultiplicationCodeComputer animationLecture/Conference
Transcript: English(auto-generated)
00:03
I'm Andreas, as he said already, I live and work in Germany in a small town in the outskirts of Düsseldorf. I'm not really used to giving talks at conferences, so please be patient with me, and I hope you can stand my accent for the next 25 minutes.
00:20
This talk is obviously going to be about Cotti. It will cover why Cotti came into existence in the first place, who these giants are, on whose shoulders Cotti is standing, how Cotti is built on top of them, and finally, I'll show you some examples to make it easier for you to understand what it's like to work with Cotti.
00:45
So, yet another web framework, really. Aren't there already dozens or even hundreds of them? Why not just not pick one of them? As a matter of fact, all of the Cotti core developers have some common history, and that history is Plone.
01:02
Plone is a full-featured content management system. It has lots and lots of add-ons, and it has some unique features. One of them is the object file system, which is a simple yet effective way to persist objects. You can more or less store plain Python objects into an object database,
01:23
and it supports a tree or file-like object tree. Plone has some great security features. It has the concept of groups, users, roles, and permissions, which enables you to write applications with very fine-grained security.
01:46
Also, workflows are an absolute killer feature of Plone or of the underlying SOAP that was unprecedented at that time. So, if you have a classic content management use case or you need an enterprise content management system application,
02:05
it's probably a perfect solution. But if you need to implement a highly customized application with some CMS-like features, it can quickly become overkill and a major pain. It often has too many features out of the box that you just don't need,
02:22
and you constantly find yourself fighting against the framework. This is particularly caused by the underlying software stack. The Plone system itself is huge. It consists of some 300 packages. It uses multiple competing technologies under the hood, which don't conform to the Zen of Python to have preferably one and exactly one obvious way to do things.
02:48
It's based both on the monolithic SOAP 2 and the component-based SOAP 3 stacks. It uses adapters, multi-adaptors, utilities, all over the place. And long story short, it doesn't fit my brain.
03:03
So, fortunately, there are smarter people than me who have similar history. And wanted to preserve the most important features that SOAP pioneered back in the days. This is all with a modern, clean, and maintainable codebase.
03:21
This was the birth of Pyramid, then known as Repose BFG. The probably greatest thing about Pyramid is it even fits my brain. It has a small core, it has excellent documentation, it is Pythonic and rather low level, which is why you can also sort it into the group of micro-frameworks.
03:44
And it's also unopinionated, which means it makes no assumptions about stuff like persistence and forms, and only basic assumptions about authentication and authorization, which you could call the least common denominator for all kinds of web applications.
04:01
This all makes it a great framework, a framework to build your own framework which supports your opinions. So, Pyramid only provides what you need in every web application. It doesn't come with unneeded ballast, and there's no need to waste time fighting the framework's decisions.
04:22
This makes it a perfect foundation for your own framework. So, what's left to do is make some choices that Pyramid doesn't make for us by intention. Choices we need to make are about persistence. Obviously, you need some means of storage for most applications,
04:40
templating, forms and user input validation, and authentication and authorization sources, amongst others. So, first, persistence. SQLAlchemy, it's probably the most advanced Python object relational mapper out there. It's completely database agnostic,
05:01
meaning that it supports all your relational databases. It has many nice useful features that Koty makes use of, like hybrid properties, association proxies and ordering lists. I won't go into detail here. And particularly useful is that transactions can be bound to the lifecycle of a Pyramid request
05:22
through the use of the Pyramid TM and SOAP.SQLAlchemy packages. So, what Koty does is it implements a node class in pure SQLAlchemy that implements what we had with the object file system in SOAP.
05:40
It follows the adjacency list pattern, which means every node in the tree knows about its parent and its children. There obviously is a single root node, the one which doesn't have a parent. And the node class also supports the dictionary protocol.
06:01
Implementing the dictionary protocol means that you can treat any instance of the node class as a kind of dictionary and get items by their name and key, meaning get children by their name, and you can also set them like you would with a dictionary. So this code here actually is working code in Koty,
06:25
which allows you to get a child of the root element or to add a new child to the root of type document. The dictionary protocol allows us instantaneously to,
06:45
or gives us instantaneously traversal support with Pyramid, because Pyramid only requires an under-get item method to be implemented, which is exactly what the DIC protocol does. Beyond that, we also have an ACL property on the node class
07:05
to have persistent and inheritable access control lists. In Koty, SQLAlchemy is set up to do polymorphic queries. This means a single query can return objects of multiple types.
07:23
For example, if you're querying for nodes, it will hand you back instances of document and file if that node happens to be of these types. Joint table inheritance is another nice feature in SQLAlchemy
07:41
that we use in Koty. It means that the class hierarchy, inheritance hierarchy, is broken up among dependent tables. So each class is represented by its own table, and the respective table only includes attributes that are local to this class.
08:02
The last feature of SQLAlchemy that I'd like to mention are events. SQLAlchemy has a lot of them, but in Koty we actually only use one, the before flush event. A flush is basically the moment when SQLAlchemy talks to the database. It's not the same as a commit.
08:20
It's rather SQLAlchemy emitting SQL and sending it to the DB, which can happen multiple times during a transaction. Of course, a commit always includes a flush. A SQLAlchemy before flush event can trigger multiple,
08:40
more nuanced events in Koty. This is a mere convenience thing, so it eases you to have respective event subscribers. So not exactly part of SQLAlchemy,
09:03
but also closely related and also written by the same author, is Alembic. Alembic is a database migration tool that supports transactional data definition language operations, if your database does support it, and transactional data manipulation language operations.
09:24
Another useful feature is that it supports multiple environments. This comes in extremely handy for add-ons in Koty. So each add-on can have its own migration environment, which is independent of the others.
09:43
Koty includes a script to perform either specific migrations or all migrations for itself and all its add-ons at once. You can either upgrade to a specific version or to the latest known revision. Downgrades are also supported by Alembic
10:00
and also by Koty, for the case that something goes wrong during a migration and your database does not support transactional data definition language. So user interface. Unless you have a user interface designer on your team and can't be sure that he will stay,
10:21
just use Bootstrap, you can't do much wrong with it. Next, forms. Most form libraries handle creation of forms as well as their validation. Kolanda and Dform are different in that aspect. With Kolanda, you can define the data schema,
10:41
validate and deserialize HTML forms as well as JSON or XML data, and serialize Python structures to strings, mappings, and lists that can be consumed by Dform. Which in form only renders the form.
11:02
Dform 2 uses Bootstrap 3 out of the box. You could change that, but we don't need to because that's the choice we also made for Koty. In fact, Dform 2 is a merge of Dform 1 and Dform Bootstrap, which was a package by the Koty developers that provided bootstrap templates for Dform 1.
11:28
So, repost.workflow is another package from the pylons ecosystem. A content workflow system that supports the concepts of states and transitions. States define roles to permission mappings
11:43
and transitions define transitions between these states and the circumstances in which they might be executed. And actually this package allows us to implement the complete and exact feature set that we had in Zope earlier.
12:05
Depo is another giant that recently made it into Koty. I won't cover that because I've seen there's a talk tomorrow at 11 in Baria 2. So, visit that one. And let's finally come to Koty itself.
12:24
As mentioned in the talk outline, Koty is a rather small package that wires up all those giants in a sensible way. It was started in 2011 by Daniel Nouri and he did the first version in just two days
12:42
and that already contained the Node class with all its features. I learned about Koty a year later and joined the project almost immediately. I'm skipping the rest because I'm running out of time already.
13:01
Code quality is an extremely important thing to us. We have an extensive test suite based on Pytest. We make heavy use of reusable fixtures that are exported as a Pytest plug-in that can be consumed by Koty add-ons. We have continuous integration.
13:21
We use some tools for static source code analysis. You shouldn't blindly follow all of their suggestions, but they can give very valuable insights on your code and help you to improve it further. And we also try to keep our requirements up to date
13:40
and succeed with it until now. Koty is exclusively configured through any files. As Koty is a plain pyramid and therefore also a whiskey application, you can run it under your preferred whiskey server.
14:02
Almost every aspect of Koty can be configured with an option in these files. Koty provides sensible defaults for each option so that you don't have to specify any of them. But the important thing to take away from this is you can override almost everything in Koty,
14:22
but you don't need to. I'm skipping those options also. So let me just give you one example of how we combine multiple best-of-breed components to a fully functional system. With regard to security,
14:40
we use SQLAlchemy to store principles that is users and groups in the database and attach inheritable ACLs to each node. We use pyramid for the process of authentication and authorization, and we use repost.workflow to recompute these ACLs on every workflow state change.
15:04
So, code example. We provide a complete Koty scaffold based on the pcreate tool from pyramid with Koty, and after running the quant, you will have a fully functional add-on for Koty with a complete test suite
15:22
and continuous integration on Travis also set up already. That created package contains a custom content type, default and alternatives used for that type, completely set up internationalization structure, also a migration environment for use with Alembic,
15:42
and that scaffold is also always kept up to date to adhere to the recent coding conventions as suggested by the Koty team, and it's as well completely tested within our continuous integration infrastructure, so we can be sure that it works at any time.
16:04
Okay, I have to run a bit through this. The custom content type looks similar to this. It inherits from Koty's content class, which is again just the same default that you should probably inherit all your types from. It has a primary and foreign key
16:22
to the parent stable ID column that's needed for the joint table inheritance of SQLAlchemy, and then it adds the attributes that are actually added by this class in addition to the inherited ones. So last, there's a type info that tells Koty how, where, and under which conditions
16:43
that content type should be made available through the user interface, which then results in this dropdown. A colander schema is responsible for serialization,
17:01
deserialization, validation, and form creation through deform. It also inherits from the parent schema and only adds schema nodes that are provided by the custom content class. The add and edit forms,
17:21
they're like everything in Koty plain, pre-emitted views. They are configured by the user view config decorator. The actual form rendering and validation is again provided by the base classes, so you don't have to write a single line of code for that
17:42
unless you want something to be done differently. And what you see here is a complete and working example. There's nothing left out. So this is what the form looks like with the validation error.
18:00
Yeah, last but not least, here's the code for a simple view. Again, just ordinary pre-emitted views, just like everything in Koty. In this first view, there is a template specified for rendering. In Koty, we use the chameleon templating language,
18:22
but you can choose anything that's supported by pre-emitted. Okay, I have to come to the end. Okay, future of Koty. Koty will always stay lean and mean in all of the right ways,
18:41
so there won't go any features into the core that can be packaged into an add-on. We're even thinking of moving some features out of the core into add-ons. For example, we have a mass file upload feature that requires some JavaScript libraries that are rather heavy and only require for that particular feature,
19:03
so the primary goal of Koty is to stay lean and clean. Yeah, and we don't have Python 3 support yet, but all of our dependencies right now support it, and it simply hasn't been done,
19:22
but it shouldn't be a lot of work, and if someone would be interested in working on that during the conference, I'd be happy to team up. So, thank you, try out yourself, and thank you for listening. Thank you.
19:43
We do have time for questions. About five minutes. Any questions? Yes. When representing metrics, how do we correctly measure what we care... Thank you.
20:00
Try again. As defining your content types, you have the polymorphic join that'll bring back the actual sub... or the content types that you declare. As you declare more content types, on a lot of the Koty queries, is it doing a join across all those tables, and what's the impact on performance?
20:22
It depends how long this join gets, but actually modern relational databases are really good at this. So, it's never come to an issue for us yet, and we have quite large systems with a lot of inheritance and mighty billion rows, so it works.
20:42
Second question, if I may. So, I downloaded it and was just looking at the debug output with a lot of the ACL checks for a nested hierarchy of content, and it seemed to be querying the database for the same ACL validations
21:02
quite a bit for different parts of the UI, so it didn't seem to be caching the result. Is that something that I was just noticing, or is it...? I don't know right now what you mean by that. You could show me later, perhaps. Any more questions?
21:27
We need to get it in a microphone, so it's... Hello, thank you very much for the talk. So, I'm not very familiar with all this technology, I just know the basics. What would be the major advantage versus Django or Flask, for example?
21:44
Yeah, you cannot probably compare it directly. You would compare Pyramids to Flask, which both are kind of micro-frameworks. Django is more a macro-framework, if you will, which contains its own ORM. And Koty is just a bundle of technologies
22:07
that we use in combination with Pyramid. So, if you like to use Pyramid and your preferred persistence layer happens to be SQLAlchemy, then Koty does a lot of things for you already that you can use.
22:22
Any more questions? I do have a quick one. What kind of applications do you make with this framework? We do make all kinds of applications, some of them with large user-contributed content.
22:40
We do make custom applications where a small part needs a content management system but not a full-featured beast-like clone or Django CMS or something like that. The advantage is that the framework doesn't get in your way.
23:02
You basically have a plain Pyramid application with all you can do in Pyramid. Koty is nothing more. It doesn't add conventions. So it just configures Pyramid in a meaningful way. Any more questions? I think we're good, so thank you again.
23:21
Thank you.