Turn it up, okay. Who was not at the Barcelona Guillotina presentation? Okay, just a bit recap. Three years ago, we started to do something that we named Plone Server.

Naming, it's always difficult. And we tried to push asynchronous mode to see what we can do. We meet in Barcelona, 15 Plone developers. We try to do what we can do to write from scratch a backend using ZODB and everything else.

One year later at Cologne Spring, we decided to rename it and to rename it to Guillotina and to use Postgres database directly. And we are here now. So now, one year later after Barcelona Plone Conference,

Guillotina, it's really stable, really mature, and I will be really happy to explain what is now and what is going to be from our point of view. So, first of all, who am I? For those among me, you already did the presentation. I'm co-author of Guillotina, Plone Foundation member.

He's co-founder and an entrepreneur. I like to start new companies and I'm a full stack developer. And I'm also a musician, a dancer, I'm mostly a Catalan over everything. So, what is Guillotina? Well, I try to write a sentence.

An ecosystem is not a framework. I like to repeat that a lot of times. It's not that it's one package, that it's Guillotina package. It's a bunch of packages that are creating an ecosystem of packages that are used to work together

and they are designed to scale and to manage resources with security and a traversal storage. That's the main goal of Guillotina. The two main authors and main contributors

are Nathan and me. And he's not able to be here, but he sends, he's in the slides a lot of times, so I really want to mention him because we've been working on this for the last two years together and it's been a pleasure. And most of the ideas are just he and me

discussing and deciding what to do every time. So, before starting, this is going to be a code talk. And before starting with the gray slides, I wanted to do a bunch of colorish slides.

And this is my dream of what is Guillotina, or what I want that Guillotina becomes and what is it right now. So first, we wanted to be something that it's easy to use, besides it's a sync and you need to write a sync and await all the time, sorry for that,

that allows to develop ideas that are scalable and extensible with options to grow from one user to thousands. We had a use case, we were working on the same company, and we had a use case where we needed to grow a project from a really small use case to a really large use case with lots of users and a lot of data. We wanted to have a pipeline of components

to host resources, resources is the name we use for what Plone calls content, and to split into smaller problems, something that it may be a really complex problem from outside.

My dream here is that everybody could use this to deploy large systems to iterate and to deliver fast, mostly designed for startups, where they start with a really small prototype and they need to grow, grow, and grow, adding new uses and new resources. That is always community driven,

that no company belongs, has the IP or the rights on the software, it belongs to the Plone Foundation, and I'm really happy about that. And trying to keep it simple and small. As you see, my dream is not to replace Plone. So it's not a tool to replace Plone,

to replace the backend of Plone, it may become, but it's not why we created this thing. We created this thing to be a tool for creating projects. Okay? Because it's the most asked question. So, let's start with the technical side

of the presentation. First of all, Villatin is based in Plone REST API, that thank you to Timo and all the team that was building that. We defined this API as the entry point, it's a valid API, it's a good API to manage resources and to manage content.

And, well, it was easy to implement the backend with this API, because we also are traversal, and it's the main request for this kind of APIs. Then for the storage layer. Here, after iterating over different databases,

we went through MongoDB, and we went through, even Redis, we tried. We decided that the best database that fits our needs for the transactional model, and for the kind of way we were willing to interact with the database layer was Postgres, and later on, appeared to CockroachDB.

I don't know if you know, CockroachDB is an amazing database built on Go. It talks the binary protocol from Postgres, so you can connect anything that you can connect to a Postgres, to a CockroachDB. And it uses a raft consensus algorithm in order to distribute the data across different nodes with replication, and we've been having some deployments

with nearly, I don't know, 1,500 requests, transactions per second, with millions of documents inside the database. So, it performs really well, it's a bit tricky,

but, and it's new, so you need to follow up the book fixing and the versioning, but it's a really, really amazing database. And from our point of view, we only needed to change a few amount of things, because it talks Postgres binary protocol. For the indexing, indexing the content and being able to search and do full-text searching,

we choose Elasticsearch, that we kinda like it. And we also are providing support for Postgres JSON-B, so we can serialize on the Postgres JSON, as a JSON-B field. On the catch layer, we are using Redis

and memory catch, it's a really small memory catch, and it's mostly everything delegated to Redis, which is doing the invalidation, and I'm going to talk a bit more about that later. And about blobs, we decided to choose S3

and Google Cloud Storage as the main citizens of blobs on Guillotina. We, as everything is in a sync framework, we are able to stream files up and down to S3 and Google Cloud Storage, so we don't need to host the file and memory ever,

and we are just streaming from the web browser or whatever client you are connecting to Guillotina to the real storage. And we only have a buffer of 500 key kilobytes, and we are just streaming the file up and down.

And we also support database blobs that Cookroach and Postgres also has, in case that you don't want to use a cloud file storage. Well, these are the layers, the main layers that we needed to choose to reuse the tools

that we have already stable on the ecosystem. But now let's go into inside the Guillotina architecture. I just speak from Guillotina the most interesting concepts, and I'm going to try to explain a bit of them.

First, the configuration. We kind of decided that we don't want to store the configuration on the database, so everything is explicitly configured externally to the database in a YAML that you can provide when you are starting Guillotina.

This config.yaml that you are sending may overwrite any of the configurations that you have on your system. And what we do is we merge this configuration with all the configurations from the different packages that we are loading on our system.

All this information can be accessible from the code when you are importing app settings from Guillotina. So here you get a large dictionary that with all the configuration that you have for your site, everything, the Elasticsearch URL, the database URL, anything that you need is going to be

on this large dictionary. Decorators, because maybe we suffered a lot about GCML, we decided to move to the pyramid way of defining the different elements that developer needs.

So we have a long list of decorators to define nearly everything that we think that it's needed. So the first one, configure.service, it's to define an endpoint. You want to define an endpoint for a specific kind of content. This one, you want to define a new content type,

a new resource type. Second one, a vocabulary, a behavior. We implement behaviors as the first-class citizens of Guillotina. Add-ons, if you want to register an add-on to be able to be installed. Adapter subscribers and utility. Different languages, even the permissions,

the roles, granting, and granting all can be defined with this decorator. JSON schema definition, it's a way that all the input-output from the API is done with JSON. So we needed to find a way to be able

to define pieces of JSON that are going to be reutilized in different places of the response. Value serializer and binary deserializer are two decorators to define how you want to serialize and deserialize a specific field.

Imagine that you have an integer, how you want to serialize that in JSON, and from JSON, how you want to convert that to the value that we are going to use. So you can configure your own for your own types. And renderer, right now there is only one render register that is JSON, but we are working on having more renderers

to be able to convert any internal structure to protobuffer or any other kind of format that you want to use. More things, what happens on Guillotina when we are running Guillotina, no?

So the first thing that happens is that we register all the configuration, the merge that I said about all the dictionaries of configuration. Then we register all the adapters, events, and utilities that Guillotina has out of the box.

Then we scan all the packages that are on the key applications on app settings. That means that if your application is not listed on this key, it's not going to be loaded. Then we copy it from pyramid also.

That include me root function that we are calling on the package in order to do specific things that the package needs to do at bootstrapping them. We register all the adapters from this internal package. Then we load multiple databases. This is important. Guillotina, you can define on the configuration file

multiple databases that you want to load. And you can also add new databases when the application has already started or removed them. Then you configure a static folder and JavaScript apps folder. These two are for static files, clear?

And JavaScript applications. JavaScript applications is, you know that single page app needs to have this wildcard routing that everything that it's from one point to, as the children's of that point, are going to be needed to be rendered as the same application.

So we created this kind of mapping where everything that you are pointing is going to render the same JavaScript file. So we can serve one page applications without engines or anything else. We just need the Guillotina. Then we create an error sake

for creating tokens. And load the asynchronous utilities. I'm going to go there later. Security policy. Well, like Zop, similar to what Zop is doing,

we have the code configuration. It's the permissions and roles that we write down with the decorators that I explained before. The user provided global configuration. So maybe we have a user that it's manager. So this information comes from the user information.

And then we have our local security policy. It's quite similar to the Zop one. So you have roles, permissions, and principles, and the combination of all of them linked to users and groups.

Maybe the difference from the latest versions of Zop is that we have something called allow single, which means that if you give a permission on a specific node on the tree, this permission is not going to be inherited in the children one.

So you can assign. On this specific node, I want that this user is manager. So that the children of this node is, this permission is not going to be inherited. You can also deny or unset.

Guillotina is not opinionated about users. Doesn't have nearly anything about users. There is only one user, it's the root user, nothing else. Why? Because we wanted that anything that provides

a user information, groups, or permissions is delegated to another package that can implement maybe, I don't know, an OAuth system or whatever. So we needed to create an interface in order to extract and validate the credentials from the user that is connected and to get the user that is connecting to the system.

First, we get the credentials. We have provided bearer support, basic support, our WebSocket token, and cookie. So we can extract the credentials from any of the systems. Then we can validate. We have JSON Web Token and Salted Hash.

And finally, we get the user. We have only the root user on Guillotina and then we have three different packages that provides different databases that provides users. For example, GuillotinaDB users stores the users as content.

So each user, it's the membrane project from Plone a long time ago. So it's the same thing but really simple. Either IDP connects to an OAuth2 provider that it's written in Go that you can deploy on-premise that it's really cool.

And authentication provides delegated authentication with Twitter, Google, Facebook, and all these kind of social networks. More things, special permissions that we have.

Well, first, access content. We created this permission that is a bit controversial because we wanted to make sure that a user is able to traverse to an object. So Guillotina doesn't implement the security proxying from ZOAP.

It's not wrapping the objects in a security model that you make sure that you can access or not a specific field or do whatever with the object. So our protection of the specific object, it's done at the traversal time, just making sure that you can access to this object with the access content permission.

And then the view permission that you specifically need to run the view. If you want to modify, you want to add, you want to delete, whatever. So you need to have, in order to edit specific content, this content needs to have access content and modified content.

Another special permission that we have on the system, it's mount a database. So you can dynamically decide, I want to mount this Postgres that it's wherever on this Guillotina or I want to mount a file system database.

There is another permission that it's get API definition. All the endpoints as it's based on completely a REST API, we are not rendering anything on Guillotina. It's just providing REST API responses, a permission to get the API definition

and Guillotina public to have anonymous access to it. Well, resources. Whoever is used to Plone knows that Plone has dexterity content types and behaviors.

And you have a large dexterity content type and some behaviors that you are applying to it. We decided to copy more or less the same idea but adding something new. It's called dynamic behaviors. The behaviors that Plone has is what we have here called static behaviors.

It means that on the class, when you are defining the class, you are saying this class is going to have this amount of content, this amount of behaviors. But dynamically, when there is an instance of that class running, you can decide to apply a specific behavior on that instance. And it gets added to the index.

Also, if it involves adding annotations or whatever. So you can have objects that has the static behaviors and suddenly you want that it has an attachment. So I add the behavior of attachment or I add a behavior of Dublin Core, whatever.

Fields. Well, we support most of the Zoopa schema fields. The only one that it's noticeable here is cloud file field. This is a specific field which depending

if you have S3 storage or Google Cloud storage installed, it will store the file on the database on your local Postgres or on a Google Cloud or an S3. You just always use this field and depending on what you have installed,

it's going to use one. Support or the other. And we have some more kind of hacky fields. One is called dynamic field behavior. And this is what we call, through the API, new fields that get indexed.

It means that, imagine that you have a, that you want to have a, I don't know, a mosaic kind of layout where you are defining new fields and you are defining these fields and you want that the content of this field gets indexed so when you are searching on full text search,

you are also finding the content of these fields that you are creating. So we created this behavior where you can create any kind of fields on a behavior and then gets automatically everything indexed. JSON field, it's easy. You define the JSON field with a JSON schema and you can push any JSON data

that it's validated by the JSON field. Bucket field. This is a need that we had on the last project we did with Nathan. Sometimes you have an object that is really, really, really large.

Imagine that you want to store on a node a conversation that you have on a slack, no? Forever. That maybe it's kind of millions of messages on the system, no? And you want to store all on the same object instead of nesting them on children.

So we created something that it's able to group 1000 sub-objects inside an annotation and link to the next one so we have a pointer to the first and to the last and we can go through the history of that large list.

It performs really well and it's mostly designed to really long objects that you want to store on one object. Patch field. This field, it's really not a field. It's a wrapper of a field.

You use that as a wrapper of a list or a dictionary or an integer, for example. And it means that in order to interact with a field through the API, I don't know, Timo, do you know if long REST API supports patch operations right now?

Yeah? No patch in terms of the verb. Patch in terms of the JSON that you have a list on a field and you just want to add one element on that list. It's hard-coded. So this is a way to, so you define the field saying, okay, this field that it's a list

instead of if I want to interact through the API, I don't need to send again all the list if I want just to delete one element or add one element. So you can define operations onto other existing fields and then you just need to say, I want to append and this value

and then automatically Python is able to just append this element on the list. You don't need to send again on the API and we also implemented for integers if we want to increase or decrease the integer value or reset to the default value.

Storage. Well, this is just, I'm trying to go over everything, maybe too much, but just to talk a bit about the storage layer. This is the schema that we have for the database.

This is how we are storing everything on Postgres and CockroachDB. And mostly we have two kind of records on the SQL schema. One is the main object, meaning the object that is going to be on the tree. And the other one, it's the annotation of one of these objects.

There is only these two kind of objects. The first element, the first column, it's the object ID and kind of romantic so we maintain it a lot of names from ZOBE to make it friendly. Transaction ID, in order to, when we are in the boating phase, which is the size

of the object that we are serializing. Part. Part is a specific column that we are using when the size of the guillotina grows a lot, which can be defined with an adapter. It means that you can define if one object

goes to one partition or another partition of the database. Postgres 10 supports automatically partitioning of tables and mapping all of them in one view. So this is used for that triggering on Postgres

to split the data across multiple tables and to be able to scale on Postgres. Resource, it's a boolean that just shows if it's a resource, meaning a main object or a notation. Of, it does the reference to the object ID

in case that I am a notation. All transaction ID is the previous transaction ID. Paran ID, it's used for the main objects to know which is the parent, the ID. It's the, the ID is the real ID,

the ID that we have on the URL. We want it to serialize as a specific column. Type, what kind of resource is it? Adjacent B field that we are mostly using on, if we are doing indexing on the database. And the state that it's the pickle of the object.

Okay, we have also a registry. You can access a request that container settings, register an interface and get and set the value of the field.

No, no strange things. Async utilities. This is something that I really like is that you can define utilities that are going to be bootstrapped at that bootstrapping time and are asynchronous utilities. So they can run forever doing any input open.

Or the operation that you need during the cycle of the whole life of the process. And it's going to connect to different systems if you need. For example, this is a stupid use case that it's a work for utility. And we provide, we define this on the config.yaml

that we want to load that when we are bootstrapping. Then we define the utility. Just inheriting from ISing utility and configuring the utility that provides this utility.

And then we have these two specific methods, initialize and finalize, which is going to be executed when we are starting and when we are shutting down the process. And both are synchronous. They have the generic loop so they are able to do operations like, I don't know, give me the credentials from go cloud to storage

or make sure that everything is clean on the database layer or whatever. And then on the code, of course, we can also get this utility whenever we want using the standard get utility.

Well, the view at the end is what's important. I think I wrote three examples of build definition. It's using the decorator config.service. And here you define which is the context, which is the resource interface that we are going to apply this endpoint,

which is the method, which permission is required to execute this view. Then we have the endpoint. In this case, you see that we have also URL dispatch inside the endpoint. So you can parameterize with multiple levels

the endpoint inside the view to give the publish traverse method on ZOOP, publisher, so to be able to go into different levels. Summary and responses. Most of the calls from Guillotina

have a lot of documentation on the definition of the service because we provide a swagger out of the box so all this information is serialized on the swagger information so you can check all the endpoints and make sure try them out and know which are the values that you need to send.

Then you have the class where you have the request and the context. This is another example. A special thing about this example is the allow access true. This is the only way to skip the access content check

on the traversal. So you can define, okay, I want to allow to access to this view no matter if the user has access to the content that it's going to be access. For example, the login endpoint needs to have this. Otherwise, you're never going to be able to call it.

Well, another example of how to do the explanation about responses. So when we have a 200, we are going to receive something that it's a schema that it's a reference to something we call application

and we define it application with that decorator I explained at the beginning where you're defining JSON blocks. Well, the flow of the request is quite simple. We just do the traverse to the object. We check access content.

We look for the view. We check there is some language, the language translation mechanism that we check if you're looking for another language to get the multilingual idea, it's already implemented. We check the permission, the view permission, we execute the view, then we do the commit

or the abort and we do the post-commit operations and one really nice feature that we are using a lot is that if you need to do something when this is already committed, so maybe, I don't know, logging on some stupid place in a synchronous way,

we have the features. You can define multiple features in your view that is going to be executed after the commit and this is done after the response has already been delivered to the client. So maybe you want to lock something on lock stash or whatever, you need to do any kind of

after commit operation that the user doesn't need to know if it worked or didn't work, so we can register all of them in the features. Cache invalidation, well, there is a package called guillotino-redis-cache.

This is quite a bit complex problem to fix. It's not perfect, but it works really fast and at the end, all the guillotinos has a pub-sub connection where they receive when an object gets invalidated and so they need to retrieve again

from the database and we also store the objects with the TTL on the redis to get them faster, the pickles. Indexing, there is a lot of things

to explain about indexing. We have the Elasticsearch plugin, it's really cool and it works really well. At the end, we convert to JSON with a specific mechanism where you can define indexes with directives on the interface and then an Elasticsearch

is on a post-commit hook and on Postgres is sent on the same commit and then for security, on Elasticsearch, we're using access roles and access users that we are defining for each object. Okay, all this backend thing, it's really nice.

I have only five minutes, so I'm going to go a bit faster We have a UI, thanks to Eric Brehold that it's really nice. You can log in, get the database, get a container

and being able to browse like a ZMI, but the ZMI, you can create new content here and you can do whatever you want. Now I think it's done, but well, it's really nice.

Tomorrow we will show more about this on the Guillotina CMS because there is also rich text editor. So which is the status?

We know that there is five companies that are using it. We know deployments that think it's more than 20 million right now. It's nearly 30 million or 50 million objects that are deployed. We have 93% of code coverage. We are now in four to 11. We are releasing five by the end of the year.

There is a lot of documentation. There is a lot of tutorials. There is a Gitter channel. There is two main repositories on GitHub called Guillotina Web and the Plone Repository. And I want to do a really short demo of something specific.

Pop, pop, pop. There is a repository called Guillotina Processing that opens, if you run it, opens a Jupyter notebook where there is some kind of interesting examples. It's just for teaching Guillotina,

not for production, of course. This first notebook, it's about using the API in order to push onto the system a CSV file with a lot of articles. There is, I think, this is standard REST API,

Plone REST API. We create a container. We install an add-on, another add-on. We read a CSV file and we push all the articles as documents on the system. And there is 2,579 articles.

So this is using the REST API, no strange thing. If you want to check, it's in the Guillotina underscore Processing. But the interesting one, it's the second one, the compute one, that we are able, since a Jupyter supports asyncIO and wrestling in the last month,

you're able to start a Guillotina inside the Jupyter. You can create the application. You can connect. We use this context manager called Content API, sending which database you want to use. So you can do, it opens a transaction

and getting out of it closes the transaction. And this example goes through all the content and creates a machine learning model with TensorFlow and Keras to classify news between if it's political or sport news.

It's quite interesting. I think I have, I'm running out of time, so I'm just going to the last two slides. There's the roadmap and where I'm going to spring. We have two main problems right now. One is the how to index the security information

and how to be able to search the security information without the need of re-indexing the children when we are changing the security information of OneNote. And we have some proven concepts of ways of organizing the data on the indexer

so we don't need to serialize everything to each object. And how to distribute more, we are reaching limits on the performance and we want to be able to distribute the data or sharding more

on different systems. We have a lot of ideas on mine. We are going to try to do something with Rust and Raft, zero time. So thank you.

Hey, thanks for your talk. What Swagger version to export to then? Is the exported schema, does it include the schemas of the content and all the validation fields?

Or is it just a simple schema? I think it's the latest one.

Can you show me the YAML or the JSON file?

It's automatically, everything is automatically generated so I cannot show you, I cannot access it. I can show you later. But I think it's the latest version of Swagger because we are using the latest version of Swagger, the client. Can I use, for example, React Swagger library to generate the React client?

Then you need to provide me a JSON or a YAML file. Yeah, the problem is that this is traversal. It's based on traversal. So you need to change the path to define, to know which endpoints do you have on that path. So you cannot do that. You cannot do automatically generating code because the definition depends on which path

you want to access to. Do you use, like between the code itself, do you use an ORM or how do you get things into the data? The ORM is something we built ourselves

based on persistent dot persistent. We created something a bit different that it's much thinner and that allows to work with a sync.io point of view. Anyone else, others?

So our transaction started at the beginning of our request. How does the transaction system work and as well as the life cycle around persistence,

if you have ghosts and these sorts of things? Yeah, the transaction starts after the traverse, after we get the view, just before the view, just before executing the view and just finish, commit it after the view. So we keep it as small as possible on time. And about the state of the objects,

we don't have ghosts. We are just, the objects that are on memory are getting validated and are linked to the specific request that you are connected to. We have weak references to the objects

but we don't have the concept that persistent dot persistent have as ghosts.

So it's amazing what you've built, but I'm a bit afraid, like I'm asking myself why? For example, like you said, you have support for automatic table partitioning but the official documentation for Postgres says until your tables are several hundred gigabytes long,

you don't need table partitioning. You most probably don't need table partitioning and you know, pyramid can run with async G unicorn worker so what's like, comparing to pyramid, what would be the main selling point and if that is speed, then why didn't you use Go or Rust to write it? That's a really good question. Mainly because I didn't know Rust and Go.

So is the main selling point speed compared to pyramid, for example? The main selling point is the security system from my point of view, that we are using this traversal security system where we are able to use the same kind of traversal pyramid. Doesn't have it the same way as we implemented,

as powerful, out of the box. I don't know if it has evolved a lot during the last year. But, and the asynchronous, one of the really cool features is that you can connect everything, events, to WebSockets, for example.

And so you are modifying an object, or you are throwing a modification of an object that is an event, events are asynchronous here. So you can plug a WebSocket to connect this and send to another place, any kind of information. This kind of connections of asynchronous input-output

is really powerful. There is a project where they are using a lot this. And mostly we reached limits of scalability with pyramid. With Zoop, we tried to go deep to tune it.

We were not able to make them faster. Maybe it was because we were not enough experts on that technology. And we needed something with a nice developer experience. So Python, it's a good tool.