We are going to do a technical presentation of Guilutina.

Well, mostly we already explained why we created it. And I was saying all the time, we are going to explain about Asin Callao, we are going to explain what was the problems that we were facing when we were doing the framework two years ago. And this is the talk that we are going to go through,

all the different points that we had pain doing, using Plone and using ZODB and using real storage. So we are going to explain all the decisions we needed to make in order to build the framework that fulfills our requirements.

First of all, if you don't know, I'm Ramon Navarro. I have two companies, Iskren and StasiFi. I'm a Plone Foundation member and I'm Catalan, and this is the demonstration in Catalonia last week. Hi, and I'm Nathan Van Geem. I work at ONA as principal engineer.

I'm a Plone Foundation member and I don't really play guitar anymore. I just felt I needed that because Ramon always talks about playing music and I'm an American. Normally I wouldn't say anything about that either, but you know.

Yeah, so we're just going to try to go topic by topic to talk about some of the different areas that we used to tweak performance and create a framework for scaling to a lot of data in Python. And some of those are, I mean, I guess I can repeat them, but you can read them as well by asyncio.

And then just the design of the database, caching and lifecycle management, indexing, file storage and queues. And so the first thing we'll go over is asyncio. And I guess I just want to stress that asyncio is really important

for Python and the web in general going forward. The threaded approach, like, okay, we'll just talk about how asyncio is different than a threaded app. In a threaded app, when a request, an HTTP request comes in, it's assigned a thread and that thread is active

through the duration of the request. And then once it finishes, the thread is returned to do another request. So you're limited to the number of simultaneous connections for the number of threads that you have on your application. In Python, that usually means two, because Python doesn't perform very well with lots of threads.

The best way to deploy Python is with two threads per process. So you can just keep doing more processes to scale, which is fine, but then you don't have any memory sharing or anything like that. But moreover, it's just also, it's the differences between how you block on your application.

So if you have the threaded approach, a request comes in and then you communicate with, say, PostgreSQL and maybe it happens to be a slow query and you're blocking for three seconds to wait for that query to finish. Now that thread is consumed for that whole three seconds while that query is communicating that Python's not doing any CPU,

it's not doing anything at all, but it's still blocking and it's preventing other people from being able to have their requests serviced. So that's the threaded model of request per thread and that's Django right now, that's Flask and Pyramid.

And so think about how this applies to situations where you're communicating with lots of services. You're sending things to a message queue, you're communicating with Redis for caching, you're doing OAuth for authentication or any kind of authentication or authorization provider,

you're doing S3 for file storage, like any of these services, they're all over like HTTP or some kind of TCP connection, and now you're blocking on communication. If there's anything slow in any of those services, you're blocking and you're not doing anything. And so this is where asyncio is useful.

It's non-blocking network IO now and you can communicate with all these other services and while you're communicating, there's very little overhead in maintaining that connection and you can really push the performance of Python where you weren't able to before. Moreover, just think about the context of microservices.

Even if you think microservices is crap, the reality is you don't have to be microservices. You're still using a lot of other services. You're connecting to other APIs in a number of ways. You're at least using multiple database layers. Most applications you develop now are not just database only.

You're going to be communicating with something, some API, maybe an authentication service or something. So this is going to be a problem if you have any sort of substantial load. And to illustrate this even further, I do have an example of some code

to show how this can be... This is going to be an example of a CPU-bound application versus a network-bound application. So I'll back up a little bit because I forgot to explain the downside of asyncio.

So asyncio is a single-threaded approach to running your application. So one process, one thread. Your event loop, which is running the async code, is only on one thread. So if there's any CPU-bound code there,

it's going to block on CPU. Because no other code can run at the same time if you're executing CPU. It's single-threaded. So it's really important that you don't block on CPU. Whereas threads, if you have high CPU load, you can switch threads every once in a while

and you won't ever have a situation where you're blocking all other code from running. So that's one downside. So if you know you have CPU-bound code, you have to be careful about it. In asyncio, there's a way to run CPU-bound code in a thread pool. So you just have to know what you're doing. But anyways, this is the demonstrate. This is some code that just crunches numbers

for half a second, and we can see how it behaves. And the regular asyncio approach is two requests a second. And the threaded approach, without setting any threads, two requests a second, behaves the same. Asyncio with the executer, that's what I was talking about, how you can throw CPU-bound code in a thread pool

that does slightly better than a threaded application with 20 threads, which you don't really want to do, but that's just to say you needed to try to scale it further with a single process. So the problem isn't that much different

with the CPU-bound app. But if you go to a network-bound app, and this is the same sort of thing, we're just connecting to a service, and that service is just something, just a service that is delaying for half a second before it returns, which is an asyncio service, of course. And then the difference is this.

And this is only bound by the number of concurrent requests the library was using per host. So it could go further. It's just that was some arbitrary limit they had on the number of concurrent connections per host.

And the threaded app is abysmal for the performance. Yes? Just to say that Asyncio, it's amazing for network, but there is a lot of other input-output things that we have on our computers that Asyncio, it's also really helpful. We are not using a Guillotina specifically

because most of the things is kind of a REST API and it's mostly network communication, but we are using also a lot for, for example, accessing the disk with long operations that you need to do on disk, or I'm sure that there is a lot of input-output communications with devices with the kernel

that you could build drivers for Asyncio in order to be also on the event loop. So everything that you need to wait, Asyncio, it's amazing. Sub-processing. Sub-processing, yeah, you just start a new process and we have processes that start

a lot of different processes and wait for them. Meanwhile, they are doing other things, serving HTTP so we can monitor the processes that are running. All this input-output, it's so easy with Asyncio. And there was another thing is Python delivers with a standard event loop, a basic one that it works really well

for most of the cases, but there are other implementations of event loops. For example, the UBLOOP, that it's much faster than the standard one that it's the same as Node.js. And there is also the Tokyo event loop that it's written in Rust, that it's also really fast. So the system is so plugable

that if you really need this extra 20% of a speed, you can even switch the event pool to something that it's much more performant. That's me? Sorry. So Asyncio was the main important thing that we needed to come. So it was the first thing that we decided

that were going to happen. Then the second thing that we decided is that we needed a database, a place to store. We started storing things on a file system, on folders, and well, it was okay. But we needed some kind of more strong way

of storing with transactions so there is a lot of databases that offers that. And we went on to understanding how ReLa storage works. We say ReLa storage has been there for a while. It's been growing. So we want a pickle system, what we can copy.

And we went to check ReLa storage. And we decided a design that it's a bit inspired with ReLa storage also with that database that Jim built, NewDB. So Lutino, it's traversal.

So there is main nodes that you have on the tree. And these main nodes are registries on a table. Which this makes easy to be able to do a select on the table and to do a dump and to get all that objects out of it.

Then we decided that we don't have enough with a row, with a byte field on Postgres to store all the information that we want from an object because an object maybe can grow up a lot and we need to include extra information.

So we copied the concept of annotation from Plone. And we decided to have annotation registries that links to the node object. So if you do a select on a Lutino database, you will see rows that are the node objects, the basic nodes on the tree

and nodes that are annotations to these node objects. Then, of course, we are a tree. So on the SQL structure, we needed to define a way of defining pointers even to the children or to the parent. In our case, we decided to point to the parent.

So each row has a pointer to the parent or to the node that it's an annotation for. Why we did that? Because, of course, it's much easier to check conflicts because you are only pointing to one.

At the beginning when we started to do the design, we did not use the JSONB indexing from Postgres. So we decided that by then,

we are going only to store the pickle with a byte field on Postgres and then indexing, it's for another system. We are microservices, so Guillotine is only taking care of storing with transactions the main objects and the notation objects.

These are some of the configurations on Postgres that we have right now. Mostly, it's a simple one table, one big table with all the information. So each database that is mounted on a point of the tree,

it's at that table, and you can have multiple tables and mount on different endpoints on Postgres and on the URLs of Guillotine. Then there is a lot of foreign keys that make sure that you cannot delete. If you delete the children,

all the parent, all the children gets deleted. If, for example, you want to do a post with the Plone REST API to create an object and you define which is the ID, checking if that ID exists, it's a simple operation because there is a primary key that doesn't allow to have different IDs for the same parent.

So the database itself provides consistency. But related to that, you don't know until you try to commit if there is a conflict in this way because we didn't want to, as we'll go through more, you need to cut out all possible places where you're going to do a lookup.

So if on, like when you're doing a post to create a new object and you have the ID, and then at that moment you check to say, is there an ID of this already exists on the database, that's always going to be a lookup to the database. You always need a connection to the database in order to do that. So we were like, no,

we'll just wait until we actually succeed and let the database do the constraint check for us. If the database gives a conflict error, then we kick it back out and it's a conflict error and the client can handle the 409. We're a REST API, so they should know how to handle 409s and that's kind of how it's handled.

Great. So another thing that is important that we use a lot and we are really thankful that Postgres can give us, to go fast, is the sequence number. So the transactions ID is quite, it's really important that are sequential and Postgres itself offers us a sequential ID that allows us to have a sequence number for doing that.

There is two things that this architecture makes. There is a lot of things that makes this a bit complex, but two major ones. One is that we lost the history, the ZODB kind of append only that you are storing all the information,

all each transaction, so you can go on time and decide how far you can go. We don't have history, so we have a snapshot of actual situation. We can implement versioning on top of the objects by the data model itself, but the transactions, we support what Postgres support as transactions

and we have the snapshot of the database. Of course, as the previous talk explained, you define a specific snapshot on time on the database, then you can go back if you don't remove the history. Another thing that is difficult is the relation of children and parent. The children pointing to the parent, ordering.

Ordering is really complex because we really need to, you cannot store the order on the children. You need to store the order on the parent or you can store the, and then you need the list, or if you store the order on the children, then you need to index that properly. If you guys are familiar with how this is done in Plone,

ordering is like they're storing a complete list of the order of the folder on the parent. Every single time you add an object or you change the order, it's modifying the parent object. We don't do ordering. The GILTI and the CMS thing does ordering,

and it's just setting an attribute. It's like a fuzzy, pseudo-enforced ordering. You're still only writing to the child, and that's the safest way to do it where you can actually not have huge performance implications with it. Anyways, there's more.

There's a lot of things. Just to see, an image is always, after explaining all these things, an image helps a lot, no? This is an SQL table from Glutina. You see that there are two specific registries,

the DDDDD and the 0000. A 000 is the root object, so it's the first object, and DDDD, it's the trash. When we are deleting something, we are making the pointer of the parent of that element to point to DDDD. Why? Because we discovered that the reference integrity

foreign key deletion trigger, when we were deleting folders with millions of objects, were shutting down the database. We decided to do the deletion on batch operations, so we point the object to the DDDD,

and then there is a batch job that cleans up everything. You will see during some slides that we talk about vacuuming or batch jobs to clean up. We need to do a lot of these things. We cannot do everything in real time because we want to answer fast,

and we want to provide integrity. So if you do a delete of a really large folder in Plone, it might take 30 minutes or something like that because it's doing all of that in one single transaction, making sure that transaction is synchronized across all the other transactions that are happening, and then hopefully it actually works.

So all we do is set this attribute, and all of our cleanup re-indexing, all that stuff is in an async task or in a queue, and that's how we manage doing large operations that affect a large amount of data at a time.

Yeah, some other differences with Rala Storage is that, of course, we have the object ID. It's the Zio ID. I am a romantic guy, so I chose to name it Zio ID for romantic reasons. I understand. The transaction ID, this is the sequential number.

State size is also a romantic name to store the size of the byte. Part, part is a field that you can create an adapter so each object can define in which partition is stored. This is used for what's been explained to us in the talk before,

that Postgres 11 supports partitioning. Postgres 10 also, 11, no? Supports partitioning, so you automatically do a partition. You can partition your database table with multiple sub-tables, and using this attribute to define in which partition you want to store it.

Resource, just to know if it's a main object or it's just an addition. Auth, in case that you are in a rotation. The auth transaction ID is for the cycle of commit, the two-phase commit. Paran ID is the pointer to the parent.

The ID is the real ID, and this is different, for example, from, I think, on Rala Storage, or maybe it's the type, that you have the real ID that appears on the URL. Then the type, the real type from Blutino, that we have. JSON is the serial edition that we want to use

for indexing, in case that we use Postgres catalog, and a state that is the Pico. One second. One other thing about the ZOID is you can see from row three to six, all of the ZOID keys start with 824. That's for, if you're using something like Cockroach,

we have Cockroach DB support as well, in order for it to distribute the data. It does key sorting, and you want to have your data near each other if you're querying it. So if you didn't have this sort of key storing, which allowed Cockroach to put chunks of data on the same node,

it would be having to do the queries between Cockroach DB nodes and synchronize those queries, and it can be really slow. That's the reason why the ZOIDs are structured that way. Yeah, the three digits are the parent first three digits, and each children gets the three digits

from the parent also besides its own ZOID. So everything gets ordered on Cockroach. Yeah, and we already explained it a bit about that. Deletion, it's complex. Synchronize, making sure that you have a clean system, it's complex. And right now we have two systems of auto-vacuuming.

By default, Guillotina tries to create an async job to clean the database, but we also added an option to disable that so you have an external job that does it manually. No? Okay, no caching.

Yeah, so we use Redis to facilitate our caching and then an in-memory cache. I mean, I guess in that respect, it's a lot like the way ZODB caches objects. We just have an LR...

Yeah, I guess I say that next too, but we have an LRU cache that's pretty fast and written in C, and we have cache keys for all operations that are done on the database. And so we set the in-memory cache, and then we use Redis to coordinate invalidations between Guillotina application servers.

And yeah, so that's kind of it. And optionally, it can also push cache out to other application servers as well. Depending on your load, that can be actually a negative performance hit because that can put a lot of pressure on Redis to push out all that data

if you have a lot of writes going on. So how I usually deploy, or how we deploy, I'm sorry, is we just use Redis for invalidating cache across the application servers, and everything else is just straight in-memory cache then, and we don't push out any new values with Redis.

Yeah, so the L2 cache would be a reference to the Redis actually storing cache of the pickles. So I don't use that, but it is supported, and if you don't have a write-heavy system, it can be helpful.

Yeah, so in the design, you always just want to reduce the amount of lookups ever. So everything that you, every operation that you can potentially do on the database should be cacheable, and you shouldn't ever need to be doing any types of operations that require lookups. So like we had mentioned before

when we inserted into the database, it's like it can produce 409s that the client just needs to deal with, and that's because we don't want to have, like we want to optimize for the normal case instead of the edge cases, because that's what most of our requests are, and then the client can have some of the logic

to deal with those edge cases. Yeah, I guess I kind of already said that, yeah. Okay, so one of the cool things that AsyncIO has

is the option to have tasks that run in parallel, in parallel meaning that they are waiting that there is some input-output to be run, because we cannot use more than one CPU. And one of the things that we have is request features. So you have a request and you can add tasks or functions or operations or wherever you want

that is going to be done after the request is finished. So that helps, for example, indexing, or you want to call some out lock or do whatever you want. And while it's so easy that you just get the execute global function of Guillotina and you do after this request, call this function with these arguments,

and it will be called after the request has already been delivered to the client, so that you can really have operations that work after the request has already been finished by the client in AsyncIO. It's like the pre-post.

We also have pre and post commit hooks, so you can, yeah, we are also romantic on that. So you can do whatever you want before a transaction or before a commit or whatever, but they are done before the request has been delivered to the client.

So whatever you do there, it's going to make the request a bit longer. Wherever you add an after request, it's never going to affect the request time. And then we have another interesting thing that we needed to develop to make this kind of AsyncIO jobs work better.

It's a pool and a queue of tasks. That means that we call something Async utilities, also for romantic names. Utilities are single items that are running in memory that does whatever you want. And Gluten out of the box delivers us with a queue

that we can queue tasks there. I don't know. Connect to these servers and say that I'm alive every five minutes. Or call Celery and do whatever. Or call this queue system and do whatever. And so the queue and the pool,

it's used, it's like before. You just get the pool and you say run this. This can run forever and it's never going to block. Well, it needs to be something that it's AsyncIO if it's CPU bound will block all the system, of course. But you can just execute wherever you want there.

Short, long, whatever. And these AsyncIO utilities that we have, we have a lot. We have for caching, for the indexing, for working pools. There is a lot right now.

Yeah, transaction mechanism. As Nathan said before, we don't want to access the database because accessing the database, it's slow. So we are trying to demorate as much as possible accessing the database. So we have a traversal system

that goes through all the objects. If they are in cache, we don't need to access the database. We go through the security. We validate whatever JSON web token you have. If we don't need to access any service or you need to access the service, you access it. And then we look for the view. There is some other things on the traversal.

But you look for the view. We start the transaction. At the moment that we are going to execute the view and then we execute whatever is inside and we close as soon as possible when we finish the view, the rendering. So we tried always to demorate as much as possible

and we also want to try to be as smart as possible with read-only and write transactions. And we are using something that we are really proud. It's the GET, the HTTP verbs. So we have GET, HEAD, PAUSE, DELETE, PATCH.

So why don't we use these verbs to define on advance that the connection is read-only or is write. So what we do is if you do any HEAD or any GET or any HEAD on Glutina, we even don't open writing transaction. We just do a read-only transaction

that we are not going to commit. For sure, you cannot write to the database on any GET or HEAD operation. And we use the PAUSE, the GET, add the PAUSE, the PATCH, and the DELETE and then we open transactions. So that helps us because then you can tune that you have, for example, a slave of Postgres that it's only for read operations

that all the GETs go there and then you avoid overloading the main database. You can do a lot of things if you have this kind of tooling so you can define read operations and write operations. Yeah, another important thing is that we have our concept of what's a transaction

that we are reusing on our tasks and we are using now in 3.7 the context bars to store these transactions. But a transaction in Glutina is not a transaction in that database. So we start a Glutina transaction at the beginning of the request and we end at the end.

But as I said before, we try to make as small as possible the transaction and if we don't need the database we even don't open the transaction. Yeah, finally, in this block of lifecycle management kind of what's happening,

we have an option that is called xdebug if you call Glutina and it's enabled with a parameter that is called xdebug. You get really interesting debug headers where you can see the timing of each of the phases that you have on the traversal

and the performance of the cache, the memory cache and how many queries you are doing too as well. It's a really interesting tool too. So when you finish, okay, this call, where am I spending more time? No?

Oh yeah, we also support, we have packages for Prometheus to send metrics to Prometheus on a statc. So you can deliver all this information to see if there is any problem. Indexing, that's me.

Okay, we will go faster because we are running out of time. So indexing, it's a really complex problem. And if you want to,

we are storing the pickles in the database, so we wanted to make something that is fast but it's also synchronous. So the first approach that we went is to index things in Elasticsearch. What it means is that we have a process that extracts the indexing information and sends to Elasticsearch completely out of the transaction. That makes that the index may be out of sync

with the main database. So you may have objects that are not indexed or information on the index that is not on the object. But it was a really fast first approach. And it fixed it, well maybe we were losing some information on the index, maybe it was late.

For example, now if you try to use Elasticsearch on Guillotina and you want to use Volto, you create an object and you go to the folder and you don't see until one second because Elasticsearch needs one second to index that information. So, yeah, this is synchronous even from a Guillotina point of view,

even from Elasticsearch or whatever you're using to communicate with Elasticsearch. It's a problem. So you need to have jobs that are every some days going through all your data making sure that everything is indexing or re-indexing all this information. We will

and explain it before a good solution for this problem with Kafka, but we will explain something if we have time. Then we decided to create also transaction tie-up indexing so it means that there is as we are storing on JSON beyond Postgres

that means that is on the same transaction and if the transaction fails the information is not stored on the index. It's really goes really fast. It's scalable using JSONB. We have indexes for all the fields that we have used to index. The problem is that the full text search is not as powerful as Elasticsearch

and if your needs require Elasticsearch then you need Elasticsearch. So there is API to initialize the indexing to repopulate the indexing to delete the indexing. It's not only a container a container is what alone should be a site

in Lutina. It's an index. You can define that any folder is a sub-index. You can define multiple sub-index for each folder of your site if you want. And yeah, file storage, storing files it's a bit easier than indexing and we are lucky that there is a lot of services

that provide this kind of backend in order to support storing files. We have S3, Google Cloud Storage and Minio that it's an on-premise S3 that you can install in your Kubernetes cluster. You can also use database

but as we said before in the talk before it's not recommended to have large volumes of blob files on Postgres. Oh yeah, we are building a new a new driver for Lutina to store also on the file system if it's needed. But what's important is that

it's asyncio any file coming in and out of Lutina even because it's coming from the storage to the client or from the client to the storage it's a stream. It means that we are chunking wherever if you are sending all together or with TOS.

We are chunking with buffers of 256 or up to 4 megabytes of bytes and we are sending this information to the service and the other way around. So I remember that in ZOOP maybe a long time ago I needed to download files locally and then return to the client.

It's so easy with asyncio to have this buffering of input-output meanwhile you're doing other things with other tasks on the same thread. Yeah, it's also important to know that if you're using S3 or Google Storage or Minio you need to have a back-keeping in case there is being some un-sync

information between the storage and Lutina. Yeah, I'm going to go really fast. As is asyncio queues are really friendly with asyncio so we support AMQP. Mostly designed when you really need to

ensure that the task is being done so you need to have an ECK this is being done. You have Kafka support that means that you can even log in use sequence for indexing and being able to have a consumer that indexes an elastic search or to also assign tasks more kind of a streaming service

and then you have NATS and STAN that is the best of AMQP and the best of Kafka together. Sorry, I'm using that. And it's yours. Yeah, some other things that we found out in tuning Guiltina is

be careful with reference cycles. So objects that end up referencing through a chain of references of objects themselves. It's really hard to garbage collect for Python to garbage collect reference cycles. It takes a lot of reference or garbage collection attempts before it actually gets it out of memory.

Also in AsyncIO each task is like this object that doesn't get cleaned up immediately and if there's ever some exception even if it's like a conflict error or whatever it might be that exception an exception object

usually wraps the local context of what happened in that exception in it and so that might have references to objects inside of it and that might not be cleaned up right away because that task even though it may have been finished the object that holds the information about that task hasn't been cleaned up. I don't even know the rules behind when I think it's like a

weak ref. I don't know the rules behind when that actually does get cleaned up in AsyncIO but there's like this this weak ref dictionary of tasks or something that doesn't always get cleaned up and that can be a lot of memory sitting around potentially. There's also the profile built in profile in Python

that's really nice that you should use to know profile like CPU intensive code and then there's this really nice library called line profiler and that will give you the full time of every single line of code that you decide to profile and so like you can look at a line

and then that took so many microseconds and then you can go into like what function that called and then you can see it all split into like where inside of that function like you can keep going down and see like where the time was spent and it's actually time not CPU time so this will include if you're like connecting to

a database and that database is slow or whatever you can be really helpful to identify slower parts of your code. And Guiltina comes with built in support for both of them there's the minus minus line profiler command option and also just profile command option

and then is slow maybe I oh yeah so you can kind of see this is the line profiler output but if you want to actually profile a function you have to add that at profilable decorator and then once you

use this it will register that this needs to be profiled and this is the outline profiler output though it tells you line by line how much time was spent on every line. I think that's that's all. Thank you. Any questions? Hi

I noticed you turned off the auto vacuum on PostgreSQL oh ok so you're out of vacuum because it's not safe ok

you answered my question thank you cause when we delete we just mark an object as being deleted and so like there's another application vacuuming process where we're actually going through those and doing going in a shallow way and deleting them

individually instead of relying on referential integrity to clean them all up because referential integrity it's a substantial amount of objects that need to be cleaned up can like take just gigabytes of memory and destroy it

Hi maybe this is a crazy question but how much SQL are you really using in Lilliotina I mean if it is just a little part of SQL for performance you can bring more and more using directly or writing

and reading from files using the posters libraries for example and better would be to use a database built with asyncio directly so it can be plugged in on your application this would get

the best performance but I don't know if it exists so I'm asking if you when you did your research if something like this can happen or exist I guess at the time Postgres was really

by far the best option there was this really nice asyncpg library that's been asyncio that's an asyncio driver for Postgres that's really fast and even when we started this not all databases had asyncio drivers

and Postgres is really fast as long as you use it correctly it's really fast and then if you're going to use a database that spoke HTTP there's overhead with that and they might be able to scale horizontally but then regular operations aren't necessarily really fast and there's no

database that's just asyncio in the end it's still Python it's not going to be as fast as C or Rust or C++ or Go but we use Python we like it and there's other benefits to using Python so this is fast in Python we mean Python fast you know you can still do a lot with Python Instagram uses Python and asyncio

they're pretty big it's just a matter of choices and tradeoffs other question? comment? by the way Instagram also uses Postgres

you use buildout right? no no what do you use then? just pip I saw slash bin in the example slash bin? oh I just a virtual environment nice try

other question? hi sure I agree with the lookup prevention schedule because that takes time but the actual issue there is race conditions because first you select

then you update so we have a transaction ID that we look up to make sure that it's a safe operation so that would work in New York but there could still be another thread or another operation that will insert and then you get a 409 because it will be invalid

and then you get another 409 so you might have to do multiple 409 tricks so the actual client needs to take care of that anyhow yeah retries we have built in retries too for database conflicts just like ZODB has retries it's kind of the same way in that respect

but the client still needs to handle it in some cases