I have one here and maybe only the client for this particular website online, hello. My name is Maurits van Zest, that's already said. I'm going to talk about the MineRAD website. That's a website in Belgium. I've got the link there, MineRAD.pa. Yesterday something was wrong with the DNS, at least from my laptop, so it wasn't visible, but everything should be fine again.

The MineRAD is the Murje Inst Naturad of von Flandren. It's an advisory council on mostly environmental issues for the Flemish government, so for the Dutch speaking part of Belgium. There's been a client that says for, I don't know how long, I guess 14 years,

something like that, I guess. The site started, I'm not sure if that was blown 2.5 or 3.0. It was around that time when we switched to that. It has also been migrated to 3.1, 3.2, et cetera, all the way to 4.3.

It has moved data centers a few times. It has moved to rail storage and back one or two times. A lot has happened over the years. About two years ago we started migrating that to Plan 6. Not all the time. It doesn't take two years full time, of course, to migrate the site.

It was some work. Last week the migration was actually finished. Everything is live and that has worked. There's one mark left that I need to look at again. So talk about that, introduce what the site does. Talk also about in-place migration versus the export import.

It will be a mix of a grand overview and some details if there's time to go into that. The current site, you can click around on that a bit. I have that locally, just to be sure.

That can be a bit bigger. This is the home page. We've got a list of several themes for content. On the left-hand side, some current documents, a gen of the next meeting. This is an older copy, so it's a meeting from July.

A few more content. I think most are with related items and some recent tweets. They should be recent. The same list of themes was a bit bigger and we'll pick one theme.

This is the most important content. You've got great advisories. I think that's the most important content. That's really the advisory that a minister may read, or other people from the government, so that they know what they should be doing.

Some other more basic studies or events that they organize. We go to one advisory. Everything is in Dutch here, of course. There's a summary of what I think is important. Each advisory or most of the content has a coordinator

and one or more co-authors. That's a small content type also on the Plone site. That's just shown here. This is the actual download. There's the actual full advisory text.

There are editors and files on the site. We used to use product simple attachment for that. We've refactored that more to default Plone files and images. Anything else here? There's some more content. The nicest part is the search page.

This is the old one. This is using EEA faceted navigation. This is the only page where that is used, and it's basically hard coded. They could change all kinds of search widgets here. We set it up how they like it, and it never really needs to change. There are a few custom widgets in here

that we made especially for them. That's the public content. They've also got some internal content, such as the Digibip or digital library. A lot of people can log into the website,

or I think about 200. I'm not sure how many people can log in. Some people will be there to create content, and other people can just log into if they are involved in a project. You've got here the grayed out. This is a bit of internal information.

You've got a list of projects on the left-hand side and a list of upcoming meetings on the other side. Mostly, those two are linked. A meeting will be about one or more projects. When you go to a project, you'll also see the list of meetings that involve that project. Products, that's the basis for an advisory.

They start making it over there, and in the end, they publish the advisory. For a meeting, you'll see that here. I've got a list of agenda points that they create there. Every agenda point can have some attachments. One or more, you can move the agenda points around.

They say, okay, this is probably going to take, well, a few here are very short. It will take two minutes, and you can rearrange it in the start time. The estimated start time will be updated as well. There's also one button that says, okay, give me all PDFs from all agenda items.

Make it into one big PDF, which can be 100 pages at times, which they should hopefully not print for the environment. They can just download that to their own laptop so they are prepared for the meeting. Okay, look at page 40.

We're dealing with that now. One other nice thing, they have open registration. I think it's still open. With email login, there's code in core clone to support email login. That's actually created for this client. I've moved it into core clone, I think, in clone 4.2.

They basically paid for that feature that some of you may also use. The core clone, either me or some other people have made sure that this feature that they have supplied has been actually updated to in clone. It also works in clone 5 and clone 6.

They got something out of that, some support from the community. There are a few challenges for migration. There's a lot of custom archetypes, content types. Everything was still archetypes. Also, a lot of custom views for that,

a lot of snippets. There are about three ways to show a theme in different places. You saw the faceted navigation with custom widgets. We need to take over those custom changes and see if that all works. We've got agenda items combining one PDF. Does that still work? That was quite easy, actually.

They have custom user properties, which is done slightly differently in clone 5. A theme still based on Bootstrap 3, where in clone 5 or 6, you have moved to, in clone 6, you've moved to Bootstrap 5. That needed a lot of changes. I won't say much about that

because it's mostly my boss, who did the changes there. There are some dangerous or old persistent utilities in the database. That has caused problems for the migration. The question is then, which clone and Python version do you really go to?

This was in about summer 2020, when we started thinking about this. We could have chosen to go with clone 5.2 on Python 2.7. Then you could have kept the current archetypes, content types. That means much less migration troubles.

That's not really sustainable for too much longer because Python 2.7 is not maintained anymore and it needs some updates. We're using Quinte Group sale optimizer, which did not work yet with clone 5. That would need to be updated. There's a reference browser,

which did not work yet, which was used by fields, which then did not work yet. That's just a few things that you would need to get running on clone 5.2, which if you then move to Python 3 maybe a year later, that work has basically been for nothing. You would polish things on this stage

and which you then throw away. We can then move on to Python 3. Then you don't need to do those things. You still need to migrate a lot of stuff. The Quinte Group sale optimizer, you would need to replace that with Collective Behavior Sale,

which was already existing for dexterity. You have to see, okay, does this match for the migration? They've used Plum Form Gen. Yeah, that needs to be migrated to easy form. The good thing there was that all current forms, there were not many of them, and they could actually be deleted and just downloaded some lists of data that they needed.

All new forms, they really didn't need that much migration. All your custom types, you need to migrate that to dexterity, let the code work on dexterity. That was probably the most work. All your code needs to run on Python 3, of course,

and the database needs to be migrated to Python 3, which is then a separate step. A lot of things to do. It's all needed in the end. That's the question. Do we go on from there to Plum 6 immediately, or do we stay on Plum 5.2 for a while?

At the moment, there was no alpha release there yet, so it could have been a good point to stay on 5.2. It took a long enough time for us to do all the coding. By that point, the alpha release was much closer, and we moved on to Plum 6.

Which migration do you choose? The standard way would be the in-place migration. That's also what you use if you go from Plum 4.3.19 to 4.3.20. You click one button and you wait five seconds in most cases, and it's done. For a major version, that will usually take a long time.

For here, you need to have everything work. You need to use all those steps. From the previous page, let everything work on Plum 5.2 Python 2. Then to Python 3, then Plum 6. You can't skip any steps in there.

The other way could have been Transmogrifier, where you could export the content and import it directly into Plum 5 or Plum 6. That's always difficult to configure. In my experience, it was mostly hard. I did not really try it for this case.

It was not too much of an option. Currently, you would probably say, okay, use collective export imports. If I would do it now again, that's what I would use. When we started in 2020, this did not exist yet. The first commit to that was in February 2021.

At this point, that wasn't an option. We went for the in-place migration. Let me check what I'm doing for time. How do you go about it? First, yeah, we were still using buildout and still doing that, still working fine,

despite what some people think. You upgrade the buildout version, you say, okay, don't use 4.3, use Plum 5.2. Basically, you first get the basis running on Plum 5.2, Python 3, upgrade some versions of Plum, of all kinds of add-ons, or probably remove any version pin specific that you have

and just run buildout and see if it complains about anything. You might get, okay, hey, there's no version of this add-on that works on, that already works. Easiest is then to, okay, just comment out this add-on for the moment and you can later see, okay, do we still need this

or can we find a replacement for it? The same with your custom code. Yeah, if something does not work directly on Python 3, 5.2, yeah, maybe you can just comment out some parts. All the front-end stuff, if the browser views, you don't need that at first. You can worry about that later.

Just make sure that you can start up with Plum 5.2 on Python 3 as a basis. Once that is running, then you can start creating your dexterity content types, create the code for that, and see if you can create them online. Then you have a nice enough basis to say, okay, now we can start seeing, at first, yeah,

does it run also on Python 2 as well, this part of the code? It's usually easy enough to get it running on both versions, at least for the dexterity part. There are automated code tools for that. Then you can test migration. It's better to use a smaller database

instead of a two-gigabyte database. Once that is all running, at that point you can move to Plum 6, and that's really where you then do the polishing. That's where you start working on your theme, implement all your browser views, because in this case, the part on Plum 5.2,

no one is ever going to look at that. All your browser views don't need to work on 5.2. They only need to work on the final one on Plum 6. How do we do that from a version? From a control point of view, and how does it look on my laptop when I'm working on it?

In the Git version control, yeah, that's 7.x, yeah, that was the maintenance branch, because we still needed to do some bug fixes sometimes. We also did a final cleanup for a tree branch where we did a few bits of cleanup, like, okay, remove the old theme already in Plum 4.3,

delete some add-ons or delete some, delete the Plum 4th Gen items, and then the master is the part of the code that needed to run on, yeah, both 5.2 and Plum 6, and also Python 2, Python 3. So I had basically four directories on my laptop, one for 4.3 and three other for,

yeah, all the stages that we were working on. And then really the same on the server, and we'll see a bit about that. Small thing about the code structure, the archetypes, content types, as they were in products.minerad, just something in the build out,

and the newer stuff for the digital library was in a separate minerad.projects. I've decided for all the dexterity content types to create a completely new package, minerad.content, and here every bit of code from the old packages you have basically moved or maybe copied that

to the new code, because you also have the problem, yeah, there's, for example, the advisory content type, yeah, it was called advisory, and in dexterity I also wanted to call that advisory, that makes a lot of things easier, but then it gets tricky if both are

in the same package at the same time. So I've moved everything to a new one, and if everything is correct, then at this point I could actually remove, since the migration has finished, remove the old code. There's a lot of cleanup still that we could do. We had an old theme, and the new theme

was just in a major new way. There's training that you can follow for that training.plone.org. We've basically followed that. How do they do that? And for the search I've did it completely differently based on Svelte. That is also a completely new code, which was better to put in a separate package.

What you really want to do is to automate the migration. Do everything by hand, click everything. A lot of times I've done the migration on the server, at least 10 times, maybe 20 times, could be more. So we've used Fabric for that with a FAB file.

A bit old school, but it works. For each directory in the stage that needs to run, you make a clone of the code, you run your bootstrap and build out, you copy the data from the previous step. You start the site, you run some upgrades,

you stop the site again and you pack the database. That is always the smallest amount of data that you need to copy to the next stage. And that all kinds of old code, old data is really gone. For the binupgrades step part, we're using FTWupgrades for teamwork in Switzerland.

Some details here. I've called this fwarewaycup. I'm calling it commandbinupgrades, install for the site with the ID minirad. In 4.3, we run all proposed upgrade steps. That's all kinds of cleanup. When moving to 5.2,

first removing some persistent import steps to avoid getting very many warnings. I run the default Plone upgrades, which probably takes just a minute or two. And then for the rest of the upgrade,

from archetypes to dexterity, that needed to be done in two stages. In the first stage, install Plone content types, which is the new default content types for Plone. Then you really need a transaction commit for some reason. I've split it in two commands.

In the second command, everything has been committed and you can find the base view that you need for the migration. You migrate from 80 content types. When everything is done, this takes about two hours, three hours on our side of the database. Then you move to Python 3.

That's a binzdbupdate. That's a small package that's used for that. You run some upgrade steps after that again and move to Plone 6 and run more upgrade steps and install the new theme. I can look a bit, clean up on 4.3.

First thing I did was disable caching and also disabled the purging. Otherwise, hey, I've removed something or edited something and sent purge requests to the Varnish server, which gets swamped and warns you about anything that's totally not needed. Using collective fingerprinting for an audit log.

Yeah, that's not needed during migration. I switched that off. Removal image skills. That's not needed. Just bloating the database. Uninstalling some add-ons. Removing tools, utilities, all kinds of persistent stuff that was in the way. There were some weird mail hosts that were still lingering around

from early Plone 3 probably that somehow survived it this far. It should be dead now, but in some cases, you're not sure. Some details that I'll skip. I've got some time. We are using Collective Migration Help

as a package from Filip, which isn't on PyPy yet, so you can just do a git clone and use it. That will change hopefully soon after I've maybe added some more code to that. I've got some handy code in there. Remove overrides. It goes to the portal skins,

the custom folder. Just remove everything there, some overrides that are probably only needed for the older Plone version. You should check beforehand what you have in there, and if you still need it, then just put it in your real code. Also, the point of view customizations does the same there, I think.

Release all web.logs. Remove anything from Plone 4. Then folders. Remove all kinds of uninstalled products, things like that. Remove all revisions. If you create a page and edit it five times, then you have five versions of the page. That's not really useful for migration.

There are ways also with Collective Export Import to keep that if you really want to, but usually don't bother about that. It's just more bloat. For testing, it was a few things that were really handy. You can call delete large items to delete all files and images that are larger than five megabytes,

or I don't know what the limit is. Or you call trim content, which basically removes 90 percent of your content. You just have a smaller database, but guaranteed with all the, at least one instance of all your content types.

Now they're scarier code. In Plone 4.3, I'm removing some persistent utilities. I do the same in 5.2. In some cases, I try to remove some persistent utility five times. I think it kept coming back. For some reason,

these are a few things here. An old discussion tool, some interface which has a utility that's probably a bit too technical, perhaps where they are full. A few things that you will see cannot be there anymore. You can remove utilities. This is a bit scary code to remove utility,

but it's always hard to remove all kinds of things that should no longer be there. On Plone 5.2, you migrate the default content. There's some code to just default Plone folders, pages, et cetera, remove that. That's all from Plone content types that does that.

I've set an environment variable catalog optimization, disabled is one, which prevents some optimizations. The optimizations were in the way somehow for the migration. This helped for me. Otherwise, it didn't finish.

I got all kinds of weird errors that might depend on the order in which you migrate things. It's at least reportedly best to first migrate folders content. Then migrate all the pages that are inside the folders. The other way around, I got some problems which might have been solved

by this environment variable as well. Some options you can give here. For this case, you mostly look at Plone content types and documentation. Filip has done a good job there. For your own content types or maybe some add-ons, you'd need to write custom migration.

Let's not do the details, but that's also our first of the folders. I just made a few small functions. First, migrate the content contact person because it was quite separate. Then migrate the theme. Within the theme, we have events. Then you migrate them.

Then agenda items are within that again. Image attachments and file attachments. This all takes a long time. In the end, the code behind it is not too hard. Mostly, look at Plone content types, migration, documentation. A few more details. At the end, you could call removeArchetypes,

which removes some more utilities. Then archetypes was gone, which is good. Then moving from the database from Python 2 to Python 3. That's a separate step because if you take the you are in Plone 5.2 on Python 2.7, a database that was created there.

If you then say, okay, I've changed all my code to use Python 3. Then you start Plone. Then you get an error because the code can't read the old database. You need to use the CODB update script. If you use build-out, yeah, you would create a build-out part like this. Include the CODB update package.

I've written zest.cdbupdate. This has a few extra definitions that we needed to tell the scripts, okay, if you encounter this kind of class, replace it with this other kind of Python class. A lot of things are already there in Zope or Plone or CDB update itself. The script needs all axl packages from the site.

You point it to a configuration file, which basically just points to your file storage and your blob storage. You call it like that. This was a run from earlier when I did not,

which still created a lot of warnings. If you haven't done any optimizing yet or any fixes, you can see lots of warnings here. Hey, I'm seeing something like a metadata tool, but I don't know. This class does not exist anymore in your code. This will be a problem.

Those are things you need to fix. That was most of the stuff. I think most of these were persistent utilities that lingered there, which weren't needed anymore. If you encounter this, then you may need to do some cleanup in your data.

All kinds of things that are picked up automatically. I think this comes from Plone upgrade, which has an alias from an old tiny MCE class to just treat it as a simple item. That's handled automatically. Another script you can use, UDB verify.

Basically, it's complained about mostly the same things that you have an ID. There are some things to do. In the end, with all the cleanup, I managed to run this without any warnings and errors. If you need to know more about that, Phillip has written a blog post which has the real information.

Your database is migrated. It works on Python 3. There are still some upgrade steps to run. In this case, you cleaned up some skin layers, all the PDFs from the meetings.

They were stored on the meeting themselves. We needed to regenerate them. Could be done on the fly, but it's better here. At this point, it's a good idea to clear and rebuild the catalog. At least, I thought it was needed. It takes a while, but then at least everything is correct again.

There was still some portal languages thing that I think finally died after hitting it with the hammer five times. This should have been the last one. We've disabled caching, so we should switch it on again. The same for the finger pointing. Seems to be almost there, which is good.

The nicest thing probably about the new site. Is this the new site? Wait, I'm now confused. That was the old one, yeah. This is the new one. Checking again. Yeah, this is the correct one, sorry.

This looks kind of the same as what you saw earlier, or just now with the switching of the tabs. Still some widgets, but now everything is built with Svelte. Not really details to go into,

but at least I've built. Okay, you scroll down, you scroll down. New images are being loaded, new search results are being loaded, and more and more and more. That was fun to build. Svelte is quite a nice tool to use, at least for a search page like this.

Yeah, the theme were, as I said, my boss, Jean-Paul Adagie, did that, so I don't really know too much about that. I fixed a few things, but nothing to really discuss here. We are still at Plone 6 Alpha 3. We did not there. This was from the beginning of January.

It should be perfectly possible to upgrade it to Beta 3, but we're a bit scared that it will take yet more time and we've taken enough time already for this year, for the front-end mostly with some JavaScript and theme has had some updates in core Plone,

so that could still be some more work, but it should be reasonably easy to do next year a bit from that. All should be fine. One bug that somehow left the user fields seem empty sometimes. If you somehow triggered, okay, you create a new user and then as an admin,

you go to the user's control panel and then all the fields are empty. You don't have a full name, not an email address. The data is still there. If you restart the website, then it's back again. You see it. That's a bit weird and I need to look at that. Someone already mentioned that for Plone 5.1 a few years ago

and no one has really managed to solve it yet. I'll have to look at that. I'm not sure if those are good, but we'll have to look at that probably in the next weeks. The link to the site, if you still want to see it, minera.ba, my company,

there's a work for software.nl, on the website Marut van Rijspedorg and we've got a few minutes for questions. Are there any?

Two seconds after I finish, my alarm goes off. That's good. Seems that you can ask questions later as well. Yeah.

Not a real question. Do you have the button there? Does that work? Really? Cool. Not a real question. I would much appreciate if you would add these nice helper methods as you already announced. Put that in Migration Helpers and then we can make a release. That would be cool. For those people who do in-place migrations,

this could be really useful. There's some stuff that I haven't thought about, like disable caching. I always do that by hand. It hasn't occurred to me to write a line of code to do that, but it's useful because I've done that so many times already. Why do I do that all the time? Yeah, it's the right approach to write that code.

I can do that, yeah. Any more? Okay, then I thank you and enjoy your next talk.