Data migration to Plone 5.2 and Volto
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 50 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/55205 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2019 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Plone Conference 201931 / 50
5
8
10
11
14
16
17
25
26
27
28
29
32
34
35
36
37
41
42
44
46
49
50
00:00
Point cloudExpert systemInformation technology consultingKolmogorov complexitySystem programmingAbstractionMusical ensembleMultiplication signHuman migrationJSONXMLComputer animationLecture/ConferenceMeeting/Interview
00:32
Plane (geometry)Human migrationGame theorySinguläres IntegralPersonal digital assistantGeneric programmingPattern languageDiagramClient (computing)WebsiteProduct (business)System programmingTerm (mathematics)Function (mathematics)Content (media)Client (computing)Multiplication signContent (media)Object (grammar)CASE <Informatik>MiniDiscView (database)Line (geometry)Revision controlComputer configurationProduct (business)BitPlanningPoint (geometry)Different (Kate Ryan album)Physical systemSource codeForm (programming)Entropie <Informationstheorie>Scripting languageSoftware design patternRight angleTraffic reportingData structureType theoryDecimalSoftware testingInterface (computing)GodWebsiteInstance (computer science)Network topologyHuman migrationAlgebraDatabaseGeneric programmingIterationChainMereologyInternet service providerFront and back endsElectronic mailing listNumberoutputUniverse (mathematics)Profil (magazine)Computer animation
09:21
Content (media)Human migrationPlane (geometry)Utility softwareSoftware testingServer (computing)BuildingComputer wormBlock (periodic table)PortletTerm (mathematics)Type theoryDefault (computer science)Web pageData conversionUniform resource locatorMaxima and minimaWeb pageMedical imagingHuman migrationLink (knot theory)VotingSoftware testingKey (cryptography)Point (geometry)Resolvent formalismData conversionElectronic mailing listServer (computing)Query languageNetwork topologyMathematicsBlock (periodic table)MereologyPortletClient (computing)WebsiteInstance (computer science)Computer fileRepresentational state transferData structureMultiplication signUtility softwareCodeProbability density functionDefault (computer science)Content (media)Covering spaceType theoryRight angleCASE <Informatik>EmailResultantRepetitionSoftware development kitMiniDiscReading (process)Message passingoutputVoltmeterXMLUML
18:09
Plane (geometry)Insertion lossGamma functionBeat (acoustics)OvalPay televisionInstance (computer science)Coma BerenicesIcosahedronRegulärer Ausdruck <Textverarbeitung>1 (number)WritingSoftware testingPoint (geometry)Error messageCodeMultiplication signPlanningNetwork topologyObject-oriented programmingWebsiteContent (media)Integrated development environmentMessage passingDefault (computer science)Source codeComputer animation
21:34
Plane (geometry)Module (mathematics)Data conversionStreaming mediaHuman migrationPresentation of a groupMoment (mathematics)Self-organizationCloningComputer animation
22:37
Plane (geometry)Ring (mathematics)World Wide Web ConsortiumOvalType theoryEquivalence relationCodeMachine codeFlow separationHuman migrationBlock (periodic table)Multiplication signModule (mathematics)Data structureMereologyResultantOverhead (computing)Right angleStreaming mediaPoint (geometry)Graph (mathematics)Reverse engineeringInheritance (object-oriented programming)BitLine (geometry)Computer animation
25:50
Kolmogorov complexitySystem programmingVideoconferencingMusical ensembleLecture/ConferenceMeeting/InterviewJSONXMLComputer animation
Transcript: English(auto-generated)
00:13
Like he said, this is my first conference, and my first time speaking English for people, so
00:21
bear with me, thank you for your attention. And okay, the topic I'm going to talk now is what I worked in the last year. It's about data migration from 2.52 and Volto. So at KitConcept, we have some websites that need to be deployed in the last version,
00:49
and we have some options to migrate. For Plone 4.3 to Plone 5 Plus, normally use collective transmogifier, and to Plone 5.1 to Plone 5.2, we have the option to migrate this ODB. So maybe we
01:11
can migrate from Plone 4.3 to Plone 5.1, then migrate this ODB. What happens here when we migrate is that there is a script. If I'm right, Philip wrote? No? Yeah. And this script basically
01:34
changed the structure of the pickles inside the ODB to work with Python 3.
01:43
So the basic way to deal with it is you run the script in Python 2, in the buildout still in Python 2, then you must not start the instance, then
02:02
you update the buildout to Plone 5.2 and Python 3, run the tests, start the instance. That's how basically it works. Sometimes have problems related with interface, Philip talked about it in his bit, and might need to do something to fix it. Another option
02:33
is to use collective transmogifier also. What? At least for me it looks more cleaner because
02:42
we deal with the final database, don't have any problem related with the payments that don't match. So, okay. This is how the name transmogifier comes. It's about Calvin and Rob's fame.
03:08
Just he did a joke with a cardboard box and call it transmogifier when something goes inside, get out in different way. Okay. Why we use transmogifier? We use transmogifier because
03:33
there are many generic pipelines available for common case, it's easy to use, we have flexibility to deal with different use case, some clients have custom data, and so on.
03:46
And it's a brilliant way to use iterator design pattern. So what I see to understand easily how transmogifier works, it have a chain of pipelines, I made a rough example here, that
04:09
we have some objects inputs, I think like it's numbers, so in the first pipeline we have the source that will give a list of items for each item. It's modified
04:30
inside the pipeline and then it's yield to the next chain. So it's starting the source, like in this example, number one plus five, last three, divide by two, multiply by six, and the
04:47
end is seven, seven dot five, and this is this happen for all the objects. This is the way it works. Of course there are some minor things, but the general idea is this.
05:02
So what we basically make in transmogifier, in my point of view, is like if we make a production line. The object is the product that will go from many pipelines on many hands, chained a little bit, and sent to the next one, and then we get the final object working
05:26
the way we need after migration. The use case, we have three use case, sorry I can't show the name of the place, but it's a large universe client. The first use case,
05:46
just a minute, this client, we have not so big database, many custom packages, the build out looks like a Frankenstein, so yeah.
06:07
The second client, it's a high profile client website, has main data, migration takes usually around four hours to run, have some add-ons. There is one custom report object type that is
06:30
archetypes, and this simple object, we decided to split in ten textured content types difference.
06:47
The other client, this is not the website of the client, the website is internet, and have a third party system that is, we are going to migrate into Plone, so
07:07
we will have to make a new pipeline, a source pipeline, to get this data exported from the system, and import somehow inside the website.
07:20
Okay, so the challenge we have to do here is from Python 2, Plone 3, 5, or 5.1, archetypes or dexterity, old products, sometimes other systems, and in the end we want to get Python 3, Plone 5.2, and Volto. Okay, for our clients, the advantage to use this way
07:52
is that they spare migration from Plone 5 to Plone 6, at least part of it.
08:02
Advantage for Plone solution providers, it's a way to sell clients Plone 2.3 upgrade, like Timo said, we have a new front-end, Volto, and it's easier to sell both, it's the right time to do it, so this is our advice, because to sell to a client Python 3,
08:28
it's costly, but the client would not understand the difficulty about it, so it's the right time to try to make this.
08:43
So at KitConcept, we have our way to deal with it, we create some packages, three packages, the migrator package is the principal package, oh sorry, the migration Plone 5 is the principal
09:08
package, inside him there is the KitConcept migrator, and the content creator. The content creator basically is used to create content in the Plone 4.3, we use a JSON structure
09:29
and create some data for testing, and when we run the migrator, in the end we get the result
09:40
and test with a smoke test. Inside the KitConcept migrator package, we have the pipelines, the custom pipelines we wrote, and inside the migration Plone 5, we have a commander
10:00
utility that we say hey command, export data, and then import to Plone 5.1, import to Plone 5.2, and after that we can run the tests. In the end, we have a CI Jenkins node, and
10:21
there the steps are, we build the commander tool, there is a conversion tool in nodes that we also build, Plone 4.3, Plone 5.1, Plone 5.2. Plone 5.1 and 5.2 are empty in this time, Plone 4.3, we have some custom data created for test proposals, we export from Plone 4.3
10:47
and import in Plone 5.1, 5.2, and run the tests, that's how it works, our Jenkins. And also we have a migration server, this one is the one that, well, now takes around
11:07
four hours, this time when I took this screenshot was two hours, almost three, and basically this Jenkins node, we run it when I push the button, not automatically,
11:25
when there is a comment in the master, basically this, so we are working on it to finish all the needs, and that's how we test. In the end, we have a website that we can show
11:43
to the client and so on. Details how we did things. From, to migrate 80 topics to dexterity collection, we use it, we prefer to
12:03
to do this, the changes needed inside the collective G25 package, we have a fork, and basically what we do there is, oh, there is 80 topics, we run the Plone default migration
12:23
on the query to get the new structure of the query, and we prepare the export data to look like dexterity, so during the import the data is already ready to become a collection.
12:45
For, okay, the next topic. To migrate the read text HTML to volt blocks, we created a package in node that we call the utility, pass the HTML, it convert to
13:11
draft JS, and then we get this JSON input inside the volt block.
13:20
For portlets, we did some code to migrate the portlets, at least now we don't use the portlet in the end, but we have in one client, there is one portlet with download that people link a PDF or something like this, and we need to keep this link to use
13:48
at the other part, but, well, the portlet itself is not used, we just take the the link and try to convert to UID or something and keep it to use in the new website.
14:03
And post-migration. Post-migration is something like this, for instance, collective cover. If our client had collective cover, we mark this content type to not be imported, and after that,
14:26
we have an empty document, and we have a structure in our code to say, oh, hey, in this path, we want to overwrite the content with this JSON, so we export the JSON,
14:46
we do the changes online in the website, in the volt, use REST API to export the data, the JSON data, and then save it in the right structure, something like
15:06
Jbot does to import after migration. Okay, I already talked something about volt part, what we need in our use case and is in our code. We use collective folders types,
15:30
because volt don't have the the idea of default pages, so to keep the same page structure, we have some code that deals with this to the end, the end URL
15:47
be almost the same. Need to deal with default page, if there is a default page, sometimes I need to fix how to show it, and okay,
16:12
some content we didn't import, so we create default HTML that just points to the old website so the client can check it,
16:23
and see, oh, this cover page, we want to make it different, so just to be able to check it with the client easily. URLs, well, inside the ref.js HTML, sometimes we have images or links that have resolved
16:47
UID, today the volt don't deal with it, so right now it just pointed at the image, no resolve UID, and we plan to fix it in volt also, so probably will be something
17:08
that you will use the same idea of resolve UID but with REST API and volt. Folders, for simple folders, we just show, we plan to show just a list of the items
17:31
in the vote in the folder, so it's a simple document with the, oh, now it's listing block, not collection, sorry, and the listing block showing what's inside the folder for collection
17:45
is a document with listing block with the query of the collection, so that's the steps we did. Well, I have some time, so I'm gonna show you, okay, why it's not there, just a minute.
18:09
This is how we work today, so we have three plans here, plan 4.3, 5.1, 5.2, when we run the build out in plan 4.3, we already
18:27
create automatically some content, and we wipe the data in plan 5.1, 5.2, so if I run the smoke test here, we'll get error, so make test plan 5.1, here we'll not have the content type,
18:51
I still didn't migrate, just a minute, and after migrate, we get the test passing.
19:14
We plan to open those code to the community, but it will get some time to polish it before.
19:29
Okay, error, then I run, make, okay, import, just import, and then it
19:42
fires a plan, website, and run the import pipelines, it's creating the data that was exported by plan 4.3, I forget to show the export comment,
20:03
the export comment just create the data inside the json data, okay, let's see one of them, then the data is exported, collective
20:31
JSONify, the default way we do with it,
20:42
and okay, I already run the import, and if I run again the tests, it will pass, it will take some time again, and we does the same at both plan 5.1 and 5.2,
21:19
that's how it works, that's the environment I'm working in the last year,
21:27
maybe I've finished too late, you have questions, sorry, thank you,
21:47
thank you so much, questions, thank you very much for your presentation,
22:07
so now we have like four or five different ways at the moment by many organizations worldwide using clone build to migrate it, and one of your modules in here which will be very interesting for
22:23
everybody will be the rich text to volto conversion, do you think it will be feasible to, for the things that you've done new now in your migration stream to extract those, for example this as a separate module that other people can start to reuse in their stream?
22:44
The code is really simple, this is the code, just 38 lines, I'm using the same structure that is inside the volto code, just this part I extracted instead of
23:09
import and use because I had some problem but still it was really simple part, and when I run it I just pass the atml and get the ref.js code,
23:26
super simple, and maybe this code should be inside the next upgrades, upgrades type of clone, something like this, so when we migrate from clone 5.2 to clone 6
23:43
should use something like this, okay, other question? I think I can show this to people, okay,
24:06
so is there a python equivalent to that because for the migration from 5.2 to 6 we need to iterate over all the dexterity schemata and put that into separate blocks
24:22
into a draft.js thingy and these usually run in python. I tried to do it just with python but today it's not possible, it's difficult because the code we have in python for draft.js is just
24:41
to do the the other part, the reverse, draft.js to atml but not the atml to draft.js, so that's why we created T-elite and we fire with subprocess and get the result,
25:01
and because it's running just one time I think it's not too much overhead, okay, another question, another one, here, I don't know if someone don't take the picture,
25:24
so I can show the first part, right, okay, another one, uh-oh, no, just saying, I mean we have a sprint at the end of the week, right,
25:44
so if somebody's like interested in that, that could also be a sprint topic, like start working on like upgrade steps and migrations, it's a bit early but why not if people are interested, nice, okay, thank you.