Data migration to Plone 5.2 and Volto
Ferreira de Souza, Rodrigo
Plone evolved! Now it's time to talk about how to evolve our customer's websites together! After working last year with Plone database migration, during this talk will share the knowledge acquired with the community! What are the concepts involved? How to get ready for Plone 6? At KitConcept we already have some use cases and our way to deal with this!
 Well, like he said, this is my first conference and my first time speaking English for people, so bear with me. Thank you for your attention. Okay, the topic I'm going to talk now is what I work in the last year. It's about data migration from 2.5.2 and Volto. So at concept, we have some websites that need to be deployed in the last version, and we have some options to migrate. For plan 4.3 to plan 5 plus, normally use collective transmogifier, and for plan 5.1 to plan 5.2, we have the option to migrate this ODB. So maybe we can migrate from plan 4.3 to plan 5.1, then migrate this ODB. What happens here when we migrate is that there is a script. If I'm right, Philipp wrote, no? Yeah. And this script basically changed the structure of the pickles inside the ODB to work with Python 3. So the basic way to do with it is you run the script in Python 2, in the build out, still in Python 2, you must not start the instance, then you update the build out to plan 5.2 and Python 3, run the test, start the instance. That's how basically it works. Sometimes have problems related with interface. Philipp talked about it in his speed. And might need to do something to fix it. Another option is to use collective transmogifier also. But at least for me, it looks more cleaner because we deal with the final database don't have any problem related with the penance that don't match. So okay. This is how the name transmogifier comes. It's about calving the enrolment frame. Just he did a joke with cardboard box and call it transmogifier when something goes inside, get out in different way. Okay. Why we use transmogifier? We use transmogifier because there are many generic pipelines available for common case. It's easy to use. We have flexibility to deal with different use case. Some clients have custom data and so on. And it's a brilliant way to use iterator design pattern. So what I see to understand easily how transmogifier works, it have a chain of pipelines. I made a rough example here that we have some objects inputs. I think like it's numbers. So in the first pipeline we have the source that will give a list of items for each item. It's modified inside the pipeline and then it's used to the next chain. So it's starting the source like in this example. Number one plus five, last three, divide by two, multiply by six and the end is seven dot five. And this is happening for all the objects. This is the way it works. Of course there are some minor things but the general idea is this. So what we basically make in transmogifier, my point of view is like if we make a production line, the object is the product that will go from many pipelines on many hands, change it a little bit and send to the next one and then we get the final object working the way we need after migration. The use case, we have three use case. I'm sorry I can't show the name of the place but it's a large universe client. The first use case, just a minute. This client, we have not so big database, many custom packages. The build out looks like a Frankstein. So yeah. The second client, it's a high profile client website. Have many data, migration takes usually around four hours to run. Have some add-ons. There is one custom report object type that is archetypes and this simple object we decided to split in 10 dexterity content types, different. The other client, this is not the website of the client. The website is an internet and have a third party system that is, we are going to migrate into Plon. So we will have to make a new pipeline, source pipeline to get this data exported from the system and import somehow inside the website. Okay. So the challenge we have to do here is from Python 2, Plon 3, 5 or 5.1, archetypes are dexterity, old products, sometimes other systems and in the end we want to get Python 3, Plon 5.2 and Volto. For clients, the advantage to use this way is that they spare migration from Plon 5 to Plon 6, at least part of it. Advantage for Plon solution providers, it's a way to sell clients. We upgrade, like Tim said, we have a new front end, Volto, and it's easier to sell both. It's the right time to do it. So this is our advice. Because to sell to a client Python 3, it's costly, but the client would not understand the difficulty about it. So it's the right time to try to make this. So at KIT Concept, we have our way to deal with it. We create some packages, three packages. The migrator package is the principal package. Oh, sorry. Migration Plon 5 is the principal package. Inside here, there's the KIT Concept migrator and the content creator. The content creator basically is used to create content in the Plon 4.3. We use a JSON structure and create some data for testing. When we run the migrator, in the end, we get the result and test with a smoke test. Inside the KIT Concept migrator package, we have the pipelines, the custom pipelines we wrote, and inside the migration Plon 5, we have a commander, utility, that we say, hey, command, export data, and then import to Plon 5.1, import to Plon 5.2, and after that, we can run the tests. In the end, we have a CI, Jenkins node, and the steps are we build the commander to, there is a conversion to in nodes that we also build. Plon 4.3, Plon 5.1, Plon 5.2. Plon 5.1 and 5.2 are empty in this time. Plon 4.3, we have some custom data created for test purpose. We export from Plon 4.3 and import in Plon 5.1, 5.2, and run the tests. That's how it works, our Jenkins. And also, we have a migration server. This one is the one that, well, now takes around four hours, this time when I took this screenshot was two hours, almost three. And basically, this Jenkins node, we run it when I push the button, not automatically when there is a command in the master. Basically this. So we are working on it to finish all the needs. And that's how we test. In the end, we have a website that we can show to the client and so on. Details how we did things. To migrate 80 topics to the Xteria collection, we use it. We prefer to do the changes needed inside the collective J25 package. We have a fork. And basically what we do there is, oh, there is 80 topics. We run the Plon default migration on the query to get the new structure of the query. And we prepare the export data to look like the Xteria. So during the import, the data is already ready to become a collection. For, okay, the next topic. To migrate the read text HTML to Vultublocks, we created a package in Node that we called the utility, passed the HTML. We convert to Drive.js and then we get this JSON and put inside the Vultublock. For the portlets, we did some code to migrate the portlets. At least now we don't use the portlet in the end. But we have in one client, there is one portlet with download. There that people link a PDF or something like this. And we need to keep this link to use at the other part. But, well, the portlet itself is not used. We just take the link and try to convert to UID or something and keep it to use in the new website. And post migration. It's something like this, for instance, collective cover. If our client had collective cover, we mark this content type to don't be imported. And after that, we have an empty document. And we have a structure in our code to say, hey, in this path, we want to overwrite the content with this JSON. So we export the JSON. We do the chance online in the website, in the Vultublock. Use REST API to export the JSON data and then save it in the right structure. Something like JBot does to import after migration. Okay. I already talked about something about Vultublock. What we need in our use case and is in our code. We use collective folder types because Vultublock don't have the idea of default pages. So to keep the same page structure, we have some code that deal with this to the end URL be almost the same. Need to deal with default pages. There is a default page sometimes I need to fix how to show it. And okay. Rich text to HTML from draft.js. Then we'll do a cheat it. Some content we didn't import. So we create the default HTML that just point to the old website so the client can check it and see how. This cover page, we want to make it different. So just to be able to check it with the client easily. URLs. Well, inside the draft.js, sometimes we have images or links that have to resolve UID. Today, the Vultublock don't deal with it. So right now, it just pointed at the image, no resolve UID. And we plan to fix it in Vultublock also. So probably will be something that you will use the same idea of resolve UID but with REST API and Vultublock. Folders. For simple folders, we just show, we plan to show just a list of the items in the folder. So it's a simple document with the, oh, now it's listing block, not collection, sorry. The listing block, showing what's inside the folder for collection is a document with listing block with the query of the collection. So that's the steps we did. Well, I have some time so I'm going to show you. Okay, why it's not there, just a minute. This is how we work today. So we have three plans here, plan for 3, 5.2. Then run the build out in plan for 3, we already created automatically some content and we wipe the data in plan 5.1, 5.2. So if we run the test, the smoke test here, we'll get error. So make test plan 5.1. Here we'll not have the content type, it still didn't migrate. Just a minute. And after migrate, we get the test passing. We plan to open those codes to the community, but it will get some time to polish it before. Okay, error. Then I run make, okay import, just import. And then it fire a plan website and run the import pipelines. It's creating the data that was exported by plan for 3. I forget to show the export comment. The export comment just create the data inside the JSON data. Okay, let's see one of them. Oops. And the data is exported, co-active JSONify, the default way we do with it. Oops. And okay, I already run the import and if I run again the tests, it will pass. It will take some time again. And we do the same at plan 5.1.5.2. That's how it works. It's the environment I'm working in the last year. Make me, I finished too late. You have questions? Oops. Sorry. Thank you. Thank you so much. Questions? I don't think it, thank you very much for your presentation. So now we have like four or five different ways at the moment by many organizations worldwide using plan build to migrate it. And one of your modules in here which will be very interesting for everybody will be the rich text to Volto conversion. Do you think it will be feasible to for the things that you've done new now in your migration stream to extract those, for example, this as a separate module that other people can start to reuse in their stream? The code is really simple. This is the code. Just 38 lines. I'm used the same structure of there is inside the Volto code. Just this part I extracted instead of import and use because I had some problem but still it was really simple part. And when I run it, I just pass the HTML and get the ref.js code. Super simple. But maybe this code should be inside the next upgrades, upgrades that are planned, something like this. So when we migrate from plan 5.2 to plan 6, should use something like this. Okay. I think I can show this to people. So is there a Python equivalent to that because for the migration from plan 5.2 to 6, we need to iterate over all the dexterity schemata and put that into separate blocks into a draft.js thingy and these usually run in Python? I tried to do it just with Python but it was today it's not possible. It's difficult because the code we have in Python for ref.js is just to do the other part, the reverse. Draft.js to HTML but not the HTML to draft.js. So that's why we created Telet and we fired with subprocess and got the result. Because it's running just one time, I think it's not too much overhead. Okay. Another question? Another one? Let me put this again here. I don't know if someone don't take the picture. I can show the first part. Right? Okay. Another one? Uh-oh. No, just saying. We have a sprint at the end of the week, right? So if somebody is interested in that, that could also be a sprint topic. Start working on upgrade steps and migrations. It's a bit early but why not if people are interested? Nice. Okay. Thank you. Thank you.
 Well, like he said, this is my first conference and my first time speaking English for people,  so bear with me.  Thank you for your attention.  Okay, the topic I'm going to talk now is what I work in the last year.  It's about data migration from 2.5.2 and Volto.  So at concept, we have some websites that need to be deployed in the last version, and we  have some options to migrate.  For plan 4.3 to plan 5 plus, normally use collective transmogifier, and for plan 5.1  to plan 5.2, we have the option to migrate this ODB.  So maybe we can migrate from plan 4.3 to plan 5.1, then migrate this ODB.  What happens here when we migrate is that there is a script.  If I'm right, Philipp wrote, no?  Yeah.  And this script basically changed the structure of the pickles inside the ODB to work with  Python 3.  So the basic way to do with it is you run the script in Python 2, in the build out,  still in Python 2, you must not start the instance, then you update the build out to  plan 5.2 and Python 3, run the test, start the instance.  That's how basically it works.  Sometimes have problems related with interface.  Philipp talked about it in his speed.  And might need to do something to fix it.  Another option is to use collective transmogifier also.  But at least for me, it looks more cleaner because we deal with the final database don't  have any problem related with the penance that don't match.  So okay.  This is how the name transmogifier comes.  It's about calving the enrolment frame.  Just he did a joke with cardboard box and call it transmogifier when something goes inside,  get out in different way.  Okay.  Why we use transmogifier?  We use transmogifier because there are many generic pipelines available for common case.  It's easy to use.  We have flexibility to deal with different use case.  Some clients have custom data and so on.  And it's a brilliant way to use iterator design pattern.  So what I see to understand easily how transmogifier works, it have a chain of pipelines.  I made a rough example here that we have some objects inputs.  I think like it's numbers.  So in the first pipeline we have the source that will give a list of items for each item.  It's modified inside the pipeline and then it's used to the next chain.  So it's starting the source like in this example.  Number one plus five, last three, divide by two, multiply by six and the end is seven  dot five.  And this is happening for all the objects.  This is the way it works.  Of course there are some minor things but the general idea is this.  So what we basically make in transmogifier, my point of view is like if we make a production  line, the object is the product that will go from many pipelines on many hands, change  it a little bit and send to the next one and then we get the final object working the way  we need after migration.  The use case, we have three use case.  I'm sorry I can't show the name of the place but it's a large universe client.  The first use case, just a minute.  This client, we have not so big database, many custom packages.  The build out looks like a Frankstein.  So yeah.  The second client, it's a high profile client website.  Have many data, migration takes usually around four hours to run.  Have some add-ons.  There is one custom report object type that is archetypes and this simple object we decided  to split in 10 dexterity content types, different.  The other client, this is not the website of the client.  The website is an internet and have a third party system that is, we are going to migrate  into Plon.  So we will have to make a new pipeline, source pipeline to get this data exported from the  system and import somehow inside the website.  Okay.  So the challenge we have to do here is from Python 2, Plon 3, 5 or 5.1, archetypes are  dexterity, old products, sometimes other systems and in the end we want to get Python  3, Plon 5.2 and Volto.  For clients, the advantage to use this way is that they spare migration from Plon 5 to  Plon 6, at least part of it.  Advantage for Plon solution providers, it's a way to sell clients.  We upgrade, like Tim said, we have a new front end, Volto, and it's easier to sell both.  It's the right time to do it.  So this is our advice.  Because to sell to a client Python 3, it's costly, but the client would not understand  the difficulty about it.  So it's the right time to try to make this.  So at KIT Concept, we have our way to deal with it.  We create some packages, three packages.  The migrator package is the principal package.  Oh, sorry.  Migration Plon 5 is the principal package.  Inside here, there's the KIT Concept migrator and the content creator.  The content creator basically is used to create content in the Plon 4.3.  We use a JSON structure and create some data for testing.  When we run the migrator, in the end, we get the result and test with a smoke test.  Inside the KIT Concept migrator package, we have the pipelines, the custom pipelines we  wrote, and inside the migration Plon 5, we have a commander, utility, that we say, hey,  command, export data, and then import to Plon 5.1, import to Plon 5.2, and after that, we  can run the tests.  In the end, we have a CI, Jenkins node, and the steps are we build the commander to, there  is a conversion to in nodes that we also build.  Plon 4.3, Plon 5.1, Plon 5.2.  Plon 5.1 and 5.2 are empty in this time.  Plon 4.3, we have some custom data created for test purpose.  We export from Plon 4.3 and import in Plon 5.1, 5.2, and run the tests.  That's how it works, our Jenkins.  And also, we have a migration server.  This one is the one that, well, now takes around four hours, this time when I took this screenshot  was two hours, almost three.  And basically, this Jenkins node, we run it when I push the button, not automatically  when there is a command in the master.  Basically this.  So we are working on it to finish all the needs.  And that's how we test.  In the end, we have a website that we can show to the client and so on.  Details how we did things.  To migrate 80 topics to the Xteria collection, we use it.  We prefer to do the changes needed inside the collective J25 package.  We have a fork.  And basically what we do there is, oh, there is 80 topics.  We run the Plon default migration on the query to get the new structure of the query.  And we prepare the export data to look like the Xteria.  So during the import, the data is already ready to become a collection.  For, okay, the next topic.  To migrate the read text HTML to Vultublocks, we created a package in Node that we called  the utility, passed the HTML.  We convert to Drive.js and then we get this JSON and put inside the Vultublock.  For the portlets, we did some code to migrate the portlets.  At least now we don't use the portlet in the end.  But we have in one client, there is one portlet with download.  There that people link a PDF or something like this.  And we need to keep this link to use at the other part.  But, well, the portlet itself is not used.  We just take the link and try to convert to UID or something and keep it to use in the  new website.  And post migration.  It's something like this, for instance, collective cover.  If our client had collective cover, we mark this content type to don't be imported.  And after that, we have an empty document.  And we have a structure in our code to say, hey, in this path, we want to overwrite the  content with this JSON.  So we export the JSON.  We do the chance online in the website, in the Vultublock.  Use REST API to export the JSON data and then save it in the right structure.  Something like JBot does to import after migration.  Okay.  I already talked about something about Vultublock.  What we need in our use case and is in our code.  We use collective folder types because Vultublock don't have the idea of default pages.  So to keep the same page structure, we have some code that deal with this to the end URL  be almost the same.  Need to deal with default pages.  There is a default page sometimes I need to fix how to show it.  And okay.  Rich text to HTML from draft.js.  Then we'll do a cheat it.  Some content we didn't import.  So we create the default HTML that just point to the old website so the client can check  it and see how.  This cover page, we want to make it different.  So just to be able to check it with the client easily.  URLs.  Well, inside the draft.js, sometimes we have images or links that have to resolve UID.  Today, the Vultublock don't deal with it.  So right now, it just pointed at the image, no resolve UID.  And we plan to fix it in Vultublock also.  So probably will be something that you will use the same idea of resolve UID but with  REST API and Vultublock.  Folders.  For simple folders, we just show, we plan to show just a list of the items in the folder.  So it's a simple document with the, oh, now it's listing block, not collection, sorry.  The listing block, showing what's inside the folder for collection is a document with  listing block with the query of the collection.  So that's the steps we did.  Well, I have some time so I'm going to show you.  Okay, why it's not there, just a minute.  This is how we work today.  So we have three plans here, plan for 3, 5.2.  Then run the build out in plan for 3, we already created automatically some content and we  wipe the data in plan 5.1, 5.2.  So if we run the test, the smoke test here, we'll get error.  So make test plan 5.1.  Here we'll not have the content type, it still didn't migrate.  Just a minute.  And  after migrate, we get the test passing.  We plan to open those codes to the community, but it will get some time to polish it before.  Okay, error.  Then I run make, okay import, just import.  And then it fire a plan website and run the import pipelines.  It's creating the data that was exported by plan for 3.  I forget to show the export comment.  The export comment just create the data inside the JSON data.  Okay, let's see one of them.  Oops.  And the data is exported, co-active JSONify, the default way we do with it.  Oops.  And okay, I already run the import and if I run again the tests, it will pass.  It will take some time again.  And we do the same at plan 5.1.5.2.  That's how it works.  It's the environment I'm working in the last year.  Make me, I finished too late.  You have questions?  Oops.  Sorry.  Thank you.  Thank you so much.  Questions?  I don't think it, thank you very much for your presentation.  So now we have like four or five different ways at the moment by many organizations worldwide  using plan build to migrate it.  And one of your modules in here which will be very interesting for everybody will be  the rich text to Volto conversion.  Do you think it will be feasible to for the things that you've done new now in your migration  stream to extract those, for example, this as a separate module that other people can  start to reuse in their stream?  The code is really simple.  This is the code.  Just 38 lines.  I'm used the same structure of there is inside the Volto code.  Just this part I extracted instead of import and use because I had some problem but still  it was really simple part.  And when I run it, I just pass the HTML and get the ref.js code.  Super simple.  But maybe this code should be inside the next upgrades, upgrades that are planned, something  like this.  So when we migrate from plan 5.2 to plan 6, should use something like this.  Okay.  I think I can show this to people.  So is there a Python equivalent to that because for the migration from plan 5.2 to 6, we need  to iterate over all the dexterity schemata and put that into separate blocks into a draft.js  thingy and these usually run in Python?  I tried to do it just with Python but it was today it's not possible.  It's difficult because the code we have in Python for ref.js is just to do the other  part, the reverse.  Draft.js to HTML but not the HTML to draft.js.  So that's why we created Telet and we fired with subprocess and got the result.  Because it's running just one time, I think it's not too much overhead.  Okay.  Another question?  Another one?  Let me put this again here.  I don't know if someone don't take the picture.  I can show the first part.  Right?  Okay.  Another one?  Uh-oh.  No, just saying.  We have a sprint at the end of the week, right?  So if somebody is interested in that, that could also be a sprint topic.  Start working on upgrade steps and migrations.  It's a bit early but why not if people are interested?  Nice.  Okay.  Thank you.  Thank you.
Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_0_47.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_48_95.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_96_155.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_156_215.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_216_263.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_264_311.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_312_359.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_360_407.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_408_467.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_468_599.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_600_647.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_648_743.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_744_803.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_804_1031.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_1032_8231.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_8232_11015.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_11016_27239.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_27240_31691.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_31692_32039.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_32040_32327.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_32328_33923.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_33924_38747.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_38748_38843.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_38844_38891.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_38892_38963.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_38964_39035.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_39036_39083.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_39084_39143.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_39144_39215.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_39216_39275.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_39276_39323.png Rodrigo_Ferreira_de_Souza_-_Data_migration_to_Plon_kb9SEsnllqE_39324_39426.png
Well, like he said, this is my first conference and my first time speaking English for people, so bear with me. Thank you for your attention. Okay, the topic I'm going to talk now is what I work in the last year. It's about data migration from 2.5.2 and Volto. So at concept, we have some websites that need to be deployed in the last version, and we have some options to migrate. For plan 4.3 to plan 5 plus, normally use collective transmogifier, and for plan 5.1 to plan 5.2, we have the option to migrate this ODB. So maybe we can migrate from plan 4.3 to plan 5.1, then migrate this ODB. What happens here when we migrate is that there is a script. If I'm right, Philipp wrote, no? Yeah. And this script basically changed the structure of the pickles inside the ODB to work with Python 3. So the basic way to do with it is you run the script in Python 2, in the build out, still in Python 2, you must not start the instance, then you update the build out to plan 5.2 and Python 3, run the test, start the instance. That's how basically it works. Sometimes have problems related with interface. Philipp talked about it in his speed. And might need to do something to fix it. Another option is to use collective transmogifier also. But at least for me, it looks more cleaner because we deal with the final database don't have any problem related with the penance that don't match. So okay. This is how the name transmogifier comes. It's about calving the enrolment frame. Just he did a joke with cardboard box and call it transmogifier when something goes inside, get out in different way. Okay. Why we use transmogifier? We use transmogifier because there are many generic pipelines available for common case. It's easy to use. We have flexibility to deal with different use case. Some clients have custom data and so on. And it's a brilliant way to use iterator design pattern. So what I see to understand easily how transmogifier works, it have a chain of pipelines. I made a rough example here that we have some objects inputs. I think like it's numbers. So in the first pipeline we have the source that will give a list of items for each item. It's modified inside the pipeline and then it's used to the next chain. So it's starting the source like in this example. Number one plus five, last three, divide by two, multiply by six and the end is seven dot five. And this is happening for all the objects. This is the way it works. Of course there are some minor things but the general idea is this. So what we basically make in transmogifier, my point of view is like if we make a production line, the object is the product that will go from many pipelines on many hands, change it a little bit and send to the next one and then we get the final object working the way we need after migration. The use case, we have three use case. I'm sorry I can't show the name of the place but it's a large universe client. The first use case, just a minute. This client, we have not so big database, many custom packages. The build out looks like a Frankstein. So yeah. The second client, it's a high profile client website. Have many data, migration takes usually around four hours to run. Have some add-ons. There is one custom report object type that is archetypes and this simple object we decided to split in 10 dexterity content types, different. The other client, this is not the website of the client. The website is an internet and have a third party system that is, we are going to migrate into Plon. So we will have to make a new pipeline, source pipeline to get this data exported from the system and import somehow inside the website. Okay. So the challenge we have to do here is from Python 2, Plon 3, 5 or 5.1, archetypes are dexterity, old products, sometimes other systems and in the end we want to get Python 3, Plon 5.2 and Volto. For clients, the advantage to use this way is that they spare migration from Plon 5 to Plon 6, at least part of it. Advantage for Plon solution providers, it's a way to sell clients. We upgrade, like Tim said, we have a new front end, Volto, and it's easier to sell both. It's the right time to do it. So this is our advice. Because to sell to a client Python 3, it's costly, but the client would not understand the difficulty about it. So it's the right time to try to make this. So at KIT Concept, we have our way to deal with it. We create some packages, three packages. The migrator package is the principal package. Oh, sorry. Migration Plon 5 is the principal package. Inside here, there's the KIT Concept migrator and the content creator. The content creator basically is used to create content in the Plon 4.3. We use a JSON structure and create some data for testing. When we run the migrator, in the end, we get the result and test with a smoke test. Inside the KIT Concept migrator package, we have the pipelines, the custom pipelines we wrote, and inside the migration Plon 5, we have a commander, utility, that we say, hey, command, export data, and then import to Plon 5.1, import to Plon 5.2, and after that, we can run the tests. In the end, we have a CI, Jenkins node, and the steps are we build the commander to, there is a conversion to in nodes that we also build. Plon 4.3, Plon 5.1, Plon 5.2. Plon 5.1 and 5.2 are empty in this time. Plon 4.3, we have some custom data created for test purpose. We export from Plon 4.3 and import in Plon 5.1, 5.2, and run the tests. That's how it works, our Jenkins. And also, we have a migration server. This one is the one that, well, now takes around four hours, this time when I took this screenshot was two hours, almost three. And basically, this Jenkins node, we run it when I push the button, not automatically when there is a command in the master. Basically this. So we are working on it to finish all the needs. And that's how we test. In the end, we have a website that we can show to the client and so on. Details how we did things. To migrate 80 topics to the Xteria collection, we use it. We prefer to do the changes needed inside the collective J25 package. We have a fork. And basically what we do there is, oh, there is 80 topics. We run the Plon default migration on the query to get the new structure of the query. And we prepare the export data to look like the Xteria. So during the import, the data is already ready to become a collection. For, okay, the next topic. To migrate the read text HTML to Vultublocks, we created a package in Node that we called the utility, passed the HTML. We convert to Drive.js and then we get this JSON and put inside the Vultublock. For the portlets, we did some code to migrate the portlets. At least now we don't use the portlet in the end. But we have in one client, there is one portlet with download. There that people link a PDF or something like this. And we need to keep this link to use at the other part. But, well, the portlet itself is not used. We just take the link and try to convert to UID or something and keep it to use in the new website. And post migration. It's something like this, for instance, collective cover. If our client had collective cover, we mark this content type to don't be imported. And after that, we have an empty document. And we have a structure in our code to say, hey, in this path, we want to overwrite the content with this JSON. So we export the JSON. We do the chance online in the website, in the Vultublock. Use REST API to export the JSON data and then save it in the right structure. Something like JBot does to import after migration. Okay. I already talked about something about Vultublock. What we need in our use case and is in our code. We use collective folder types because Vultublock don't have the idea of default pages. So to keep the same page structure, we have some code that deal with this to the end URL be almost the same. Need to deal with default pages. There is a default page sometimes I need to fix how to show it. And okay. Rich text to HTML from draft.js. Then we'll do a cheat it. Some content we didn't import. So we create the default HTML that just point to the old website so the client can check it and see how. This cover page, we want to make it different. So just to be able to check it with the client easily. URLs. Well, inside the draft.js, sometimes we have images or links that have to resolve UID. Today, the Vultublock don't deal with it. So right now, it just pointed at the image, no resolve UID. And we plan to fix it in Vultublock also. So probably will be something that you will use the same idea of resolve UID but with REST API and Vultublock. Folders. For simple folders, we just show, we plan to show just a list of the items in the folder. So it's a simple document with the, oh, now it's listing block, not collection, sorry. The listing block, showing what's inside the folder for collection is a document with listing block with the query of the collection. So that's the steps we did. Well, I have some time so I'm going to show you. Okay, why it's not there, just a minute. This is how we work today. So we have three plans here, plan for 3, 5.2. Then run the build out in plan for 3, we already created automatically some content and we wipe the data in plan 5.1, 5.2. So if we run the test, the smoke test here, we'll get error. So make test plan 5.1. Here we'll not have the content type, it still didn't migrate. Just a minute. And after migrate, we get the test passing. We plan to open those codes to the community, but it will get some time to polish it before. Okay, error. Then I run make, okay import, just import. And then it fire a plan website and run the import pipelines. It's creating the data that was exported by plan for 3. I forget to show the export comment. The export comment just create the data inside the JSON data. Okay, let's see one of them. Oops. And the data is exported, co-active JSONify, the default way we do with it. Oops. And okay, I already run the import and if I run again the tests, it will pass. It will take some time again. And we do the same at plan 5.1.5.2. That's how it works. It's the environment I'm working in the last year. Make me, I finished too late. You have questions? Oops. Sorry. Thank you. Thank you so much. Questions? I don't think it, thank you very much for your presentation. So now we have like four or five different ways at the moment by many organizations worldwide using plan build to migrate it. And one of your modules in here which will be very interesting for everybody will be the rich text to Volto conversion. Do you think it will be feasible to for the things that you've done new now in your migration stream to extract those, for example, this as a separate module that other people can start to reuse in their stream? The code is really simple. This is the code. Just 38 lines. I'm used the same structure of there is inside the Volto code. Just this part I extracted instead of import and use because I had some problem but still it was really simple part. And when I run it, I just pass the HTML and get the ref.js code. Super simple. But maybe this code should be inside the next upgrades, upgrades that are planned, something like this. So when we migrate from plan 5.2 to plan 6, should use something like this. Okay. I think I can show this to people. So is there a Python equivalent to that because for the migration from plan 5.2 to 6, we need to iterate over all the dexterity schemata and put that into separate blocks into a draft.js thingy and these usually run in Python? I tried to do it just with Python but it was today it's not possible. It's difficult because the code we have in Python for ref.js is just to do the other part, the reverse. Draft.js to HTML but not the HTML to draft.js. So that's why we created Telet and we fired with subprocess and got the result. Because it's running just one time, I think it's not too much overhead. Okay. Another question? Another one? Let me put this again here. I don't know if someone don't take the picture. I can show the first part. Right? Okay. Another one? Uh-oh. No, just saying. We have a sprint at the end of the week, right? So if somebody is interested in that, that could also be a sprint topic. Start working on upgrade steps and migrations. It's a bit early but why not if people are interested? Nice. Okay. Thank you. Thank you.
the last year. It's about data migration from because we deal with the final database don't Concept, we have our way to deal with it. We and the steps are we build the commander to, structure, we have some code that deal with this for 3, we already created automatically some community, but it will get some time to polish it