Importing data in an Open Source Wiki
Dubost, Ludovic
Using a new collaboration tools can be a challenge when it comes to pre-existing information. In this talk, we will look at available solutions when migrating data to the XWiki Open Source software. We will first review the differences between XWiki and alternate systems, namely Atlassian Confluence, Microsoft Sharepoint and document management systems. Secondly we will show multiple approaches to migrating data into XWiki and improving the quality of the information available. Finally we will demonstrate existing importing tools available in XWiki: Confluence Import & XWiki Batch Import.
 Hello, welcome at the talk, importing data in an open source wiki at FOSDEM 21 online. Thank you to the Dev Room organizers to organize this Dev Room. I'm very happy to be there. We usually are traveling to Brussels every year to see the open source community, the free software and open source community, and participate to the FOSDEM event, even though we cannot do this trip this year. We're very happy to do the FOSDEM conference online. I'd like to thank the organizer for keeping the conference and working on organizing it remotely. I'm Ludovic Dubost. I'm the creator of Xwiki and founder of Xwiki SIS. I have been working on open source for the last 16 years. The company Xwiki is 40 people that only work on open source software. We work mainly on the Xwiki software, but in the recent years we've also started a second project, CripPad, which I welcome you to go try. It's growing very fast as more and more users. We have passed 100,000 weekly users last week, especially in the education space. It's an entrant encrypted editor, competitor to Google Docs. There was a conference today about it before this conference. Everything we do is open source, and we're very proud to participate in the free software and open source movement in the last 16 years. First, why would you want to use a Xwiki and a Wiki? The main need around a Wiki is to improve knowledge sharing and organization of information. Users work much better when they have the proper information in time, especially when you have new team members. They're more easily integrated with great knowledge sharing. What we know and what we see is that using Wiki's information is much easier to create and organize than using Office Files, where it's very hard. It takes time to open a document. It takes time to create links from one to the other. You cannot organize it the same way, and it takes time to add information to it. Wiki's things are faster, and it encourages participation. This is very important. The second thing that it encourages is organizing the information, linking information from one place to another, which is something very important to help people find the relevant information. Xwiki is also a prime open source alternative to Atlassian Confluence. We're going to talk a little bit more about this when we reach the importing data from outside Wiki. Why we're actually talking about importing? We're talking about importing because even when users want to go open source, the transition is something that is difficult. There is proprietary software lock-in, but there is also content lock-in. It's not easy to move from one tool to another, especially when these tools are not sharing the same way of treating and handling information. There is also another reason is that you usually want to start a Wiki with data in it instead of an empty Wiki. It's a bit like an empty page where you don't know what to write. If you have already some content, it's easier for people to participate, and it's easier for people to understand the value of the Wiki concept. What are the types of import we were going to talk about today? We're going to talk about importing office files. We're going to talk also about importing information from a database or any database. Actually the smallest database is the spreadsheet today, but we could also talk about importing data that comes from structured database. We're going to talk about importing data from another Wiki, for example, conference here in this presentation, but also media Wiki or GitHub Wiki. We're going to talk about importing from HTML. We're not going to show this, but we're going to explain how this is possible. What are the available methods for importing? There's two types of methods, the manual one. We're going to import document by document. There's a second method that is also quite costly, which is coding using a scripting API. Then there are tools. At Xwiki, we have worked a lot on these tools over the years. First, tools to do batch importing. We start with an Excel file and some attach files. We import this in different Xwiki documents. We're importing from other Wikis using the Xwiki filter streams module, which can import data from already four different Wikis today. First, we're going to talk about the manual way. This is Wikithintax conversion and Office file importing. To do that, the first type of import that we can do in Xwiki is to do syntax conversion. Xwiki is actually polyglot and supports multiple syntax. We have more than 10 syntaxes that are supported in Xwiki. You can install these syntaxes when going in Xwiki's extension manager and searching for syntax. For example, you can import the MediaWiki syntax module, docuWiki syntax, or Twiki, or even Latik syntax, and the XHTML syntax, which is the syntax that the conference lately uses. The first thing is that you need to install additional syntaxes and then you need to potentially activate them if they are deactivated by default. This happens in the administration where you can check exactly which syntaxes are activated for your users. Once you've done that, you will have a little dropbox in the Wikii editor where you can actually change the syntax. You can create a document and decide that this document should be in markdown 1.2 syntax. If I choose this syntax, I can copy paste markdown, like in this example, and I can save the document and it will display. We actually have a support of markdown that is quite complete. You can still use the Wikii editor because we are able to do markdown conversion one way but also the other way. We can convert from markdown to XWiki, but we can convert from XWiki to markdown. We can convert from markdown to HTML and from HTML to markdown, which allows to use the Wikii editor. Once we've done that, we can actually trigger a conversion. By this, we are still in the Wikii editor. We can change the syntax. When it's supported to convert from that syntax to the XWiki syntax, for example, or to another syntax, the Wikii editor will propose to make the switch. Here I'm doing markdown 1.2 to XWiki 2.1 syntax conversion and it will convert the document. In the end, this will give us XWiki syntax so it's not very visible on the screen. We can see the result on the screen which displays. Obviously, this is an interesting way to both support syntaxes and also convert. However, it will not work very well for links because if you want to import multiple documents, then you have some issues about the links that you have between the documents. This is where you will need the more advanced tools to import Wikis. We're going to see these tools later. What's important to understand is that these tools are working with the syntax conversion module. If you import a GitHub Wikii, we're going to use the markdown module and the conversion from markdown to XWiki to convert the content. This is a first way of importing content. When it comes to HTML, you can do the same thing. You can actually copy paste HTML and setting the Wikii syntax to HTML and then convert. Or you can also copy paste HTML in the WSWiki editor. If you're in a document that would use XWiki syntax, for example, and when you actually save, XWiki will automatically translate it to the XWiki syntax. When you use the XWiki editor, it does a transformation between HTML back and forth to store the content in Wikis syntax. The second type of import I would like to show today is the Office importing. We have an Office importing tool which works with LibreOffice. You actually install a LibreOffice server on your XWiki instance on the server. It works on Windows, on Linux. You will connect the XWiki instance to that LibreOffice server. This is quite automatic. On Debian, for example, you just install XWiki using the Debian package. You install LibreOffice with a Debian package and then you just need to automatically activate the detection of LibreOffice in the XWiki configuration. And XWiki will detect where LibreOffice is installed by default and run it. Otherwise, you go in the administration to stop start the LibreOffice server. Once you have a working LibreOffice server in your XWiki instance, when you create a page, you will see a new type of document which is page from Office. You select that item which will lead you to a screen which I haven't shown here, which will give you a few options to import your document. So, it's quite complete. It converts, so LibreOffice converts to Word to HTML and then it converts HTML to XWiki syntax. There are options to filter styles, to clean up documents, and then there is a more advanced option to split documents. Before explaining this, I'd like to show also that it can import images and tables. So on the left, you can see the LibreOffice display of tables and then the XWiki display of tables with filter styles. With styles filter it. If you decided not to filter the styles, you would keep the colors of the tables, for example. Another aspect is that you can split the documents. I haven't shown this in this presentation, but you can choose to split by headings. And what will happen in this case is that you will create a different document every time the heading is found in the Word document. So for example, in this case, in this example, which is 120 pages report from the European community, if I decide to split on heading 123, I would create about 40 Wiki pages in my wiki for each section of the Word document. The second way of importing is to use scripting. So there are many, many APIs in XWiki. So we've seen the first API, which is the wiki syntax conversion. So when in the UI of XWiki, I made a conversion between the syntax markdown and the syntax XWiki, what's important to understand is that we've used the public API of XWiki to do that conversion. This means that this API is actually available to scripting in XWiki. And scripting in XWiki is actually a very powerful concept where you can use Groovy or Velocity as scripting languages to perform tasks. And in this case, when performing large imports, we would use Groovy and we would also use the job macro, which allows to run long tasks and provide some logging to know where this task is heading. And then you can automate running the API. So you can go call a remote server, ask for the pages. Once you have the pages getting the HTML content, converting the HTML to the wiki syntax, and then eventually transform some additional tasks like finding out the links on the documents so that you maintain the links between the document you're importing and some other documents you will import. This is the type of things we have done in the past. For example, we imported a very large website from the Lenovo community that was running on HTML connections, the previous Lotus IBM connections. And we were able to call the APIs of HTML and get all the documents, all the blog articles, and convert them to the equivalent XWiki documents. We've done also this for HTML imports. In some cases, you have to do two passes on the documents. So one, the first pass on all the HTML documents so that you can read them, find out what's going to be the page name of each of these documents once you're importing in XWiki. And so that you can create a map in order to know how to convert the links. And then you do a second pass and you're actually converting the syntax, converting HTML, and then modifying the links so that you can do the import. This works well. It requires some significant developer work each time. But we've been able to do this type of import between 10 to 30 days, depending on the complexity of the import and what you need to do. What's interesting with this method is that you can customize the import quite well and quite easily. For example, if you cannot import the whole document, if you get an HTML package which contains a little more than the documents themselves, like header, footers, and so on, you can strip them out before you actually run the HTML conversion. I would like now to explain a third type of import, which is the batch importing. So importing databases and automating file uploading and conversion in XWiki. So batch import is a tool that we have developed over time, which is quite a bit. Especially when we advanced it quite a lot when importing data from Lotus Notes for a customer. And what we did is that we converted the Lotus data to CSV files. We created very large CSV files and we stored all the attached files in the directory, but we referenced them in the CSV file. And then we used the batch importing tool to import the data and that's how the importing tool got a lot of functionality to support these type of imports. So what are the general principles of batch importing in XWiki using the XWiki batch importing tool? So first you install the batch import tool using the extension manager. Then you prepare data structure in XWiki. So you're going to use app within minutes to create a data structure. For example, it could be a client list or a project list. In the example I will show today, it's going to be a country database about COVID. And so I'm going to use app within minutes to do that. Second thing is I will prepare an Excel file of the data I'd like to import. In this case, I gathered an Excel file from a public database, an open data database, which contained data about the country's population size, GDP, but also some COVID information from the spring. Then I will also mention in the Excel file and the spreadsheet, I will put references to attached files that I would like to import as part of each line I'm importing. In this case, I will use a database of flags that are linked to the country code. And so I will have a file for each country, so BE.png, that will be a flag that I will find in the directory inside a zip file. Then I will upload the zip file on the server, upload the CSV file on the wiki and start mapping the data. And then I will have tools to preview and run the final import. So let's see exactly how this works. So first you create an app, you're going to use app within minutes. So we have a drag and drop interface to create fields. The app within minutes is actually a very powerful tool to help organize data in xwiki. And in this case, I create a country and I add fields. It took me about five minutes to do that, drag and dropping the different fields, the date field, two dates fields and about six or seven number fields. And I need to choose the right formats for the number, either these are longs, integers, or these are floats or even double. And for other fields, I choose the date field and I define the date format, whether it is a date time or date. Then I prepare my Excel file. So actually Excel or spreadsheets, in this case it's LibreOffice, is a very practical tool to manipulate the data quickly because you can change formulas and transform data. So initially I don't have a page name, I don't have the reference to the flag and I don't have the exact title I would like to have for every line. And so I create some formulas to allow this and I can do this cleanup. So I will create a field that will contain the page name, I will create a field that contains the space name in our case, its countries. You should always have a column which basically has the same value in it, which represents the space in which we're going to import the data. And then a column for the title. You can also clean up the numbers and the date data if necessary there in order to simplify the export. In the end I need either NixLS, 97 format or I need a CSV file. So both have their advantages and disadvantages with CSVs, you have to be careful with date because you're getting a string for the date and therefore you need to decide in which language or what the date format is exactly. So you upload your file to the wiki, you select it and then you will get a mapping screen. So the xwiki's batch import tool will read the columns from the CSV or the Excel file, choose the first line of the file as the name of the columns and ask you for it will read also the structure of data in which you'd like to import. So in my country application I'd like to import my data, it will ask me for every field which data I would like to take from the spreadsheet. So one thing that is interesting is that it's possible to run the import multiple times so you can actually import from multiple files in the same database potentially not overriding the data that you imported from the first file. If you're not using the field it won't touch the data in the wiki that is already there. So you choose and here it's almost one to one mapping, I need to make sure I put a document name, a space and a title, these are the three very important fields and then it's a one to one mapping of the data I'm importing the population and so on. Then I have some hidden settings which are a bit more tricky because they're not as well documented as the batch import. So you can find the batch import documentation on the extensions xwiki.org. Actually all the modules I've mentioned as I say are available in the extension repository of xwiki and the documentation is usually available on extensions xwiki.org. So these hidden settings they're concerned the flags. So I have a zip with all the flags, I'm uploading this to the server so I need an access to the server for this. I'm putting there somewhere on the file system that xwiki or the Java program will be able to access so you need to make sure about the rights. And you indicate the name of that zip file in the first field on the screen, it's a bit small here but if you zoom maybe you can see it properly, it's a pass and it ends with flag.zip. And you need also to tell that you want actually to do a file upload which is the last field on the screen where there is yes in the file upload field. This will make sure that when the import sees references to files in the spreadsheet it will upload these to the page that it just created. There is also an option to do file conversion. So in that case if it can find a word file it will convert the word file to wiki syntax and add it to the xwiki content field or to a text area field for example if you want to import in the blog. Finally in the past we've also customized these imports and we've been able to do pre and post treatment and one of them was to convert word files in PDF as it goes for every document. So you can also customize with code and potentially use any xwiki API to do pre and post treatment during this import. This is a bit more tricky and requires quite some experience with batch import tool. Finally you will be able to preview the results, this is actually quite important in order to avoid mistakes so you need to particularly be careful of the document full name. If you're not seeing the right data in the document full name or in the title it's probably not worth clicking the next button because you will have garbage in the data. You should also check your dates whether or not they are converted properly. This is a setting in the import to decide what is the format of your date. And then you get a screen on how to handle data de-duplication and also overwrite. So first set of settings that concern what to do if you find two lines that should end up in the same document in the same import process. And so do you ignore the second line, do you overwrite or do you update so you're going to merge the two lines into one line, the second one will win with its data. The second thing is whether what to do if there is already a page in the wiki not from this import but from a previous import and this is where you can make imports from multiple files and then you decide whether it's going to create a new document or if it's going to update the existing document or replace it. So for example in this method you can run multiple imports the same imports and make sure that you will delete the previous document before you import the new ones. So this is something to be careful when you're testing there might be already data and you can also delete all the data in app within minutes before you run a new import. And finally you can click next the import will take a bit of time you can then see the result. So before when you run the import or you run the preview you can look at the error messages there is extensive logging to really understand what is happening on all the lines. If you see an error you can try to understand exactly what is happening. So finally here I have an app within minute table with my countries with the flags of each country and I can filter and sort my data by the fields I imported so either by the population by the GDP or by the total cases of COVID that happened in the spring or by the total death that happened because of COVID in the spring. And I have imported data in my wiki. Well back to import is a very important tool to quickly do projects with Xwiki and do mass imports it's quite efficient. You've been importing gigabytes of data in the past with it. For example if you have attached files you can do a lot of things. There are some additional possibilities so as I said you can import and convert word to Xwiki you can even like import PowerPoints and it will attach them you can convert them to PDF you can also write custom processes at each stage of the import. You can also do converting of HTML or any other format to wiki syntax so if you're receiving some fields in the Excel file of HTML you can make conversion that requires scripting and you can work on the spreadsheet initially to prepare the data for the import. Lots of possibilities with batch import. For example one of the things you can do is that if you have a very large directory of files with a lot of folders well sometimes in the folder names you can find interesting information that you could make transform to metadata for example the name of the project. And so this is something you can do in Excel by manipulating the data in Excel and finally then you can import this in Xwiki. Now a part that is more advanced which is importing data from other wikis. So we will study a tool which is called Xwiki filter streams and today it's able to import from four wikis but it could also do other wikis potentially. So what is Xwiki filter streams? So you can find the documentation on extensions Xwiki org again I've put the link on this page. So it is a framework to import and export from and to Xwiki. It uses Xwiki syntax conversion capabilities but it also handles links between the pages that you're importing so that it can keep them. It can actually also import and export content in Xwiki so you can export using filter stream data from an old Xwiki instance and you can re-import it in a newer version of an instance. That is one way of doing upgrades when you don't want to keep everything from your previous instance and you want to start from scratch for your code itself. So what are the supported wikis for importing? So you have conference, media wiki, docu wiki and GitHub wiki. So some of these have been done through some of code. They don't have as much practice on the real wikis. We have quite a lot of practice for media wiki and conference. But one thing to understand is that there is no perfect import. There is always work to do on imports either manually when you come on the pages or eventually to improve the current tools. One of the reasons why it's difficult to really have perfect imports is that there can be custom features in the instances where the data comes from. There can be custom macros or a lot of macros and the importing tool will not support all the macros of conference and media wiki. There can also be absolute links. So links that are not recorded in the data as internal links between documents of the same system, but that are recorded as URLs and that can be problems. They can also be changes in recent versions of the tools from which we get the imports. There are many different versions of all the software conference media wiki and so it's not simple to follow everything that is different in all these instances. A little word about conference and why it's particularly interesting to talk about conference importing today. So first it's a wiki used by many developers including free and open source software fans. Many companies are using the Atlassian stack for development. So JIRA conference, Bitbucket and other tools. Those are using GitLab and we are big GitLab fans for development stack which is the most advanced open source software for developers. But a lot of companies are using the Atlassian stack and recently in December Atlassian has done a major shift away from on premise. So over the years Atlassian had changed their pricing policy regularly increasing the price. At xwiki we were already receiving calls from users trying to migrate from conference because they felt like that the price increase of the new version or of an increase of users was significant and not necessarily corresponding to what they would expect. And one of the difficulties with these migrations is that it's also a bit costly to do a migration. But in December what happened is that Atlassian has massively decided that the smaller customers should use their cloud including the open source users. Atlassian was giving list licenses of conference server or JIRA server to open source users for free and now they are only giving cloud instances for free. And you cannot get the server version of the Atlassian software for free. This is a major blow to controlling your own data because especially for smaller companies because you have to go for the data center offer which is starting at 20k dollars and is not suitable for smaller companies in terms of price. So this has actually made that xwiki is receiving more calls and more users and companies are looking for alternate solutions than Atlassian conference or Atlassian JIRA. So we believe xwiki is a great solution as an alternative to conference and it was already but we are seeing more customers and we are also seeing more participation to help us ease the migration path between conference and xwiki and we hope this is going to help us a lot in the future. So how does the conference import process work? So first we will use the filter stream from xwiki. We are going to export conference from conference itself as an XML export. It's going to give us a zip file. It can be pretty big. We then run the conference filter in xwiki's filter stream. This is going to import us every page in separate pages but all flat because conference doesn't have a system of nested pages. Everything is flat in conference. They use a parent-child relationship which is something we were using two a couple years ago before version 7 of xwiki. And at that time we migrated to a nested spaces system. And so this is why we need a second tool to migrate from the flat organization to the nested spaces migrator to the nested organization in xwiki and recover the data in the parent-child relationship that exists in the conference data. So in the end we'll get the same tree structure that we get from conference. We'll get it in xwiki 2. And finally there is a fourth step which is today the most complex one and it's not perfect is that we need to look at macros that were used in conference and see if we have alternate macros for these or if we need to create new macros. I'm not going to go too much into detail there because this is something we're still working on how to support more macros of conference when importing in xwiki. So first exporting from conference. So this happens in the space setting, content tool, export of conference. The user should choose XML output. It's also possible to limit what you export so select specific pages to export. It's important to note that if you're not an admin it might lead to a partial export and this can be an issue. Then you go in xwiki in the filter interface. You choose conference XML as the import source and then you point to the zip file that you need to have uploaded to the server in an area and with rights available so that xwiki can access it. So there is not a lot of options to fill in on the first screen. All this is documented on the conference XML page to make sure to choose the right parameters. The second part of the screen is the target parameter. You should choose xwiki instance. There are options to choose with and without version and then there are some other options which can have an impact on so who is the user that imported in xwiki, what common to put for that import and then you can run the input. This will lead to a screen with a progress bar and every page is being imported. It's possible to see errors. Normally errors should not stop the process of importing unless it's a major error that we don't know yet and the process will go to the end. In the meantime it's suggested not to play with the wiki and not interfere especially with the data being imported. There is another aspect to know that in case the import doesn't go till the end there could be issues with space because the data will not be deleted from the temp directory so if you run multiple imports and they all end up in error you might end up filling in the disk of your server. You need to be careful and make sure you have enough space on the server for at least three times the size of the conference import. If you have a very big import of 5 gigabytes it will take the zip file and then it will export the data and zip it in the temp directory and then it will import it in xwiki and the xwiki database. You need all that space so you need to make sure you have enough space on the server to process all this and you need to look at your temp directories in case there is an issue. If you restart the wiki then the temp data will be deleted. Finally once you have finished the first phase of the import which is importing the pages in the flat space you can run the nested pages migrator. Most options need to be unchecked. You need to choose the space that you want to convert. Usually this is the conference space key in our case it was a two letter cte and then you can compute the plan. The compute plan is not going to make any changes it is going to show you what it plans to do. In this case it is also where it might show you some errors and two long page names. Xwiki has a limit of 250 characters for the full page name including the whole tree. If you have a very big tree and in the way you have some pages that have very long titles knowing that conference is using the same data for the page name and the page title you might end up with full page names that are too long. What happens in this case is that you actually need to rename some pages not all of them in the tree because for example if your very long page name is at the third level of the tree and you have six levels of the tree the whole pages under will also be too long but you probably need only to rename the one at the top of the tree to do that. You need to use the normal Xwiki feature to rename the page. Find the page that has the too long page name. Once you've renamed it you can just redo the compute plan and see if you still have errors. It can fix multiple errors in one batch if you're renaming the right page. Once you have no errors of too long page name then you can just perform the conversion and the execute plan. It's important to understand that you cannot roll back from this conversion. This is the one-time conversion. Cannot come back to it so you need to restart from scratch your import if you're not doing the right thing here. The second aspect is about the too long page names. You don't need to rename the title. You just need to rename the page name. This is something that is not very easy for somebody that doesn't know Xwiki but if you know Xwiki it is not very difficult to fix when you have the issue of too long page name. In the end you can check the result and you should get a similar hierarchy than in conference. First you have a progress bar again and you can see the conversion that is happening. It can be long if there is a big hierarchy to move. And finally you can see your conference hierarchy on the left and you can see your pages in the right hierarchy on the right. However, it's not really finished here because the question that arises is what about macros? And so what if you see red boxes and you see a lot of them? Well, you might see a lot of red boxes if in your company there is a big practice of using a lot of macros. And this can happen because a conference tends to propose you templates which includes a lot of macros. We don't necessarily think that it's always very good to use too many macros. The value is the content, not necessarily the macros or the format that you've used for that content. But this is the way it is. You might have a lot of macros in your content. So you will have some red boxes when the macros don't exist in Xwiki and there might be work to either bridge these macros to Xwiki macros, which will require what we call wiki macro development. So in Xwiki it's possible to write macros using scripting. You can find some documentation about it. And otherwise you will need to wait for some new macros that we will build. Given the number of people asking about these conversions where we started working on trying to create more macros and support more macros, now we will be making some macros paying because it's significant work to support the companies wanting to migrate. And it is also necessary that more users support the Xwiki software so that it can continue to progress. Even if the macros are paying, everything is open source and you can find the source of our macros in our Xwiki SIS GitHub where we publish these macros. We also building a tool to list macros and count them and look at what the most important macros are. So if you're interested and if you would like to help us and if you're doing test imports you could try to tell us how many macros you have and which ones so that we can focus on the more important macros to migrate. So this tool will be available soon. We already have it. We can pass it to certain users if they want to count the macros used after importing their conference data in Xwiki. So the conference importing is a work in progress. There is still a lot of work. We'd like to support more content and we recently, very recently received a pull request by a contributor that supports that fixed issues that were reported already in our ticket tracking and for example, importing comments and labels is actually now will be supported once we merge this pull request if everything is fine with it. We'd like to support more macros and we need to continue to maintain the importer for future conference evolution. This will continue to evolve the data center version, the server until they stop it and also their cloud version. And the conference export format is not really officially documented. It's more like a database export than it is a document by document export. And so it's not always very easy to fully understand what's happening in that export format. If you would like to migrate, keep in mind that sponsoring the work is possible and you can also report issues in our ticket tracking. The more information we have, the better we can understand what is going on and improve the importer. We're actively working on this as I mentioned so that we can provide the best solutions for migration. Another aspect before going for the conclusion is that the same tool can also work for Media Wiki, for GitHub Wiki. It's possible to import the GitHub Wiki directly by giving the URL and the importer will create the pages in XWiki. And for Media Wiki, you need to export the file and same thing, we can then import the data. The conclusion is that we have actually many tools available for important content in XWiki. Over the years, these tools have been improved and refined. There's quite some solution. They generally need quite some expertise to run them. It's not so simple. They've been improved over time through service missions from XWiki SIS to its client. And I'd like to thank all the users and clients that have supported us to build these migration tools and importing tools. I'd like also to thank the contributors who have been reporting issues and also wrote code. We hope that these tools will continue to improve and particularly the conference one with more users coming to XWiki. Thank you very much. So I'll be able to take some questions or continue to take some questions in the chat. This talk has been fully built with open source. It's been written and recorded on the Cube OS operating system and that's using Fedora too. The presentation software is using Cripad only of its slide in alpha version, not available on Cripad FR in order to test the advancement of the solution. The screenshots have been done with Shotwell and the recording with open broadcasting software. And of course, it's using XWiki and the XWiki extension. Obviously the conference is not open source and it's the only software in this case that was not open source for these examples of this presentation. Thank you very much and have a great FOSDEM. We hope to see you for real next year in Brussels and that COVID will be behind us. Thank you very much.
 Hello, welcome at the talk, importing data in an open source wiki at FOSDEM 21 online.  Thank you to the Dev Room organizers to organize this Dev Room.  I'm very happy to be there.  We usually are traveling to Brussels every year to see the open source community, the  free software and open source community, and participate to the FOSDEM event, even though  we cannot do this trip this year.  We're very happy to do the FOSDEM conference online.  I'd like to thank the organizer for keeping the conference and working on organizing it  remotely.  I'm Ludovic Dubost.  I'm the creator of Xwiki and founder of Xwiki SIS.  I have been working on open source for the last 16 years.  The company Xwiki is 40 people that only work on open source software.  We work mainly on the Xwiki software, but in the recent years we've also started a  second project, CripPad, which I welcome you to go try.  It's growing very fast as more and more users.  We have passed 100,000 weekly users last week, especially in the education space.  It's an entrant encrypted editor, competitor to Google Docs.  There was a conference today about it before this conference.  Everything we do is open source, and we're very proud to participate in the free software  and open source movement in the last 16 years.  First, why would you want to use a Xwiki and a Wiki?  The main need around a Wiki is to improve knowledge sharing and organization of information.  Users work much better when they have the proper information in time, especially when you have  new team members.  They're more easily integrated with great knowledge sharing.  What we know and what we see is that using Wiki's information is much easier to create  and organize than using Office Files, where it's very hard.  It takes time to open a document.  It takes time to create links from one to the other.  You cannot organize it the same way, and it takes time to add information to it.  Wiki's things are faster, and it encourages participation.  This is very important.  The second thing that it encourages is organizing the information, linking information from  one place to another, which is something very important to help people find the relevant  information.  Xwiki is also a prime open source alternative to Atlassian Confluence.  We're going to talk a little bit more about this when we reach the importing data from  outside Wiki.  Why we're actually talking about importing?  We're talking about importing because even when users want to go open source, the transition  is something that is difficult.  There is proprietary software lock-in, but there is also content lock-in.  It's not easy to move from one tool to another, especially when these tools are not sharing  the same way of treating and handling information.  There is also another reason is that you usually want to start a Wiki with data in it instead  of an empty Wiki.  It's a bit like an empty page where you don't know what to write.  If you have already some content, it's easier for people to participate, and it's easier  for people to understand the value of the Wiki concept.  What are the types of import we were going to talk about today?  We're going to talk about importing office files.  We're going to talk also about importing information from a database or any database.  Actually the smallest database is the spreadsheet today, but we could also talk about importing  data that comes from structured database.  We're going to talk about importing data from another Wiki, for example, conference here  in this presentation, but also media Wiki or GitHub Wiki.  We're going to talk about importing from HTML.  We're not going to show this, but we're going to explain how this is possible.  What are the available methods for importing?  There's two types of methods, the manual one.  We're going to import document by document.  There's a second method that is also quite costly, which is coding using a scripting  API.  Then there are tools.  At Xwiki, we have worked a lot on these tools over the years.  First, tools to do batch importing.  We start with an Excel file and some attach files.  We import this in different Xwiki documents.  We're importing from other Wikis using the Xwiki filter streams module, which can import  data from already four different Wikis today.  First, we're going to talk about the manual way.  This is Wikithintax conversion and Office file importing.  To do that, the first type of import that we can do in Xwiki is to do syntax conversion.  Xwiki is actually polyglot and supports multiple syntax.  We have more than 10 syntaxes that are supported in Xwiki.  You can install these syntaxes when going in Xwiki's extension manager and searching  for syntax.  For example, you can import the MediaWiki syntax module, docuWiki syntax, or Twiki,  or even Latik syntax, and the XHTML syntax, which is the syntax that the conference lately  uses.  The first thing is that you need to install additional syntaxes and then you need to potentially  activate them if they are deactivated by default.  This happens in the administration where you can check exactly which syntaxes are activated  for your users.  Once you've done that, you will have a little dropbox in the Wikii editor where you can  actually change the syntax.  You can create a document and decide that this document should be in markdown 1.2 syntax.  If I choose this syntax, I can copy paste markdown, like in this example, and I can save  the document and it will display.  We actually have a support of markdown that is quite complete.  You can still use the Wikii editor because we are able to do markdown conversion one  way but also the other way.  We can convert from markdown to XWiki, but we can convert from XWiki to markdown.  We can convert from markdown to HTML and from HTML to markdown, which allows to use the  Wikii editor.  Once we've done that, we can actually trigger a conversion.  By this, we are still in the Wikii editor.  We can change the syntax.  When it's supported to convert from that syntax to the XWiki syntax, for example, or to another  syntax, the Wikii editor will propose to make the switch.  Here I'm doing markdown 1.2 to XWiki 2.1 syntax conversion and it will convert the document.  In the end, this will give us XWiki syntax so it's not very visible on the screen.  We can see the result on the screen which displays.  Obviously, this is an interesting way to both support syntaxes and also convert.  However, it will not work very well for links because if you want to import multiple documents,  then you have some issues about the links that you have between the documents.  This is where you will need the more advanced tools to import Wikis.  We're going to see these tools later.  What's important to understand is that these tools are working with the syntax conversion  module.  If you import a GitHub Wikii, we're going to use the markdown module and the conversion  from markdown to XWiki to convert the content.  This is a first way of importing content.  When it comes to HTML, you can do the same thing.  You can actually copy paste HTML and setting the Wikii syntax to HTML and then convert.  Or you can also copy paste HTML in the WSWiki editor.  If you're in a document that would use XWiki syntax, for example, and when you actually  save, XWiki will automatically translate it to the XWiki syntax.  When you use the XWiki editor, it does a transformation between HTML back and forth  to store the content in Wikis syntax.  The second type of import I would like to show today is the Office importing.  We have an Office importing tool which works with LibreOffice.  You actually install a LibreOffice server on your XWiki instance on the server.  It works on Windows, on Linux.  You will connect the XWiki instance to that LibreOffice server.  This is quite automatic.  On Debian, for example, you just install XWiki using the Debian package.  You install LibreOffice with a Debian package and then you just need to automatically activate  the detection of LibreOffice in the XWiki configuration.  And XWiki will detect where LibreOffice is installed by default and run it.  Otherwise, you go in the administration to stop start the LibreOffice server.  Once you have a working LibreOffice server in your XWiki instance, when you create a  page, you will see a new type of document which is page from Office.  You select that item which will lead you to a screen which I haven't shown here, which  will give you a few options to import your document.  So, it's quite complete.  It converts, so LibreOffice converts to Word to HTML and then it converts HTML to XWiki  syntax.  There are options to filter styles, to clean up documents, and then there is a more advanced  option to split documents.  Before explaining this, I'd like to show also that it can import images and tables.  So on the left, you can see the LibreOffice display of tables and then the XWiki display  of tables with filter styles.  With styles filter it.  If you decided not to filter the styles, you would keep the colors of the tables, for  example.  Another aspect is that you can split the documents.  I haven't shown this in this presentation, but you can choose to split by headings.  And what will happen in this case is that you will create a different document every  time the heading is found in the Word document.  So for example, in this case, in this example, which is 120 pages report from the European  community, if I decide to split on heading 123, I would create about 40 Wiki pages in  my wiki for each section of the Word document.  The second way of importing is to use scripting.  So there are many, many APIs in XWiki.  So we've seen the first API, which is the wiki syntax conversion.  So when in the UI of XWiki, I made a conversion between the syntax markdown and the syntax  XWiki, what's important to understand is that we've used the public API of XWiki to do that  conversion.  This means that this API is actually available to scripting in XWiki.  And scripting in XWiki is actually a very powerful concept where you can use Groovy or  Velocity as scripting languages to perform tasks.  And in this case, when performing large imports, we would use Groovy and we would also use  the job macro, which allows to run long tasks and provide some logging to know where this  task is heading.  And then you can automate running the API.  So you can go call a remote server, ask for the pages.  Once you have the pages getting the HTML content, converting the HTML to the wiki syntax, and  then eventually transform some additional tasks like finding out the links on the documents  so that you maintain the links between the document you're importing and some other documents  you will import.  This is the type of things we have done in the past.  For example, we imported a very large website from the Lenovo community that was running  on HTML connections, the previous Lotus IBM connections.  And we were able to call the APIs of HTML and get all the documents, all the blog articles,  and convert them to the equivalent XWiki documents.  We've done also this for HTML imports.  In some cases, you have to do two passes on the documents.  So one, the first pass on all the HTML documents so that you can read them, find out what's  going to be the page name of each of these documents once you're importing in XWiki.  And so that you can create a map in order to know how to convert the links.  And then you do a second pass and you're actually converting the syntax, converting  HTML, and then modifying the links so that you can do the import.  This works well.  It requires some significant developer work each time.  But we've been able to do this type of import between 10 to 30 days, depending on the complexity  of the import and what you need to do.  What's interesting with this method is that you can customize the import quite well and  quite easily.  For example, if you cannot import the whole document, if you get an HTML package which  contains a little more than the documents themselves, like header, footers, and so on,  you can strip them out before you actually run the HTML conversion.  I would like now to explain a third type of import, which is the batch importing.  So importing databases and automating file uploading and conversion in XWiki.  So batch import is a tool that we have developed over time, which is quite a bit.  Especially when we advanced it quite a lot when importing data from Lotus Notes for  a customer.  And what we did is that we converted the Lotus data to CSV files.  We created very large CSV files and we stored all the attached files in the directory, but  we referenced them in the CSV file.  And then we used the batch importing tool to import the data and that's how the importing  tool got a lot of functionality to support these type of imports.  So what are the general principles of batch importing in XWiki using the XWiki batch importing  tool?  So first you install the batch import tool using the extension manager.  Then you prepare data structure in XWiki.  So you're going to use app within minutes to create a data structure.  For example, it could be a client list or a project list.  In the example I will show today, it's going to be a country database about COVID.  And so I'm going to use app within minutes to do that.  Second thing is I will prepare an Excel file of the data I'd like to import.  In this case, I gathered an Excel file from a public database, an open data database,  which contained data about the country's population size, GDP, but also some COVID information  from the spring.  Then I will also mention in the Excel file and the spreadsheet, I will put references  to attached files that I would like to import as part of each line I'm importing.  In this case, I will use a database of flags that are linked to the country code.  And so I will have a file for each country, so BE.png, that will be a flag that I will  find in the directory inside a zip file.  Then I will upload the zip file on the server, upload the CSV file on the wiki and start mapping  the data.  And then I will have tools to preview and run the final import.  So let's see exactly how this works.  So first you create an app, you're going to use app within minutes.  So we have a drag and drop interface to create fields.  The app within minutes is actually a very powerful tool to help organize data in xwiki.  And in this case, I create a country and I add fields.  It took me about five minutes to do that, drag and dropping the different fields, the  date field, two dates fields and about six or seven number fields.  And I need to choose the right formats for the number, either these are longs, integers,  or these are floats or even double.  And for other fields, I choose the date field and I define the date format, whether it is  a date time or date.  Then I prepare my Excel file.  So actually Excel or spreadsheets, in this case it's LibreOffice, is a very practical  tool to manipulate the data quickly because you can change formulas and transform data.  So initially I don't have a page name, I don't have the reference to the flag and I don't  have the exact title I would like to have for every line.  And so I create some formulas to allow this and I can do this cleanup.  So I will create a field that will contain the page name, I will create a field that  contains the space name in our case, its countries.  You should always have a column which basically has the same value in it, which represents  the space in which we're going to import the data.  And then a column for the title.  You can also clean up the numbers and the date data if necessary there in order to simplify  the export.  In the end I need either NixLS, 97 format or I need a CSV file.  So both have their advantages and disadvantages with CSVs, you have to be careful with date  because you're getting a string for the date and therefore you need to decide in which  language or what the date format is exactly.  So you upload your file to the wiki, you select it and then you will get a mapping screen.  So the xwiki's batch import tool will read the columns from the CSV or the Excel file,  choose the first line of the file as the name of the columns and ask you for it will read  also the structure of data in which you'd like to import.  So in my country application I'd like to import my data, it will ask me for every field which  data I would like to take from the spreadsheet.  So one thing that is interesting is that it's possible to run the import multiple times  so you can actually import from multiple files in the same database potentially not overriding  the data that you imported from the first file.  If you're not using the field it won't touch the data in the wiki that is already there.  So you choose and here it's almost one to one mapping, I need to make sure I put a document  name, a space and a title, these are the three very important fields and then it's a one  to one mapping of the data I'm importing the population and so on.  Then I have some hidden settings which are a bit more tricky because they're not as well  documented as the batch import.  So you can find the batch import documentation on the extensions xwiki.org.  Actually all the modules I've mentioned as I say are available in the extension repository  of xwiki and the documentation is usually available on extensions xwiki.org.  So these hidden settings they're concerned the flags.  So I have a zip with all the flags, I'm uploading this to the server so I need an access to  the server for this.  I'm putting there somewhere on the file system that xwiki or the Java program will be able  to access so you need to make sure about the rights.  And you indicate the name of that zip file in the first field on the screen, it's a bit  small here but if you zoom maybe you can see it properly, it's a pass and it ends with  flag.zip.  And you need also to tell that you want actually to do a file upload which is the last field  on the screen where there is yes in the file upload field.  This will make sure that when the import sees references to files in the spreadsheet it will  upload these to the page that it just created.  There is also an option to do file conversion.  So in that case if it can find a word file it will convert the word file to wiki syntax  and add it to the xwiki content field or to a text area field for example if you want  to import in the blog.  Finally in the past we've also customized these imports and we've been able to do pre  and post treatment and one of them was to convert word files in PDF as it goes for every  document.  So you can also customize with code and potentially use any xwiki API to do pre and post treatment  during this import.  This is a bit more tricky and requires quite some experience with batch import tool.  Finally you will be able to preview the results, this is actually quite important in order  to avoid mistakes so you need to particularly be careful of the document full name.  If you're not seeing the right data in the document full name or in the title it's probably  not worth clicking the next button because you will have garbage in the data.  You should also check your dates whether or not they are converted properly.  This is a setting in the import to decide what is the format of your date.  And then you get a screen on how to handle data de-duplication and also overwrite.  So first set of settings that concern what to do if you find two lines that should end  up in the same document in the same import process.  And so do you ignore the second line, do you overwrite or do you update so you're going  to merge the two lines into one line, the second one will win with its data.  The second thing is whether what to do if there is already a page in the wiki not from  this import but from a previous import and this is where you can make imports from multiple  files and then you decide whether it's going to create a new document or if it's going  to update the existing document or replace it.  So for example in this method you can run multiple imports the same imports and make  sure that you will delete the previous document before you import the new ones.  So this is something to be careful when you're testing there might be already data and you  can also delete all the data in app within minutes before you run a new import.  And finally you can click next the import will take a bit of time you can then see the result.  So before when you run the import or you run the preview you can look at the error messages  there is extensive logging to really understand what is happening on all the lines.  If you see an error you can try to understand exactly what is happening.  So finally here I have an app within minute table with my countries with the flags of  each country and I can filter and sort my data by the fields I imported so either by  the population by the GDP or by the total cases of COVID that happened in the spring  or by the total death that happened because of COVID in the spring.  And I have imported data in my wiki.  Well back to import is a very important tool to quickly do projects with Xwiki and do mass  imports it's quite efficient.  You've been importing gigabytes of data in the past with it.  For example if you have attached files you can do a lot of things.  There are some additional possibilities so as I said you can import and convert word  to Xwiki you can even like import PowerPoints and it will attach them you can convert them  to PDF you can also write custom processes at each stage of the import.  You can also do converting of HTML or any other format to wiki syntax so if you're receiving  some fields in the Excel file of HTML you can make conversion that requires scripting  and you can work on the spreadsheet initially to prepare the data for the import.  Lots of possibilities with batch import.  For example one of the things you can do is that if you have a very large directory of  files with a lot of folders well sometimes in the folder names you can find interesting  information that you could make transform to metadata for example the name of the project.  And so this is something you can do in Excel by manipulating the data in Excel and finally  then you can import this in Xwiki.  Now a part that is more advanced which is importing data from other wikis.  So we will study a tool which is called Xwiki filter streams and today it's able to import  from four wikis but it could also do other wikis potentially.  So what is Xwiki filter streams?  So you can find the documentation on extensions Xwiki org again I've put the link on this  page.  So it is a framework to import and export from and to Xwiki.  It uses Xwiki syntax conversion capabilities but it also handles links between the pages  that you're importing so that it can keep them.  It can actually also import and export content in Xwiki so you can export using filter stream  data from an old Xwiki instance and you can re-import it in a newer version of an instance.  That is one way of doing upgrades when you don't want to keep everything from your previous  instance and you want to start from scratch for your code itself.  So what are the supported wikis for importing?  So you have conference, media wiki, docu wiki and GitHub wiki.  So some of these have been done through some of code.  They don't have as much practice on the real wikis.  We have quite a lot of practice for media wiki and conference.  But one thing to understand is that there is no perfect import.  There is always work to do on imports either manually when you come on the pages or eventually  to improve the current tools.  One of the reasons why it's difficult to really have perfect imports is that there can be  custom features in the instances where the data comes from.  There can be custom macros or a lot of macros and the importing tool will not support all  the macros of conference and media wiki.  There can also be absolute links.  So links that are not recorded in the data as internal links between documents of the  same system, but that are recorded as URLs and that can be problems.  They can also be changes in recent versions of the tools from which we get the imports.  There are many different versions of all the software conference media wiki and so it's  not simple to follow everything that is different in all these instances.  A little word about conference and why it's particularly interesting to talk about conference  importing today.  So first it's a wiki used by many developers including free and open source software fans.  Many companies are using the Atlassian stack for development.  So JIRA conference, Bitbucket and other tools.  Those are using GitLab and we are big GitLab fans for development stack which is the most  advanced open source software for developers.  But a lot of companies are using the Atlassian stack and recently in December Atlassian has  done a major shift away from on premise.  So over the years Atlassian had changed their pricing policy regularly increasing the price.  At xwiki we were already receiving calls from users trying to migrate from conference because  they felt like that the price increase of the new version or of an increase of users  was significant and not necessarily corresponding to what they would expect.  And one of the difficulties with these migrations is that it's also a bit costly to do a migration.  But in December what happened is that Atlassian has massively decided that the smaller customers  should use their cloud including the open source users.  Atlassian was giving list licenses of conference server or JIRA server to open source users  for free and now they are only giving cloud instances for free.  And you cannot get the server version of the Atlassian software for free.  This is a major blow to controlling your own data because especially for smaller companies  because you have to go for the data center offer which is starting at 20k dollars and  is not suitable for smaller companies in terms of price.  So this has actually made that xwiki is receiving more calls and more users and companies are  looking for alternate solutions than Atlassian conference or Atlassian JIRA.  So we believe xwiki is a great solution as an alternative to conference and it was already  but we are seeing more customers and we are also seeing more participation to help us  ease the migration path between conference and xwiki and we hope this is going to help  us a lot in the future.  So how does the conference import process work?  So first we will use the filter stream from xwiki.  We are going to export conference from conference itself as an XML export.  It's going to give us a zip file.  It can be pretty big.  We then run the conference filter in xwiki's filter stream.  This is going to import us every page in separate pages but all flat because conference doesn't  have a system of nested pages.  Everything is flat in conference.  They use a parent-child relationship which is something we were using two a couple years  ago before version 7 of xwiki.  And at that time we migrated to a nested spaces system.  And so this is why we need a second tool to migrate from the flat organization to the  nested spaces migrator to the nested organization in xwiki and recover the data in the parent-child  relationship that exists in the conference data.  So in the end we'll get the same tree structure that we get from conference.  We'll get it in xwiki 2.  And finally there is a fourth step which is today the most complex one and it's not perfect  is that we need to look at macros that were used in conference and see if we have alternate  macros for these or if we need to create new macros.  I'm not going to go too much into detail there because this is something we're still working  on how to support more macros of conference when importing in xwiki.  So first exporting from conference.  So this happens in the space setting, content tool, export of conference.  The user should choose XML output.  It's also possible to limit what you export so select specific pages to export.  It's important to note that if you're not an admin it might lead to a partial export  and this can be an issue.  Then you go in xwiki in the filter interface.  You choose conference XML as the import source and then you point to the zip file that you  need to have uploaded to the server in an area and with rights available so that xwiki  can access it.  So there is not a lot of options to fill in on the first screen.  All this is documented on the conference XML page to make sure to choose the right parameters.  The second part of the screen is the target parameter.  You should choose xwiki instance.  There are options to choose with and without version and then there are some other options  which can have an impact on so who is the user that imported in xwiki, what common to  put for that import and then you can run the input.  This will lead to a screen with a progress bar and every page is being imported.  It's possible to see errors.  Normally errors should not stop the process of importing unless it's a major error that  we don't know yet and the process will go to the end.  In the meantime it's suggested not to play with the wiki and not interfere especially  with the data being imported.  There is another aspect to know that in case the import doesn't go till the end there could  be issues with space because the data will not be deleted from the temp directory so  if you run multiple imports and they all end up in error you might end up filling in the  disk of your server.  You need to be careful and make sure you have enough space on the server for at least three  times the size of the conference import.  If you have a very big import of 5 gigabytes it will take the zip file and then it will  export the data and zip it in the temp directory and then it will import it in xwiki and the  xwiki database.  You need all that space so you need to make sure you have enough space on the server to  process all this and you need to look at your temp directories in case there is an issue.  If you restart the wiki then the temp data will be deleted.  Finally once you have finished the first phase of the import which is importing the pages  in the flat space you can run the nested pages migrator.  Most options need to be unchecked.  You need to choose the space that you want to convert.  Usually this is the conference space key in our case it was a two letter cte and then  you can compute the plan.  The compute plan is not going to make any changes it is going to show you what it plans  to do.  In this case it is also where it might show you some errors and two long page names.  Xwiki has a limit of 250 characters for the full page name including the whole tree.  If you have a very big tree and in the way you have some pages that have very long titles  knowing that conference is using the same data for the page name and the page title  you might end up with full page names that are too long.  What happens in this case is that you actually need to rename some pages not all of them  in the tree because for example if your very long page name is at the third level of the  tree and you have six levels of the tree the whole pages under will also be too long but  you probably need only to rename the one at the top of the tree to do that.  You need to use the normal Xwiki feature to rename the page.  Find the page that has the too long page name.  Once you've renamed it you can just redo the compute plan and see if you still have errors.  It can fix multiple errors in one batch if you're renaming the right page.  Once you have no errors of too long page name then you can just perform the conversion  and the execute plan.  It's important to understand that you cannot roll back from this conversion.  This is the one-time conversion.  Cannot come back to it so you need to restart from scratch your import if you're not doing  the right thing here.  The second aspect is about the too long page names.  You don't need to rename the title.  You just need to rename the page name.  This is something that is not very easy for somebody that doesn't know Xwiki but if you  know Xwiki it is not very difficult to fix when you have the issue of too long page name.  In the end you can check the result and you should get a similar hierarchy than in conference.  First you have a progress bar again and you can see the conversion that is happening.  It can be long if there is a big hierarchy to move.  And finally you can see your conference hierarchy on the left and you can see your pages in  the right hierarchy on the right.  However, it's not really finished here because the question that arises is what about macros?  And so what if you see red boxes and you see a lot of them?  Well, you might see a lot of red boxes if in your company there is a big practice of  using a lot of macros.  And this can happen because a conference tends to propose you templates which includes  a lot of macros.  We don't necessarily think that it's always very good to use too many macros.  The value is the content, not necessarily the macros or the format that you've used  for that content.  But this is the way it is.  You might have a lot of macros in your content.  So you will have some red boxes when the macros don't exist in Xwiki and there might be work  to either bridge these macros to Xwiki macros, which will require what we call wiki macro  development.  So in Xwiki it's possible to write macros using scripting.  You can find some documentation about it.  And otherwise you will need to wait for some new macros that we will build.  Given the number of people asking about these conversions where we started working on trying  to create more macros and support more macros, now we will be making some macros paying because  it's significant work to support the companies wanting to migrate.  And it is also necessary that more users support the Xwiki software so that it can continue  to progress.  Even if the macros are paying, everything is open source and you can find the source  of our macros in our Xwiki SIS GitHub where we publish these macros.  We also building a tool to list macros and count them and look at what the most important  macros are.  So if you're interested and if you would like to help us and if you're doing test  imports you could try to tell us how many macros you have and which ones so that we  can focus on the more important macros to migrate.  So this tool will be available soon.  We already have it.  We can pass it to certain users if they want to count the macros used after importing their  conference data in Xwiki.  So the conference importing is a work in progress.  There is still a lot of work.  We'd like to support more content and we recently, very recently received a pull request by a  contributor that supports that fixed issues that were reported already in our ticket  tracking and for example, importing comments and labels is actually now will be supported  once we merge this pull request if everything is fine with it.  We'd like to support more macros and we need to continue to maintain the importer for future  conference evolution.  This will continue to evolve the data center version, the server until they stop it and  also their cloud version.  And the conference export format is not really officially documented.  It's more like a database export than it is a document by document export.  And so it's not always very easy to fully understand what's happening in that export  format.  If you would like to migrate, keep in mind that sponsoring the work is possible and you  can also report issues in our ticket tracking.  The more information we have, the better we can understand what is going on and improve  the importer.  We're actively working on this as I mentioned so that we can provide the best solutions  for migration.  Another aspect before going for the conclusion is that the same tool can also work for Media  Wiki, for GitHub Wiki.  It's possible to import the GitHub Wiki directly by giving the URL and the importer will create  the pages in XWiki.  And for Media Wiki, you need to export the file and same thing, we can then import the  data.  The conclusion is that we have actually many tools available for important content in XWiki.  Over the years, these tools have been improved and refined.  There's quite some solution.  They generally need quite some expertise to run them.  It's not so simple.  They've been improved over time through service missions from XWiki SIS to its client.  And I'd like to thank all the users and clients that have supported us to build these migration  tools and importing tools.  I'd like also to thank the contributors who have been reporting issues and also wrote  code.  We hope that these tools will continue to improve and particularly the conference one  with more users coming to XWiki.  Thank you very much.  So I'll be able to take some questions or continue to take some questions in the chat.  This talk has been fully built with open source.  It's been written and recorded on the Cube OS operating system and that's using Fedora  too.  The presentation software is using Cripad only of its slide in alpha version, not available  on Cripad FR in order to test the advancement of the solution.  The screenshots have been done with Shotwell and the recording with open broadcasting software.  And of course, it's using XWiki and the XWiki extension.  Obviously the conference is not open source and it's the only software in this case that  was not open source for these examples of this presentation.  Thank you very much and have a great FOSDEM.  We hope to see you for real next year in Brussels and that COVID will be behind us.  Thank you very much.
migratetoxwiki_0_179.png migratetoxwiki_180_299.png migratetoxwiki_300_875.png migratetoxwiki_876_1775.png migratetoxwiki_1776_3503.png migratetoxwiki_3504_5711.png migratetoxwiki_5712_7103.png migratetoxwiki_7104_7655.png migratetoxwiki_7656_8015.png migratetoxwiki_8016_8279.png migratetoxwiki_8280_9395.png migratetoxwiki_9396_9443.png migratetoxwiki_9444_9671.png migratetoxwiki_9672_9779.png migratetoxwiki_9780_9959.png migratetoxwiki_9960_11519.png migratetoxwiki_11520_13127.png migratetoxwiki_13128_13883.png migratetoxwiki_13884_16259.png migratetoxwiki_16260_17279.png migratetoxwiki_17280_18359.png migratetoxwiki_18360_19415.png migratetoxwiki_19416_21071.png migratetoxwiki_21072_22103.png migratetoxwiki_22104_26579.png migratetoxwiki_26580_28199.png migratetoxwiki_28200_31223.png migratetoxwiki_31224_32603.png migratetoxwiki_32604_35015.png migratetoxwiki_35016_35243.png migratetoxwiki_35244_37199.png migratetoxwiki_37200_40919.png migratetoxwiki_40920_41963.png migratetoxwiki_41964_43955.png migratetoxwiki_43956_45215.png migratetoxwiki_45216_45287.png migratetoxwiki_45288_45839.png migratetoxwiki_45840_47435.png migratetoxwiki_47436_47579.png migratetoxwiki_47580_48323.png migratetoxwiki_48324_49979.png migratetoxwiki_49980_52499.png migratetoxwiki_52500_57755.png migratetoxwiki_57756_59411.png migratetoxwiki_59412_59471.png migratetoxwiki_59472_59531.png migratetoxwiki_59532_60575.png migratetoxwiki_60576_61211.png migratetoxwiki_61212_61283.png migratetoxwiki_61284_62195.png migratetoxwiki_62196_62507.png migratetoxwiki_62508_62963.png migratetoxwiki_62964_65507.png migratetoxwiki_65508_66479.png migratetoxwiki_66480_70319.png migratetoxwiki_70320_70655.png migratetoxwiki_70656_70751.png migratetoxwiki_70752_70835.png migratetoxwiki_70836_71051.png migratetoxwiki_71052_74123.png migratetoxwiki_74124_75131.png migratetoxwiki_75132_77111.png migratetoxwiki_77112_77723.png migratetoxwiki_77724_78527.png migratetoxwiki_78528_79811.png migratetoxwiki_79812_81875.png migratetoxwiki_81876_82151.png migratetoxwiki_82152_82274.png
Hello, welcome at the talk, importing data in an open source wiki at FOSDEM 21 online. Thank you to the Dev Room organizers to organize this Dev Room. I'm very happy to be there. We usually are traveling to Brussels every year to see the open source community, the free software and open source community, and participate to the FOSDEM event, even though we cannot do this trip this year. We're very happy to do the FOSDEM conference online. I'd like to thank the organizer for keeping the conference and working on organizing it remotely. I'm Ludovic Dubost. I'm the creator of Xwiki and founder of Xwiki SIS. I have been working on open source for the last 16 years. The company Xwiki is 40 people that only work on open source software. We work mainly on the Xwiki software, but in the recent years we've also started a second project, CripPad, which I welcome you to go try. It's growing very fast as more and more users. We have passed 100,000 weekly users last week, especially in the education space. It's an entrant encrypted editor, competitor to Google Docs. There was a conference today about it before this conference. Everything we do is open source, and we're very proud to participate in the free software and open source movement in the last 16 years. First, why would you want to use a Xwiki and a Wiki? The main need around a Wiki is to improve knowledge sharing and organization of information. Users work much better when they have the proper information in time, especially when you have new team members. They're more easily integrated with great knowledge sharing. What we know and what we see is that using Wiki's information is much easier to create and organize than using Office Files, where it's very hard. It takes time to open a document. It takes time to create links from one to the other. You cannot organize it the same way, and it takes time to add information to it. Wiki's things are faster, and it encourages participation. This is very important. The second thing that it encourages is organizing the information, linking information from one place to another, which is something very important to help people find the relevant information. Xwiki is also a prime open source alternative to Atlassian Confluence. We're going to talk a little bit more about this when we reach the importing data from outside Wiki. Why we're actually talking about importing? We're talking about importing because even when users want to go open source, the transition is something that is difficult. There is proprietary software lock-in, but there is also content lock-in. It's not easy to move from one tool to another, especially when these tools are not sharing the same way of treating and handling information. There is also another reason is that you usually want to start a Wiki with data in it instead of an empty Wiki. It's a bit like an empty page where you don't know what to write. If you have already some content, it's easier for people to participate, and it's easier for people to understand the value of the Wiki concept. What are the types of import we were going to talk about today? We're going to talk about importing office files. We're going to talk also about importing information from a database or any database. Actually the smallest database is the spreadsheet today, but we could also talk about importing data that comes from structured database. We're going to talk about importing data from another Wiki, for example, conference here in this presentation, but also media Wiki or GitHub Wiki. We're going to talk about importing from HTML. We're not going to show this, but we're going to explain how this is possible. What are the available methods for importing? There's two types of methods, the manual one. We're going to import document by document. There's a second method that is also quite costly, which is coding using a scripting API. Then there are tools. At Xwiki, we have worked a lot on these tools over the years. First, tools to do batch importing. We start with an Excel file and some attach files. We import this in different Xwiki documents. We're importing from other Wikis using the Xwiki filter streams module, which can import data from already four different Wikis today. First, we're going to talk about the manual way. This is Wikithintax conversion and Office file importing. To do that, the first type of import that we can do in Xwiki is to do syntax conversion. Xwiki is actually polyglot and supports multiple syntax. We have more than 10 syntaxes that are supported in Xwiki. You can install these syntaxes when going in Xwiki's extension manager and searching for syntax. For example, you can import the MediaWiki syntax module, docuWiki syntax, or Twiki, or even Latik syntax, and the XHTML syntax, which is the syntax that the conference lately uses. The first thing is that you need to install additional syntaxes and then you need to potentially activate them if they are deactivated by default. This happens in the administration where you can check exactly which syntaxes are activated for your users. Once you've done that, you will have a little dropbox in the Wikii editor where you can actually change the syntax. You can create a document and decide that this document should be in markdown 1.2 syntax. If I choose this syntax, I can copy paste markdown, like in this example, and I can save the document and it will display. We actually have a support of markdown that is quite complete. You can still use the Wikii editor because we are able to do markdown conversion one way but also the other way. We can convert from markdown to XWiki, but we can convert from XWiki to markdown. We can convert from markdown to HTML and from HTML to markdown, which allows to use the Wikii editor. Once we've done that, we can actually trigger a conversion. By this, we are still in the Wikii editor. We can change the syntax. When it's supported to convert from that syntax to the XWiki syntax, for example, or to another syntax, the Wikii editor will propose to make the switch. Here I'm doing markdown 1.2 to XWiki 2.1 syntax conversion and it will convert the document. In the end, this will give us XWiki syntax so it's not very visible on the screen. We can see the result on the screen which displays. Obviously, this is an interesting way to both support syntaxes and also convert. However, it will not work very well for links because if you want to import multiple documents, then you have some issues about the links that you have between the documents. This is where you will need the more advanced tools to import Wikis. We're going to see these tools later. What's important to understand is that these tools are working with the syntax conversion module. If you import a GitHub Wikii, we're going to use the markdown module and the conversion from markdown to XWiki to convert the content. This is a first way of importing content. When it comes to HTML, you can do the same thing. You can actually copy paste HTML and setting the Wikii syntax to HTML and then convert. Or you can also copy paste HTML in the WSWiki editor. If you're in a document that would use XWiki syntax, for example, and when you actually save, XWiki will automatically translate it to the XWiki syntax. When you use the XWiki editor, it does a transformation between HTML back and forth to store the content in Wikis syntax. The second type of import I would like to show today is the Office importing. We have an Office importing tool which works with LibreOffice. You actually install a LibreOffice server on your XWiki instance on the server. It works on Windows, on Linux. You will connect the XWiki instance to that LibreOffice server. This is quite automatic. On Debian, for example, you just install XWiki using the Debian package. You install LibreOffice with a Debian package and then you just need to automatically activate the detection of LibreOffice in the XWiki configuration. And XWiki will detect where LibreOffice is installed by default and run it. Otherwise, you go in the administration to stop start the LibreOffice server. Once you have a working LibreOffice server in your XWiki instance, when you create a page, you will see a new type of document which is page from Office. You select that item which will lead you to a screen which I haven't shown here, which will give you a few options to import your document. So, it's quite complete. It converts, so LibreOffice converts to Word to HTML and then it converts HTML to XWiki syntax. There are options to filter styles, to clean up documents, and then there is a more advanced option to split documents. Before explaining this, I'd like to show also that it can import images and tables. So on the left, you can see the LibreOffice display of tables and then the XWiki display of tables with filter styles. With styles filter it. If you decided not to filter the styles, you would keep the colors of the tables, for example. Another aspect is that you can split the documents. I haven't shown this in this presentation, but you can choose to split by headings. And what will happen in this case is that you will create a different document every time the heading is found in the Word document. So for example, in this case, in this example, which is 120 pages report from the European community, if I decide to split on heading 123, I would create about 40 Wiki pages in my wiki for each section of the Word document. The second way of importing is to use scripting. So there are many, many APIs in XWiki. So we've seen the first API, which is the wiki syntax conversion. So when in the UI of XWiki, I made a conversion between the syntax markdown and the syntax XWiki, what's important to understand is that we've used the public API of XWiki to do that conversion. This means that this API is actually available to scripting in XWiki. And scripting in XWiki is actually a very powerful concept where you can use Groovy or Velocity as scripting languages to perform tasks. And in this case, when performing large imports, we would use Groovy and we would also use the job macro, which allows to run long tasks and provide some logging to know where this task is heading. And then you can automate running the API. So you can go call a remote server, ask for the pages. Once you have the pages getting the HTML content, converting the HTML to the wiki syntax, and then eventually transform some additional tasks like finding out the links on the documents so that you maintain the links between the document you're importing and some other documents you will import. This is the type of things we have done in the past. For example, we imported a very large website from the Lenovo community that was running on HTML connections, the previous Lotus IBM connections. And we were able to call the APIs of HTML and get all the documents, all the blog articles, and convert them to the equivalent XWiki documents. We've done also this for HTML imports. In some cases, you have to do two passes on the documents. So one, the first pass on all the HTML documents so that you can read them, find out what's going to be the page name of each of these documents once you're importing in XWiki. And so that you can create a map in order to know how to convert the links. And then you do a second pass and you're actually converting the syntax, converting HTML, and then modifying the links so that you can do the import. This works well. It requires some significant developer work each time. But we've been able to do this type of import between 10 to 30 days, depending on the complexity of the import and what you need to do. What's interesting with this method is that you can customize the import quite well and quite easily. For example, if you cannot import the whole document, if you get an HTML package which contains a little more than the documents themselves, like header, footers, and so on, you can strip them out before you actually run the HTML conversion. I would like now to explain a third type of import, which is the batch importing. So importing databases and automating file uploading and conversion in XWiki. So batch import is a tool that we have developed over time, which is quite a bit. Especially when we advanced it quite a lot when importing data from Lotus Notes for a customer. And what we did is that we converted the Lotus data to CSV files. We created very large CSV files and we stored all the attached files in the directory, but we referenced them in the CSV file. And then we used the batch importing tool to import the data and that's how the importing tool got a lot of functionality to support these type of imports. So what are the general principles of batch importing in XWiki using the XWiki batch importing tool? So first you install the batch import tool using the extension manager. Then you prepare data structure in XWiki. So you're going to use app within minutes to create a data structure. For example, it could be a client list or a project list. In the example I will show today, it's going to be a country database about COVID. And so I'm going to use app within minutes to do that. Second thing is I will prepare an Excel file of the data I'd like to import. In this case, I gathered an Excel file from a public database, an open data database, which contained data about the country's population size, GDP, but also some COVID information from the spring. Then I will also mention in the Excel file and the spreadsheet, I will put references to attached files that I would like to import as part of each line I'm importing. In this case, I will use a database of flags that are linked to the country code. And so I will have a file for each country, so BE.png, that will be a flag that I will find in the directory inside a zip file. Then I will upload the zip file on the server, upload the CSV file on the wiki and start mapping the data. And then I will have tools to preview and run the final import. So let's see exactly how this works. So first you create an app, you're going to use app within minutes. So we have a drag and drop interface to create fields. The app within minutes is actually a very powerful tool to help organize data in xwiki. And in this case, I create a country and I add fields. It took me about five minutes to do that, drag and dropping the different fields, the date field, two dates fields and about six or seven number fields. And I need to choose the right formats for the number, either these are longs, integers, or these are floats or even double. And for other fields, I choose the date field and I define the date format, whether it is a date time or date. Then I prepare my Excel file. So actually Excel or spreadsheets, in this case it's LibreOffice, is a very practical tool to manipulate the data quickly because you can change formulas and transform data. So initially I don't have a page name, I don't have the reference to the flag and I don't have the exact title I would like to have for every line. And so I create some formulas to allow this and I can do this cleanup. So I will create a field that will contain the page name, I will create a field that contains the space name in our case, its countries. You should always have a column which basically has the same value in it, which represents the space in which we're going to import the data. And then a column for the title. You can also clean up the numbers and the date data if necessary there in order to simplify the export. In the end I need either NixLS, 97 format or I need a CSV file. So both have their advantages and disadvantages with CSVs, you have to be careful with date because you're getting a string for the date and therefore you need to decide in which language or what the date format is exactly. So you upload your file to the wiki, you select it and then you will get a mapping screen. So the xwiki's batch import tool will read the columns from the CSV or the Excel file, choose the first line of the file as the name of the columns and ask you for it will read also the structure of data in which you'd like to import. So in my country application I'd like to import my data, it will ask me for every field which data I would like to take from the spreadsheet. So one thing that is interesting is that it's possible to run the import multiple times so you can actually import from multiple files in the same database potentially not overriding the data that you imported from the first file. If you're not using the field it won't touch the data in the wiki that is already there. So you choose and here it's almost one to one mapping, I need to make sure I put a document name, a space and a title, these are the three very important fields and then it's a one to one mapping of the data I'm importing the population and so on. Then I have some hidden settings which are a bit more tricky because they're not as well documented as the batch import. So you can find the batch import documentation on the extensions xwiki.org. Actually all the modules I've mentioned as I say are available in the extension repository of xwiki and the documentation is usually available on extensions xwiki.org. So these hidden settings they're concerned the flags. So I have a zip with all the flags, I'm uploading this to the server so I need an access to the server for this. I'm putting there somewhere on the file system that xwiki or the Java program will be able to access so you need to make sure about the rights. And you indicate the name of that zip file in the first field on the screen, it's a bit small here but if you zoom maybe you can see it properly, it's a pass and it ends with flag.zip. And you need also to tell that you want actually to do a file upload which is the last field on the screen where there is yes in the file upload field. This will make sure that when the import sees references to files in the spreadsheet it will upload these to the page that it just created. There is also an option to do file conversion. So in that case if it can find a word file it will convert the word file to wiki syntax and add it to the xwiki content field or to a text area field for example if you want to import in the blog. Finally in the past we've also customized these imports and we've been able to do pre and post treatment and one of them was to convert word files in PDF as it goes for every document. So you can also customize with code and potentially use any xwiki API to do pre and post treatment during this import. This is a bit more tricky and requires quite some experience with batch import tool. Finally you will be able to preview the results, this is actually quite important in order to avoid mistakes so you need to particularly be careful of the document full name. If you're not seeing the right data in the document full name or in the title it's probably not worth clicking the next button because you will have garbage in the data. You should also check your dates whether or not they are converted properly. This is a setting in the import to decide what is the format of your date. And then you get a screen on how to handle data de-duplication and also overwrite. So first set of settings that concern what to do if you find two lines that should end up in the same document in the same import process. And so do you ignore the second line, do you overwrite or do you update so you're going to merge the two lines into one line, the second one will win with its data. The second thing is whether what to do if there is already a page in the wiki not from this import but from a previous import and this is where you can make imports from multiple files and then you decide whether it's going to create a new document or if it's going to update the existing document or replace it. So for example in this method you can run multiple imports the same imports and make sure that you will delete the previous document before you import the new ones. So this is something to be careful when you're testing there might be already data and you can also delete all the data in app within minutes before you run a new import. And finally you can click next the import will take a bit of time you can then see the result. So before when you run the import or you run the preview you can look at the error messages there is extensive logging to really understand what is happening on all the lines. If you see an error you can try to understand exactly what is happening. So finally here I have an app within minute table with my countries with the flags of each country and I can filter and sort my data by the fields I imported so either by the population by the GDP or by the total cases of COVID that happened in the spring or by the total death that happened because of COVID in the spring. And I have imported data in my wiki. Well back to import is a very important tool to quickly do projects with Xwiki and do mass imports it's quite efficient. You've been importing gigabytes of data in the past with it. For example if you have attached files you can do a lot of things. There are some additional possibilities so as I said you can import and convert word to Xwiki you can even like import PowerPoints and it will attach them you can convert them to PDF you can also write custom processes at each stage of the import. You can also do converting of HTML or any other format to wiki syntax so if you're receiving some fields in the Excel file of HTML you can make conversion that requires scripting and you can work on the spreadsheet initially to prepare the data for the import. Lots of possibilities with batch import. For example one of the things you can do is that if you have a very large directory of files with a lot of folders well sometimes in the folder names you can find interesting information that you could make transform to metadata for example the name of the project. And so this is something you can do in Excel by manipulating the data in Excel and finally then you can import this in Xwiki. Now a part that is more advanced which is importing data from other wikis. So we will study a tool which is called Xwiki filter streams and today it's able to import from four wikis but it could also do other wikis potentially. So what is Xwiki filter streams? So you can find the documentation on extensions Xwiki org again I've put the link on this page. So it is a framework to import and export from and to Xwiki. It uses Xwiki syntax conversion capabilities but it also handles links between the pages that you're importing so that it can keep them. It can actually also import and export content in Xwiki so you can export using filter stream data from an old Xwiki instance and you can re-import it in a newer version of an instance. That is one way of doing upgrades when you don't want to keep everything from your previous instance and you want to start from scratch for your code itself. So what are the supported wikis for importing? So you have conference, media wiki, docu wiki and GitHub wiki. So some of these have been done through some of code. They don't have as much practice on the real wikis. We have quite a lot of practice for media wiki and conference. But one thing to understand is that there is no perfect import. There is always work to do on imports either manually when you come on the pages or eventually to improve the current tools. One of the reasons why it's difficult to really have perfect imports is that there can be custom features in the instances where the data comes from. There can be custom macros or a lot of macros and the importing tool will not support all the macros of conference and media wiki. There can also be absolute links. So links that are not recorded in the data as internal links between documents of the same system, but that are recorded as URLs and that can be problems. They can also be changes in recent versions of the tools from which we get the imports. There are many different versions of all the software conference media wiki and so it's not simple to follow everything that is different in all these instances. A little word about conference and why it's particularly interesting to talk about conference importing today. So first it's a wiki used by many developers including free and open source software fans. Many companies are using the Atlassian stack for development. So JIRA conference, Bitbucket and other tools. Those are using GitLab and we are big GitLab fans for development stack which is the most advanced open source software for developers. But a lot of companies are using the Atlassian stack and recently in December Atlassian has done a major shift away from on premise. So over the years Atlassian had changed their pricing policy regularly increasing the price. At xwiki we were already receiving calls from users trying to migrate from conference because they felt like that the price increase of the new version or of an increase of users was significant and not necessarily corresponding to what they would expect. And one of the difficulties with these migrations is that it's also a bit costly to do a migration. But in December what happened is that Atlassian has massively decided that the smaller customers should use their cloud including the open source users. Atlassian was giving list licenses of conference server or JIRA server to open source users for free and now they are only giving cloud instances for free. And you cannot get the server version of the Atlassian software for free. This is a major blow to controlling your own data because especially for smaller companies because you have to go for the data center offer which is starting at 20k dollars and is not suitable for smaller companies in terms of price. So this has actually made that xwiki is receiving more calls and more users and companies are looking for alternate solutions than Atlassian conference or Atlassian JIRA. So we believe xwiki is a great solution as an alternative to conference and it was already but we are seeing more customers and we are also seeing more participation to help us ease the migration path between conference and xwiki and we hope this is going to help us a lot in the future. So how does the conference import process work? So first we will use the filter stream from xwiki. We are going to export conference from conference itself as an XML export. It's going to give us a zip file. It can be pretty big. We then run the conference filter in xwiki's filter stream. This is going to import us every page in separate pages but all flat because conference doesn't have a system of nested pages. Everything is flat in conference. They use a parent-child relationship which is something we were using two a couple years ago before version 7 of xwiki. And at that time we migrated to a nested spaces system. And so this is why we need a second tool to migrate from the flat organization to the nested spaces migrator to the nested organization in xwiki and recover the data in the parent-child relationship that exists in the conference data. So in the end we'll get the same tree structure that we get from conference. We'll get it in xwiki 2. And finally there is a fourth step which is today the most complex one and it's not perfect is that we need to look at macros that were used in conference and see if we have alternate macros for these or if we need to create new macros. I'm not going to go too much into detail there because this is something we're still working on how to support more macros of conference when importing in xwiki. So first exporting from conference. So this happens in the space setting, content tool, export of conference. The user should choose XML output. It's also possible to limit what you export so select specific pages to export. It's important to note that if you're not an admin it might lead to a partial export and this can be an issue. Then you go in xwiki in the filter interface. You choose conference XML as the import source and then you point to the zip file that you need to have uploaded to the server in an area and with rights available so that xwiki can access it. So there is not a lot of options to fill in on the first screen. All this is documented on the conference XML page to make sure to choose the right parameters. The second part of the screen is the target parameter. You should choose xwiki instance. There are options to choose with and without version and then there are some other options which can have an impact on so who is the user that imported in xwiki, what common to put for that import and then you can run the input. This will lead to a screen with a progress bar and every page is being imported. It's possible to see errors. Normally errors should not stop the process of importing unless it's a major error that we don't know yet and the process will go to the end. In the meantime it's suggested not to play with the wiki and not interfere especially with the data being imported. There is another aspect to know that in case the import doesn't go till the end there could be issues with space because the data will not be deleted from the temp directory so if you run multiple imports and they all end up in error you might end up filling in the disk of your server. You need to be careful and make sure you have enough space on the server for at least three times the size of the conference import. If you have a very big import of 5 gigabytes it will take the zip file and then it will export the data and zip it in the temp directory and then it will import it in xwiki and the xwiki database. You need all that space so you need to make sure you have enough space on the server to process all this and you need to look at your temp directories in case there is an issue. If you restart the wiki then the temp data will be deleted. Finally once you have finished the first phase of the import which is importing the pages in the flat space you can run the nested pages migrator. Most options need to be unchecked. You need to choose the space that you want to convert. Usually this is the conference space key in our case it was a two letter cte and then you can compute the plan. The compute plan is not going to make any changes it is going to show you what it plans to do. In this case it is also where it might show you some errors and two long page names. Xwiki has a limit of 250 characters for the full page name including the whole tree. If you have a very big tree and in the way you have some pages that have very long titles knowing that conference is using the same data for the page name and the page title you might end up with full page names that are too long. What happens in this case is that you actually need to rename some pages not all of them in the tree because for example if your very long page name is at the third level of the tree and you have six levels of the tree the whole pages under will also be too long but you probably need only to rename the one at the top of the tree to do that. You need to use the normal Xwiki feature to rename the page. Find the page that has the too long page name. Once you've renamed it you can just redo the compute plan and see if you still have errors. It can fix multiple errors in one batch if you're renaming the right page. Once you have no errors of too long page name then you can just perform the conversion and the execute plan. It's important to understand that you cannot roll back from this conversion. This is the one-time conversion. Cannot come back to it so you need to restart from scratch your import if you're not doing the right thing here. The second aspect is about the too long page names. You don't need to rename the title. You just need to rename the page name. This is something that is not very easy for somebody that doesn't know Xwiki but if you know Xwiki it is not very difficult to fix when you have the issue of too long page name. In the end you can check the result and you should get a similar hierarchy than in conference. First you have a progress bar again and you can see the conversion that is happening. It can be long if there is a big hierarchy to move. And finally you can see your conference hierarchy on the left and you can see your pages in the right hierarchy on the right. However, it's not really finished here because the question that arises is what about macros? And so what if you see red boxes and you see a lot of them? Well, you might see a lot of red boxes if in your company there is a big practice of using a lot of macros. And this can happen because a conference tends to propose you templates which includes a lot of macros. We don't necessarily think that it's always very good to use too many macros. The value is the content, not necessarily the macros or the format that you've used for that content. But this is the way it is. You might have a lot of macros in your content. So you will have some red boxes when the macros don't exist in Xwiki and there might be work to either bridge these macros to Xwiki macros, which will require what we call wiki macro development. So in Xwiki it's possible to write macros using scripting. You can find some documentation about it. And otherwise you will need to wait for some new macros that we will build. Given the number of people asking about these conversions where we started working on trying to create more macros and support more macros, now we will be making some macros paying because it's significant work to support the companies wanting to migrate. And it is also necessary that more users support the Xwiki software so that it can continue to progress. Even if the macros are paying, everything is open source and you can find the source of our macros in our Xwiki SIS GitHub where we publish these macros. We also building a tool to list macros and count them and look at what the most important macros are. So if you're interested and if you would like to help us and if you're doing test imports you could try to tell us how many macros you have and which ones so that we can focus on the more important macros to migrate. So this tool will be available soon. We already have it. We can pass it to certain users if they want to count the macros used after importing their conference data in Xwiki. So the conference importing is a work in progress. There is still a lot of work. We'd like to support more content and we recently, very recently received a pull request by a contributor that supports that fixed issues that were reported already in our ticket tracking and for example, importing comments and labels is actually now will be supported once we merge this pull request if everything is fine with it. We'd like to support more macros and we need to continue to maintain the importer for future conference evolution. This will continue to evolve the data center version, the server until they stop it and also their cloud version. And the conference export format is not really officially documented. It's more like a database export than it is a document by document export. And so it's not always very easy to fully understand what's happening in that export format. If you would like to migrate, keep in mind that sponsoring the work is possible and you can also report issues in our ticket tracking. The more information we have, the better we can understand what is going on and improve the importer. We're actively working on this as I mentioned so that we can provide the best solutions for migration. Another aspect before going for the conclusion is that the same tool can also work for Media Wiki, for GitHub Wiki. It's possible to import the GitHub Wiki directly by giving the URL and the importer will create the pages in XWiki. And for Media Wiki, you need to export the file and same thing, we can then import the data. The conclusion is that we have actually many tools available for important content in XWiki. Over the years, these tools have been improved and refined. There's quite some solution. They generally need quite some expertise to run them. It's not so simple. They've been improved over time through service missions from XWiki SIS to its client. And I'd like to thank all the users and clients that have supported us to build these migration tools and importing tools. I'd like also to thank the contributors who have been reporting issues and also wrote code. We hope that these tools will continue to improve and particularly the conference one with more users coming to XWiki. Thank you very much. So I'll be able to take some questions or continue to take some questions in the chat. This talk has been fully built with open source. It's been written and recorded on the Cube OS operating system and that's using Fedora too. The presentation software is using Cripad only of its slide in alpha version, not available on Cripad FR in order to test the advancement of the solution. The screenshots have been done with Shotwell and the recording with open broadcasting software. And of course, it's using XWiki and the XWiki extension. Obviously the conference is not open source and it's the only software in this case that was not open source for these examples of this presentation. Thank you very much and have a great FOSDEM. We hope to see you for real next year in Brussels and that COVID will be behind us. Thank you very much.
information in time, especially when you have new Atlassian Confluence. We're going to talk a importing content. When it comes to HTML, you can the links between the document you're importing that we converted the Lotus data to CSV files. We importing in XWiki using the XWiki batch or a project list. In the example I will show can actually import from multiple files in the are looking for alternate solutions than you have finished the first phase of the import and count them and look at what the most this tool will be available soon. We already have actually many tools available for important