Collaborating to describe datasets using Frictionless Data schemas: schema-collaboration
Pina Estany, Carles
schema-collaboration is a tool that helps data managers and researchers to collaborate on documenting datasets using Frictionless Data schemas. It uses Frictionless Data Package Creator and allows the collaborators to create and share dataset schemas, edit them, post messages and export the schemas in different formats (text, Markdown, PDF). The tool is implemented in Python and Django. The talk will consist of a brief explanation of Frictionless Data schemas, how data managers work with researchers and then I shall do a demo of how the tool can be used.
 Hello and welcome to the schema collaboration talk here at FOSDEM. This talk will have about six minutes more or less of presentation slides and then it will have a demo of schema collaboration. So something so with me, I'm a soft engineer, I'm not a scientist or data manager, we'll be talking about scientists and data managers later on. I do applications on Python and Django currently, before in other technologies. These applications, schema collaboration is a web application developed in Django. And for many years I've been writing software that is used by scientists, researchers, data managers and all this type of community. I'm also, at the moment I'm a part-time software engineer at the Swiss Poly Institute and is in doing this contact that I've missed having something like schema collaboration. And well, here it is. Also I'm a serial FOSDEM attendee for many years. So it is frictionless data. Frictionless data is a project from the Open Knowledge Foundation and it has a set of JSON schemas that help documenting data. And when I say data here, I mean either data packages, so a set of files. It can say the size of the files, hashes, titles, descriptions of these files, author, license, keywords and it also helps discovering the contents of CSV files, which columns contain the CSV file, which kind of format the date, time is, or other information. Also it helps, it has a set of tools to help describing, validating and working with data. So for example, we have here data package that JSON, this is what schema collaboration helps creating. And here we can see that it has the file is a data package, it has a list of resources, each one is a file or it can be a local file, it can be a remote file, it's in the path line and then they have a name, they have a title, description, format and in this case this example has a tabular data resource profile, because it's a CSV file that we'll see soon. And as I said, it has a list of files here. The CSV file, we know more or less what it looks like, but it has column names, date, time, VTC, latitude, longitude and it has data for each of these files. This is a simplified version of the file that I'm linking below. Here there's a table schema that JSON, in this table schema there's the list of files, fields and for example if we get the date, time, column name, this is represented in the name property here or we can get the date, time, format and we can specify the format that comes here, same for latitude, has a name or the units would be the renors. I've also specified here the maximum and minimum numbers of latitude. All of these can be validated, so we have tools that if a row has a latitude that is bigger than 90 or minimum than minus 90, frictionless data tools would say that this is a not valid file or not valid schema. So to create frictionless files, we can read them by hand, JSON files or we could use frictionless pi, we pass a file and it creates the schema of this file for this file or we can use data package creator work interface that is another way of working with data packages and looks like this. The data package creator is used by schema collaboration world series in a bit. So before and after or without and with. On the left we can see the data manager is sending emails to the scientists asking to describe certain fields and the data package in general. They collaborate with each other and the data manager would create the JSON files maybe with the data package creator by hand or by some tools. The data manager might be working with many scientists at the same time so I have two here but maybe it will be 15 or 20 and there are many data packages being documented at the same time. With the schema collaboration, all the interaction would happen on the schema collaboration software so it's easier to keep tabs on what's happening, who's waiting for who and the scientists have an easier way to input data and see what's missing and the results of everything in real time. You can export in JSON or you can export smart down and PDF as well. So next let's see the technologies very quickly. It's done in Python and Django. It can work with SQLite or MariaDB as a database and can be deployed using Docker or other ways that Django applications are usually deployed with. Here we'll see a demo in the next step. So now let's move to the demo time. I'll be using this URL. You can go there as well. That's a public open instance so anyone can test the system. You can do different things here but let's start pretending that we are the data manager and so we go to the management area. And the username is Alice the data manager and the password is Cliptonless. I talk often about the data management, data manager. This can be anyone who wants to set up the collaboration initial process. It doesn't need to be any data manager. So here we see a list of data packages that are being worked on but for this demo what I'm going to do is create a new data package. And it opens straight away the data package creator but this one has these external buttons save to server, lock from server and exit package creator and the standard data package creator doesn't have. For example what I'm going to do is create a package that will be, the name will be CostimTox, the title CostimTox as well. And for example is a tabular data package and I'll say the list of CostimTox. And the home page for example, this is optional, I'll choose a license, CCY consider it fills this and I could add keywords here. I'll add a resource and here I can say that this is a tabular data resource. I can say that the encoding is UDF-8 for example and the CSV and this is the CostimTox file for example. I am not a professional data manager as you can see but I cannot fill here. This is a fill name so what's in the CSV and I'll say talk or give a top name, the title the center of the talk and the description the person that presented the talk to link with this person. The Data Type is a string here and no default format that we can choose so we would choose other things. And I'll add only another one, I'll do actually the date start, start time, I'll have the start time here, the description time that the talk starts, the data manager which is my role at this moment knows this as date time and the format I'll leave it as default now. And I'll leave this blank, I'll only know this number but let's say that I don't know what is this field exactly. When I finish I can do a save to server, it's saved, I can exit the perpetrator and now we have here a new dataset. What I'm going to do now in the options is to manage this data package and for example in edit manage I'll say that Bob the scientist is going to help these two Alice and Bob the collaborators. Here in this view I can go to the preview of a collaborator view or I can copy the link, I'll copy the link now, we'll see what's this in a moment and I could download this in the future less schema in Marginal or in PDF. So now I'll open the climate window and I'll paste this link. Now I'm the scientist in this place, I can add a comment, for example I can say Bob please add the description of all the colors. And I add a comment here, it's a comment saved. Now I go to the copy link and I'll go into a new tab, this could be a different browser that needs to be logged in. The URL has the UAD so that's how we know that who has a UAD can access this. So now I'm Bob here and for this package it says okay, there's a comment here, the description of all the colors, okay I'll press edit and I see that I need to add the subscription and I'll say that this is the number, oh we said the DvS, number of people that started listening to the talk and the title it can be please us or something. Plus it's a number that actually is an integer, numbers in fictionalized floats so it's with a half people. So I'll save to server and then I'll exit the data package creator and say hi Alice, done, thank you very much. So I add a comment here, done and Alice can go to the data packages and see that there have been some change here so I'll edit the data package, oh fantastic Bob did what he needed to do so I'll exit this in order to manage the data package and I'll say that this data package is completed. I'll save this and actually if I want I can download the PDF from here, I'll open this PDF using Firefox and we would have a bit of, well I didn't add contributors keywords version but we can see that we have the Fossum talks that has these resource with this information and has these fields for example and that's the very quick demo that they had for today. Thank you very much for listening, if you have any questions I'll be happy to answer them next. See you.
 Hello and welcome to the schema collaboration talk here at FOSDEM.  This talk will have about six minutes more or less of presentation slides and then it  will have a demo of schema collaboration.  So something so with me, I'm a soft engineer, I'm not a scientist or data manager, we'll  be talking about scientists and data managers later on.  I do applications on Python and Django currently, before in other technologies.  These applications, schema collaboration is a web application developed in Django.  And for many years I've been writing software that is used by scientists, researchers, data  managers and all this type of community.  I'm also, at the moment I'm a part-time software engineer at the Swiss Poly Institute and is  in doing this contact that I've missed having something like schema collaboration.  And well, here it is.  Also I'm a serial FOSDEM attendee for many years.  So it is frictionless data.  Frictionless data is a project from the Open Knowledge Foundation and it has a set of JSON  schemas that help documenting data.  And when I say data here, I mean either data packages, so a set of files.  It can say the size of the files, hashes, titles, descriptions of these files, author,  license, keywords and it also helps discovering the contents of CSV files, which columns contain  the CSV file, which kind of format the date, time is, or other information.  Also it helps, it has a set of tools to help describing, validating and working with data.  So for example, we have here data package that JSON, this is what schema collaboration  helps creating.  And here we can see that it has the file is a data package, it has a list of resources,  each one is a file or it can be a local file, it can be a remote file, it's in the path  line and then they have a name, they have a title, description, format and in this case  this example has a tabular data resource profile, because it's a CSV file that we'll see soon.  And as I said, it has a list of files here.  The CSV file, we know more or less what it looks like, but it has column names, date,  time, VTC, latitude, longitude and it has data for each of these files.  This is a simplified version of the file that I'm linking below.  Here there's a table schema that JSON, in this table schema there's the list of files,  fields and for example if we get the date, time, column name, this is represented in the  name property here or we can get the date, time, format and we can specify the format  that comes here, same for latitude, has a name or the units would be the renors.  I've also specified here the maximum and minimum numbers of latitude.  All of these can be validated, so we have tools that if a row has a latitude that is  bigger than 90 or minimum than minus 90, frictionless data tools would say that this is a not valid  file or not valid schema.  So to create frictionless files, we can read them by hand, JSON files or we could use frictionless  pi, we pass a file and it creates the schema of this file for this file or we can use data  package creator work interface that is another way of working with data packages and looks  like this.  The data package creator is used by schema collaboration world series in a bit.  So before and after or without and with.  On the left we can see the data manager is sending emails to the scientists asking to  describe certain fields and the data package in general.  They collaborate with each other and the data manager would create the JSON files maybe  with the data package creator by hand or by some tools.  The data manager might be working with many scientists at the same time so I have two  here but maybe it will be 15 or 20 and there are many data packages being documented at  the same time.  With the schema collaboration, all the interaction would happen on the schema collaboration  software so it's easier to keep tabs on what's happening, who's waiting for who and the  scientists have an easier way to input data and see what's missing and the results of  everything in real time.  You can export in JSON or you can export smart down and PDF as well.  So next let's see the technologies very quickly.  It's done in Python and Django.  It can work with SQLite or MariaDB as a database and can be deployed using Docker or other ways  that Django applications are usually deployed with.  Here we'll see a demo in the next step.  So now let's move to the demo time.  I'll be using this URL.  You can go there as well.  That's a public open instance so anyone can test the system.  You can do different things here but let's start pretending that we are the data manager  and so we go to the management area.  And the username is Alice the data manager and the password is Cliptonless.  I talk often about the data management, data manager.  This can be anyone who wants to set up the collaboration initial process.  It doesn't need to be any data manager.  So here we see a list of data packages that are being worked on but for this demo what  I'm going to do is create a new data package.  And it opens straight away the data package creator but this one has these external buttons  save to server, lock from server and exit package creator and the standard data package  creator doesn't have.  For example what I'm going to do is create a package that will be, the name will be  CostimTox, the title CostimTox as well.  And for example is a tabular data package and I'll say the list of CostimTox.  And the home page for example, this is optional, I'll choose a license, CCY consider it fills  this and I could add keywords here.  I'll add a resource and here I can say that this is a tabular data resource.  I can say that the encoding is UDF-8 for example and the CSV and this is the CostimTox file  for example.  I am not a professional data manager as you can see but I cannot fill here.  This is a fill name so what's in the CSV and I'll say talk or give a top name, the title  the center of the talk and the description the person that presented the talk to link  with this person.  The Data Type is a string here and no default format that we can choose so we would choose  other things.  And I'll add only another one, I'll do actually the date start, start time, I'll have the  start time here, the description time that the talk starts, the data manager which is  my role at this moment knows this as date time and the format I'll leave it as default  now.  And I'll leave this blank, I'll only know this number but let's say that I don't know  what is this field exactly.  When I finish I can do a save to server, it's saved, I can exit the perpetrator and now  we have here a new dataset.  What I'm going to do now in the options is to manage this data package and for example  in edit manage I'll say that Bob the scientist is going to help these two Alice and Bob  the collaborators.  Here in this view I can go to the preview of a collaborator view or I can copy the link,  I'll copy the link now, we'll see what's this in a moment and I could download this in the  future less schema in Marginal or in PDF.  So now I'll open the climate window and I'll paste this link.  Now I'm the scientist in this place, I can add a comment, for example I can say Bob  please add the description of all the colors.  And I add a comment here, it's a comment saved.  Now I go to the copy link and I'll go into a new tab, this could be a different browser  that needs to be logged in.  The URL has the UAD so that's how we know that who has a UAD can access this.  So now I'm Bob here and for this package it says okay, there's a comment here, the description  of all the colors, okay I'll press edit and I see that I need to add the subscription  and I'll say that this is the number, oh we said the DvS, number of people that started  listening to the talk and the title it can be please us or something.  Plus it's a number that actually is an integer, numbers in fictionalized floats so it's with  a half people.  So I'll save to server and then I'll exit the data package creator and say hi Alice,  done, thank you very much.  So I add a comment here, done and Alice can go to the data packages and see that there  have been some change here so I'll edit the data package, oh fantastic Bob did what he  needed to do so I'll exit this in order to manage the data package and I'll say that  this data package is completed.  I'll save this and actually if I want I can download the PDF from here, I'll open this  PDF using Firefox and we would have a bit of, well I didn't add contributors keywords  version but we can see that we have the Fossum talks that has these resource with this information  and has these fields for example and that's the very quick demo that they had for today.  Thank you very much for listening, if you have any questions I'll be happy to answer  them next.  See you.
open_research_schema_collaboration_0_179.png open_research_schema_collaboration_180_2003.png open_research_schema_collaboration_2004_6371.png open_research_schema_collaboration_6372_10055.png open_research_schema_collaboration_10056_11639.png open_research_schema_collaboration_11640_11975.png open_research_schema_collaboration_11976_12047.png open_research_schema_collaboration_12048_12923.png open_research_schema_collaboration_12924_14627.png open_research_schema_collaboration_14628_16343.png open_research_schema_collaboration_16344_16619.png open_research_schema_collaboration_16620_17219.png open_research_schema_collaboration_17220_17435.png open_research_schema_collaboration_17436_18071.png open_research_schema_collaboration_18072_18371.png open_research_schema_collaboration_18372_20051.png open_research_schema_collaboration_20052_20795.png open_research_schema_collaboration_20796_21491.png open_research_schema_collaboration_21492_21551.png open_research_schema_collaboration_21552_21599.png open_research_schema_collaboration_21600_22055.png open_research_schema_collaboration_22056_22665.png
Hello and welcome to the schema collaboration talk here at FOSDEM. This talk will have about six minutes more or less of presentation slides and then it will have a demo of schema collaboration. So something so with me, I'm a soft engineer, I'm not a scientist or data manager, we'll be talking about scientists and data managers later on. I do applications on Python and Django currently, before in other technologies. These applications, schema collaboration is a web application developed in Django. And for many years I've been writing software that is used by scientists, researchers, data managers and all this type of community. I'm also, at the moment I'm a part-time software engineer at the Swiss Poly Institute and is in doing this contact that I've missed having something like schema collaboration. And well, here it is. Also I'm a serial FOSDEM attendee for many years. So it is frictionless data. Frictionless data is a project from the Open Knowledge Foundation and it has a set of JSON schemas that help documenting data. And when I say data here, I mean either data packages, so a set of files. It can say the size of the files, hashes, titles, descriptions of these files, author, license, keywords and it also helps discovering the contents of CSV files, which columns contain the CSV file, which kind of format the date, time is, or other information. Also it helps, it has a set of tools to help describing, validating and working with data. So for example, we have here data package that JSON, this is what schema collaboration helps creating. And here we can see that it has the file is a data package, it has a list of resources, each one is a file or it can be a local file, it can be a remote file, it's in the path line and then they have a name, they have a title, description, format and in this case this example has a tabular data resource profile, because it's a CSV file that we'll see soon. And as I said, it has a list of files here. The CSV file, we know more or less what it looks like, but it has column names, date, time, VTC, latitude, longitude and it has data for each of these files. This is a simplified version of the file that I'm linking below. Here there's a table schema that JSON, in this table schema there's the list of files, fields and for example if we get the date, time, column name, this is represented in the name property here or we can get the date, time, format and we can specify the format that comes here, same for latitude, has a name or the units would be the renors. I've also specified here the maximum and minimum numbers of latitude. All of these can be validated, so we have tools that if a row has a latitude that is bigger than 90 or minimum than minus 90, frictionless data tools would say that this is a not valid file or not valid schema. So to create frictionless files, we can read them by hand, JSON files or we could use frictionless pi, we pass a file and it creates the schema of this file for this file or we can use data package creator work interface that is another way of working with data packages and looks like this. The data package creator is used by schema collaboration world series in a bit. So before and after or without and with. On the left we can see the data manager is sending emails to the scientists asking to describe certain fields and the data package in general. They collaborate with each other and the data manager would create the JSON files maybe with the data package creator by hand or by some tools. The data manager might be working with many scientists at the same time so I have two here but maybe it will be 15 or 20 and there are many data packages being documented at the same time. With the schema collaboration, all the interaction would happen on the schema collaboration software so it's easier to keep tabs on what's happening, who's waiting for who and the scientists have an easier way to input data and see what's missing and the results of everything in real time. You can export in JSON or you can export smart down and PDF as well. So next let's see the technologies very quickly. It's done in Python and Django. It can work with SQLite or MariaDB as a database and can be deployed using Docker or other ways that Django applications are usually deployed with. Here we'll see a demo in the next step. So now let's move to the demo time. I'll be using this URL. You can go there as well. That's a public open instance so anyone can test the system. You can do different things here but let's start pretending that we are the data manager and so we go to the management area. And the username is Alice the data manager and the password is Cliptonless. I talk often about the data management, data manager. This can be anyone who wants to set up the collaboration initial process. It doesn't need to be any data manager. So here we see a list of data packages that are being worked on but for this demo what I'm going to do is create a new data package. And it opens straight away the data package creator but this one has these external buttons save to server, lock from server and exit package creator and the standard data package creator doesn't have. For example what I'm going to do is create a package that will be, the name will be CostimTox, the title CostimTox as well. And for example is a tabular data package and I'll say the list of CostimTox. And the home page for example, this is optional, I'll choose a license, CCY consider it fills this and I could add keywords here. I'll add a resource and here I can say that this is a tabular data resource. I can say that the encoding is UDF-8 for example and the CSV and this is the CostimTox file for example. I am not a professional data manager as you can see but I cannot fill here. This is a fill name so what's in the CSV and I'll say talk or give a top name, the title the center of the talk and the description the person that presented the talk to link with this person. The Data Type is a string here and no default format that we can choose so we would choose other things. And I'll add only another one, I'll do actually the date start, start time, I'll have the start time here, the description time that the talk starts, the data manager which is my role at this moment knows this as date time and the format I'll leave it as default now. And I'll leave this blank, I'll only know this number but let's say that I don't know what is this field exactly. When I finish I can do a save to server, it's saved, I can exit the perpetrator and now we have here a new dataset. What I'm going to do now in the options is to manage this data package and for example in edit manage I'll say that Bob the scientist is going to help these two Alice and Bob the collaborators. Here in this view I can go to the preview of a collaborator view or I can copy the link, I'll copy the link now, we'll see what's this in a moment and I could download this in the future less schema in Marginal or in PDF. So now I'll open the climate window and I'll paste this link. Now I'm the scientist in this place, I can add a comment, for example I can say Bob please add the description of all the colors. And I add a comment here, it's a comment saved. Now I go to the copy link and I'll go into a new tab, this could be a different browser that needs to be logged in. The URL has the UAD so that's how we know that who has a UAD can access this. So now I'm Bob here and for this package it says okay, there's a comment here, the description of all the colors, okay I'll press edit and I see that I need to add the subscription and I'll say that this is the number, oh we said the DvS, number of people that started listening to the talk and the title it can be please us or something. Plus it's a number that actually is an integer, numbers in fictionalized floats so it's with a half people. So I'll save to server and then I'll exit the data package creator and say hi Alice, done, thank you very much. So I add a comment here, done and Alice can go to the data packages and see that there have been some change here so I'll edit the data package, oh fantastic Bob did what he needed to do so I'll exit this in order to manage the data package and I'll say that this data package is completed. I'll save this and actually if I want I can download the PDF from here, I'll open this PDF using Firefox and we would have a bit of, well I didn't add contributors keywords version but we can see that we have the Fossum talks that has these resource with this information and has these fields for example and that's the very quick demo that they had for today. Thank you very much for listening, if you have any questions I'll be happy to answer them next. See you.
and then it will have a demo of schema data managers later on. I do applications on Python and Django currently, before in other software that is used by scientists, researchers, schemas that help documenting data. And when I and Django. It can work with SQLite or MariaDB as a database and can be deployed using Docker or description the person that presented the talk to link with this person. The Data Type is a string