From outer space to your browser
Tawalika, Carmen (mundialis)
With the Copernicus programme of the European Union everybody can access remote sensing data produced by the so called "Sentinels", the satellites designed for observing the earth from space. The data can be accessed in raw state or via Copernicus Services which are dedicated to a certain topic. But if you would like to extract certain information, you need to process the raw data. How would it be possible to use open source software to process the raw data and then make it available for further use, e.g. in the web by using open geospatial web standards? This talk presents a webmapping client containing footprints of all currently available Sentinel 2A scenes which you can filter and select and send a job which processes this scene including download, atmospheric correction and several image processing algorithms (e.g. NDVI). When the job is done the processed scene will be loaded in the web client, published as an OGC web service which makes it reusable elsewhere. The client is built using OpenLayers3, ExtJS6, GeoExt3 and BasiGX. The processing is done with sen2cor, GDAL and GRASS GIS. The product is published with GeoServer.
 It is better. Yes. My name is Carmen and I want to show you a map client where we get Sentinel to data. I don't have to introduce Copernicus anymore. Andrea Faisback did it in the opening session. And let's start. First I work at Mundialis and we are located at Bonn. So we didn't have to travel as far as you probably. And you can see here what we are doing. And we do the Mundialis Gambia and we also have another project. It is called Mundialis Art. And you can see it at Mundialis.de and it presents how beautiful the earth is and the imagery. So the Copernicus Broadjam provides many data. They have services, six different ones. But there is also raw data and normal users might not know how to process them. So we found a way to download it. They provide an app you can request and you can filter and search and we use this one to get the data that we like. And also we provide the preview images for all the Sentinel to data from the Copernicus Broadjam and we store them in our database. Only the bounding boxes and metadata for it. And then there is another open source tool. It is called CentroCore. This does the atmospheric correction for Sentinel to data. And this helps a lot because we can use it. And also there was a major release for another version. And before it took eight hours to process one scene, about seven gigabytes, just to the atmospheric correction. And with the newer version it is possible to do sub-scenes at the same time so it's much quicker. And then we do some algorithms with GDAL and also with Kraskis. And then we take some from the raw data and you can choose what kind of process you like to have. And then it gets processed. And at the end you will get an image out which is optimized for the web. And then we will publish it with GeoServer. We also have different processing chains for GIS because for the web we make it very small so that it can be displayed quick. And we also have different processing chains but they are not available in the web client at the moment. So what happens when the data is at GeoServer? It is put into a web client. And this web client is built with open source software as well. It contains basics, open layer 3, GeoX and XJS. And now let's look at the client. It is here. And there you can already see some of the preview images. We don't have such a smart solution as they presented before. We show all the preview images but we limited a little bit for the user because you can't filter and see all of them at the same time. And these are the paths that the satellite took probably yesterday. And here you can filter first time. You can see a time range and then it loads. And then you see much more preview images. And when you can also change the cloud coverage. So you see now when you go for more cloud coverage, the time gets less. When you go to zero, you are allowed to take the whole range of the whole time because it's not so many cloud free images at the same time. And then you can see and if you click on it, then you see the cloud percentage. It's zero because I choose zero. And then you can get all the information. And this is for the one spot that I clicked on, all the images which are available from this satellite. And then you get some information like the name, the UUID, the size, you can see here it's six gigabytes. And the date. Okay and then I wanted to show you processing but as I said before it takes a lot of time. So I prepared ID. So you can also filter by ID if you know what you want. It is possible. Search. So I switch the preview layer off. I have my search result here. And then you can order a process scene. I will choose a composite. You can also choose some indices, for example, the normal life difference vegetation index. And for the composites, we did some before. Like this is true color composite. Take some of the bands for Sentinel-2 data. And color infrared composite is also very popular. And we did some more. And I will order this one. And then it sends. And then at one point you should get a success message. Okay let me show you more features of the client. You can also see some of the footprints here in the client which are already processed because each one which is processed will be published in the client. And for example, we process Buenos Aires. And then you can slide through all the composites which are already processed. And also here you can see some of the indices we calculated. And this is the NDVI layer. We don't have it for every footprint but for some and it's growing. There it is. And normally this will take hours but for you to see what would happen. I prepared a preview. This will be the email you will get with a link for the web service at the moment only possible to view it in the client. And you will get the link like this. And then when you open it you go directly to the scene that you ordered in the composite. Do I have some time left? Oh okay. Then I was very quick. Yes. Maybe we can start earlier with the questions or I can show some of the features of the client. For example, if you look at the composites and you see they are very beautiful and you can have a look at them and on the other hand for the indices you can find out and get value out of the data. Yeah. And yeah. Okay. Let's start with the questions. Yeah, that's all Sentinel 2. And no they don't. They are only for the land. Which areas? It is. Yes. Yes. Can you show them by area? Yes. You can see it here. And when you. Yes, but they are only for the preview images and you can choose between any and when you select the data then you only get the result in this box. So you have to choose one by one at the moment. Yeah. When you click the order button then the job will be sent to a server and we have, we are always looking for server architecture for power and we have somebody who we can calculate at the moment for one week until August. So now you can order products and after one week then we talk to them and we see. So if you know how to get hardware power then we would be very happy. And then we have one server which does the, which receives all the jobs and it sends to different nodes. So we already set it up some servers where everything is, yes, where everything is installed. No, no, it's server based. So I see that you can search for data. But do you use some kind of OGC standards? For example, for catalog, you have CSWP for processing LWBS. So you make use of these standard complaints of, use the real standards to make your processing call or to research for the data. The OGC standards we only use for publishing the web services view, but we don't, yeah, not a catalog service yet. No, no. I see that you have built this platform based on open source on that. But I imagine that while building a new, maybe like you want to print equal or some code to do there together, then you have built this platform. Let's see. Yeah. So the question is, are you kind of also making open source even this blue code, the code that combines all this open source? I haven't done it yet. Probably. And you written download data from the central side up because it is not the same way. Yes. Yes. Yeah. Yeah. That's as it is for now because we can store. Yes. Yes, which is the API up from Isiah. Yes, we haven't implemented it yet. But with the email you get, we want to provide a link where you can download the image as well, probably as a WMS. Sorry. As we use the server, you can use any form of the geo server supports. We only have the predefined ones, which there was a form when you can click on the scene. I can show you. Like we have composites, predefined and also indices. And at the moment we have the vegetation index and some others. But at the moment it's not possible to put your own algorithm to it. It's not yet open. And Kraskis, mostly Kraskis, yes. And a little bit of GDAL as well. And the web interface. Small comments. You were talking about the atmospheric correction. The product, the whole product, it's in the interface is the result of the send to power processing. Or is it? Is it direct? This one? No, this one is a composite. But I was not sure that how you apply the send to power correction to all. If they are. If one scene is ordered, then we apply it to all of them. We have another client for it, but it's not in this one. But if you like, I can show it. This is just a demo client. I think sorry, the data is lost somewhere. But here it would be possible to slide through the time. Two filters, yes. For the preview images. Okay, any more questions? Okay.
 It is better.  Yes.  My name is Carmen and I want to show you a map client where we get Sentinel to data.  I don't have to introduce Copernicus anymore.  Andrea Faisback did it in the opening session.  And let's start.  First I work at Mundialis and we are located at Bonn.  So we didn't have to travel as far as you probably.  And you can see here what we are doing.  And we do the Mundialis Gambia and we also have another project.  It is called Mundialis Art.  And you can see it at Mundialis.de and it presents how beautiful the earth is and the  imagery.  So the Copernicus Broadjam provides many data.  They have services, six different ones.  But there is also raw data and normal users might not know how to process them.  So we found a way to download it.  They provide an app you can request and you can filter and search and we use this one  to get the data that we like.  And also we provide the preview images for all the Sentinel to data from the Copernicus  Broadjam and we store them in our database.  Only the bounding boxes and metadata for it.  And then there is another open source tool.  It is called CentroCore.  This does the atmospheric correction for Sentinel to data.  And this helps a lot because we can use it.  And also there was a major release for another version.  And before it took eight hours to process one scene, about seven gigabytes, just to  the atmospheric correction.  And with the newer version it is possible to do sub-scenes at the same time so it's  much quicker.  And then we do some algorithms with GDAL and also with Kraskis.  And then we take some from the raw data and you can choose what kind of process you like  to have.  And then it gets processed.  And at the end you will get an image out which is optimized for the web.  And then we will publish it with GeoServer.  We also have different processing chains for GIS because for the web we make it very small  so that it can be displayed quick.  And we also have different processing chains but they are not available in the web client  at the moment.  So what happens when the data is at GeoServer?  It is put into a web client.  And this web client is built with open source software as well.  It contains basics, open layer 3, GeoX and XJS.  And now let's look at the client.  It is here.  And there you can already see some of the preview images.  We don't have such a smart solution as they presented before.  We show all the preview images but we limited a little bit for the user because you can't  filter and see all of them at the same time.  And these are the paths that the satellite took probably yesterday.  And here you can filter first time.  You can see a time range and then it loads.  And then you see much more preview images.  And when you can also change the cloud coverage.  So you see now when you go for more cloud coverage, the time gets less.  When you go to zero, you are allowed to take the whole range of the whole time because  it's not so many cloud free images at the same time.  And then you can see and if you click on it, then you see the cloud percentage.  It's zero because I choose zero.  And then you can get all the information.  And this is for the one spot that I clicked on, all the images which are available from  this satellite.  And then you get some information like the name, the UUID, the size, you can see here  it's six gigabytes.  And the date.  Okay and then I wanted to show you processing but as I said before it takes a lot of time.  So I prepared ID.  So you can also filter by ID if you know what you want.  It is possible.  Search.  So I switch the preview layer off.  I have my search result here.  And then you can order a process scene.  I will choose a composite.  You can also choose some indices, for example, the normal life difference vegetation index.  And for the composites, we did some before.  Like this is true color composite.  Take some of the bands for Sentinel-2 data.  And color infrared composite is also very popular.  And we did some more.  And I will order this one.  And then it sends.  And then at one point you should get a success message.  Okay let me show you more features of the client.  You can also see some of the footprints here in the client which are already processed  because each one which is processed will be published in the client.  And for example, we process Buenos Aires.  And then you can slide through all the composites which are already processed.  And also here you can see some of the indices we calculated.  And this is the NDVI layer.  We don't have it for every footprint but for some and it's growing.  There it is.  And normally this will take hours but for you to see what would happen.  I prepared a preview.  This will be the email you will get with a link for the web service at the moment only  possible to view it in the client.  And you will get the link like this.  And then when you open it you go directly to the scene that you ordered in the composite.  Do I have some time left?  Oh okay.  Then I was very quick.  Yes.  Maybe we can start earlier with the questions or I can show some of the features of the  client.  For example, if you look at the composites and you see they are very beautiful and you  can have a look at them and on the other hand for the indices you can find out and get  value out of the data.  Yeah.  And yeah.  Okay.  Let's start with the questions.  Yeah, that's all Sentinel 2.  And no they don't.  They are only for the land.  Which areas?  It is.  Yes.  Yes.  Can you show them by area?  Yes.  You can see it here.  And when you.  Yes, but they are only for the preview images and you can choose between any and when you  select the data then you only get the result in this box.  So you have to choose one by one at the moment.  Yeah.  When you click the order button then the job will be sent to a server and we have, we are  always looking for server architecture for power and we have somebody who we can calculate  at the moment for one week until August.  So now you can order products and after one week then we talk to them and we see.  So if you know how to get hardware power then we would be very happy.  And then we have one server which does the, which receives all the jobs and it sends to  different nodes.  So we already set it up some servers where everything is, yes, where everything is installed.  No, no, it's server based.  So I see that you can search for data.  But do you use some kind of OGC standards?  For example, for catalog, you have CSWP for processing LWBS.  So you make use of these standard complaints of, use the real standards to make your processing  call or to research for the data.  The OGC standards we only use for publishing the web services view, but we don't, yeah,  not a catalog service yet.  No, no.  I see that you have built this platform based on open source on that.  But I imagine that while building a new, maybe like you want to print equal or some  code to do there together, then you have built this platform.  Let's see.  Yeah.  So the question is, are you kind of also making open source even this blue code, the  code that combines all this open source?  I haven't done it yet.  Probably.  And you written download data from the central side up because it is not the same way.  Yes.  Yes.  Yeah.  Yeah.  That's as it is for now because we can store.  Yes.  Yes, which is the API up from Isiah.  Yes, we haven't implemented it yet.  But with the email you get, we want to provide a link where you can download the image as  well, probably as a WMS.  Sorry.  As we use the server, you can use any form of the geo server supports.  We only have the predefined ones, which there was a form when you can click on the scene.  I can show you.  Like we have composites, predefined and also indices.  And at the moment we have the vegetation index and some others.  But at the moment it's not possible to put your own algorithm to it.  It's not yet open.  And Kraskis, mostly Kraskis, yes.  And a little bit of GDAL as well.  And the web interface.  Small comments.  You were talking about the atmospheric correction.  The product, the whole product, it's in the interface is the result of the send to power  processing.  Or is it?  Is it direct?  This one?  No, this one is a composite.  But I was not sure that how you apply the send to power correction to all.  If they are.  If one scene is ordered, then we apply it to all of them.  We have another client for it, but it's not in this one.  But if you like, I can show it.  This is just a demo client.  I think sorry, the data is lost somewhere.  But here it would be possible to slide through the time.  Two filters, yes.  For the preview images.  Okay, any more questions?  Okay.
foss4g-2016-1108-from_outer_space_to_your_browser-hd_0_287.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_288_779.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_780_4163.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_4164_6635.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_6636_10163.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_10164_11099.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_11100_11279.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_11280_12443.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_12444_12503.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_12504_14147.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_14148_15491.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_15492_15587.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_15588_15923.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_15924_16127.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_16128_22955.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_22956_23399.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_23400_26123.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_26124_26171.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_26172_26495.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_26496_26891.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_26892_27155.png foss4g-2016-1108-from_outer_space_to_your_browser-hd_27156_27569.png
It is better. Yes. My name is Carmen and I want to show you a map client where we get Sentinel to data. I don't have to introduce Copernicus anymore. Andrea Faisback did it in the opening session. And let's start. First I work at Mundialis and we are located at Bonn. So we didn't have to travel as far as you probably. And you can see here what we are doing. And we do the Mundialis Gambia and we also have another project. It is called Mundialis Art. And you can see it at Mundialis.de and it presents how beautiful the earth is and the imagery. So the Copernicus Broadjam provides many data. They have services, six different ones. But there is also raw data and normal users might not know how to process them. So we found a way to download it. They provide an app you can request and you can filter and search and we use this one to get the data that we like. And also we provide the preview images for all the Sentinel to data from the Copernicus Broadjam and we store them in our database. Only the bounding boxes and metadata for it. And then there is another open source tool. It is called CentroCore. This does the atmospheric correction for Sentinel to data. And this helps a lot because we can use it. And also there was a major release for another version. And before it took eight hours to process one scene, about seven gigabytes, just to the atmospheric correction. And with the newer version it is possible to do sub-scenes at the same time so it's much quicker. And then we do some algorithms with GDAL and also with Kraskis. And then we take some from the raw data and you can choose what kind of process you like to have. And then it gets processed. And at the end you will get an image out which is optimized for the web. And then we will publish it with GeoServer. We also have different processing chains for GIS because for the web we make it very small so that it can be displayed quick. And we also have different processing chains but they are not available in the web client at the moment. So what happens when the data is at GeoServer? It is put into a web client. And this web client is built with open source software as well. It contains basics, open layer 3, GeoX and XJS. And now let's look at the client. It is here. And there you can already see some of the preview images. We don't have such a smart solution as they presented before. We show all the preview images but we limited a little bit for the user because you can't filter and see all of them at the same time. And these are the paths that the satellite took probably yesterday. And here you can filter first time. You can see a time range and then it loads. And then you see much more preview images. And when you can also change the cloud coverage. So you see now when you go for more cloud coverage, the time gets less. When you go to zero, you are allowed to take the whole range of the whole time because it's not so many cloud free images at the same time. And then you can see and if you click on it, then you see the cloud percentage. It's zero because I choose zero. And then you can get all the information. And this is for the one spot that I clicked on, all the images which are available from this satellite. And then you get some information like the name, the UUID, the size, you can see here it's six gigabytes. And the date. Okay and then I wanted to show you processing but as I said before it takes a lot of time. So I prepared ID. So you can also filter by ID if you know what you want. It is possible. Search. So I switch the preview layer off. I have my search result here. And then you can order a process scene. I will choose a composite. You can also choose some indices, for example, the normal life difference vegetation index. And for the composites, we did some before. Like this is true color composite. Take some of the bands for Sentinel-2 data. And color infrared composite is also very popular. And we did some more. And I will order this one. And then it sends. And then at one point you should get a success message. Okay let me show you more features of the client. You can also see some of the footprints here in the client which are already processed because each one which is processed will be published in the client. And for example, we process Buenos Aires. And then you can slide through all the composites which are already processed. And also here you can see some of the indices we calculated. And this is the NDVI layer. We don't have it for every footprint but for some and it's growing. There it is. And normally this will take hours but for you to see what would happen. I prepared a preview. This will be the email you will get with a link for the web service at the moment only possible to view it in the client. And you will get the link like this. And then when you open it you go directly to the scene that you ordered in the composite. Do I have some time left? Oh okay. Then I was very quick. Yes. Maybe we can start earlier with the questions or I can show some of the features of the client. For example, if you look at the composites and you see they are very beautiful and you can have a look at them and on the other hand for the indices you can find out and get value out of the data. Yeah. And yeah. Okay. Let's start with the questions. Yeah, that's all Sentinel 2. And no they don't. They are only for the land. Which areas? It is. Yes. Yes. Can you show them by area? Yes. You can see it here. And when you. Yes, but they are only for the preview images and you can choose between any and when you select the data then you only get the result in this box. So you have to choose one by one at the moment. Yeah. When you click the order button then the job will be sent to a server and we have, we are always looking for server architecture for power and we have somebody who we can calculate at the moment for one week until August. So now you can order products and after one week then we talk to them and we see. So if you know how to get hardware power then we would be very happy. And then we have one server which does the, which receives all the jobs and it sends to different nodes. So we already set it up some servers where everything is, yes, where everything is installed. No, no, it's server based. So I see that you can search for data. But do you use some kind of OGC standards? For example, for catalog, you have CSWP for processing LWBS. So you make use of these standard complaints of, use the real standards to make your processing call or to research for the data. The OGC standards we only use for publishing the web services view, but we don't, yeah, not a catalog service yet. No, no. I see that you have built this platform based on open source on that. But I imagine that while building a new, maybe like you want to print equal or some code to do there together, then you have built this platform. Let's see. Yeah. So the question is, are you kind of also making open source even this blue code, the code that combines all this open source? I haven't done it yet. Probably. And you written download data from the central side up because it is not the same way. Yes. Yes. Yeah. Yeah. That's as it is for now because we can store. Yes. Yes, which is the API up from Isiah. Yes, we haven't implemented it yet. But with the email you get, we want to provide a link where you can download the image as well, probably as a WMS. Sorry. As we use the server, you can use any form of the geo server supports. We only have the predefined ones, which there was a form when you can click on the scene. I can show you. Like we have composites, predefined and also indices. And at the moment we have the vegetation index and some others. But at the moment it's not possible to put your own algorithm to it. It's not yet open. And Kraskis, mostly Kraskis, yes. And a little bit of GDAL as well. And the web interface. Small comments. You were talking about the atmospheric correction. The product, the whole product, it's in the interface is the result of the send to power processing. Or is it? Is it direct? This one? No, this one is a composite. But I was not sure that how you apply the send to power correction to all. If they are. If one scene is ordered, then we apply it to all of them. We have another client for it, but it's not in this one. But if you like, I can show it. This is just a demo client. I think sorry, the data is lost somewhere. But here it would be possible to slide through the time. Two filters, yes. For the preview images. Okay, any more questions? Okay.
beautiful the earth is and the imagery. So the filter and search and we use this one to get the tool. It is called CentroCore. This does the atmospheric correction for Sentinel to data. And some algorithms with GDAL and also with Kraskis. And then we take some from the raw data and you will get an image out which is optimized for the web. And then we will publish it with GeoServer. put into a web client. And this web client is built with open source software as well. It all the images which are available from this satellite. And then you get some information like So you can also filter by ID if you know what you And then you can order a process scene. I will some of the footprints here in the client which processed will be published in the client. And for the web service at the moment only possible and when you select the data then you only get order button then the job will be sent to a standards to make your processing call or to research for the data. The OGC standards we only The product, the whole product, it's in the somewhere. But here it would be possible to slide