WMS-V: Animate Raster Maps
Heuer, Tim-Hinnerk
WMS-V - Video Map Tiles for the 3rd or 4th Dimension
 Hi, I'm Tim. I work for Monarchifino-Alenka Research in New Zealand. Let me take you on a journey with geographical videos and longitudinal compression and a story about my experience with this exciting new technology. What is the problem to solve? Spatial data does not only present a constant in space, it includes the thing that is fundamental to our very nature, change. How? How can we show change? To summarize, I want to briefly explain the motivation for WMSV. Then I'd like to show you a few proof of concept applications, the first of which was the visualization of our national possum model. Furthermore, I attempted visualizing temperature change in the Antarctic in 2015. The challenges of that data led me to a visually nicer dataset. The Himawari satellite takes the photo every 10 minutes, which I downloaded sample of and concatenated to videos. There are more challenges that I'll get into. Meanwhile, I'll try to summarize the story of WMSV and briefly cover some concepts and considerations. And finally, propose one or even more standards. First, I want to ask why. Visualization of temporal data over time, such as weather patterns, forecasts, push fires, storms, or different properties such as soil profiles, traveling through depths in the ground, water or animal bodies could really help us see what is going on in very interesting ways across different domains. In general, with what we perceive as time, we can visualize at least one additional dimension of data, whether this is in 2D or in 3D, or on a screen or potentially in virtual reality. For now, let's start with 2D maps and data over time in an interactive way. WMS time is a standard in the web mapping service framework. It is a standard which enables open source maps to consider time as simple map tiles. Here you can see an example using WMS time and PNG images. As you may be able to see, we are making 110 requests here for one layer and one extent. I don't think I need to explain why this is a problem. There is no transition between frames and no server side filtering between images possible. Not only latency, but also decoding compressed images is costly. So why not have a stream of images coming in as a video and being decoded as a stream? Video technology has been researched and improved upon over decades now, and we have very efficient video formats both in terms of speed and file size. Why visualize RESTor data? A lot of data is sourced as RESTor, aerial, satellite, or UV imagery to name just a few. Models can be less costly to process using RESTors, since some operations are costly to calculate with big data and those vectors. Let me show you the national possible model. I won't go into too much detail of the model itself. The focus here is on the video map technology. As you can see, there is a navigable map showing the videos versus map tiles. The extent and positions of video tiles are shown accurately on screen. Regional council areas and their statistics are shown. There is a slider to advance time in a live preview. How did we do this? Initially with the possible model, we replaced all the image elements with HTML5 video elements. In OpenLayers 2 at the time, DOM rendering was used, which was the only approach possible then. DOM rendering is expensive and was later replaced by Canvas and WebGL rendering, both of which are hardware accelerated. An HTML Canvas element enables developers to do filtering and advanced image manipulations, which are not having to worry about all the graphics challenges. My namesake, Tim Schaub, put together a proof of concept which I learned was inspired by a Mac box post previously. This was after we did the national possible model. Here is the example he put together. You can see how this kind of visual could not be achieved with downloading separately compressed images as frames or with DOM rendering with this kind of performance and at an angle like that. I will let you appreciate this for a few seconds. I tried other data. They are remotely sensed and Arctic surface temperature from 2015. However, the data was not really suitable. So WMSV is not great for visualizing data that has little commonality. Also, some caching should be done. Video tiles and compositing them on the server may be the way to go. Copying, composing, and padding works. FFMPEG and complex filters make this possible at fast speeds. Challenges. Transparent videos are not fully supported at the time I checked. With the Antarctic data, I wrote a client-side filter that uses the green channel as transparency, which of course only works if you are not using a channel in your data visualization. Layers on top of video layer are problematic without DOM rendering. Synchronizing visual tiles is really a great challenge. The Bars APIs are not reliable enough for this purpose. I tried a lot of things. Having the videos precomputed and sitting on the server as simple files was best for now. Downloading partial content then gets handled by the web server. I spent a lot of time optimizing the synchronization. In the end, double buffering and frame caching seems to work best and surprisingly also seems okay in terms of memory use. Performance and stability are challenging. However, having tabs open for a while can crash the browser. At FostroG 2013, my demo failed because I had opened it in the wrong browser. I have been implementing a dynamic WS MSP that worked with any bounding box so that videos can be encoded on the fly. This solves the synchronization problem but increases the load on the server and server-side filtering such as blending can slow things down and not to forget image loading. FFMPEG supports GPU rendering but it makes the video quality worse. There may be faster video encoding options, libraries or similar that can solve these problems. Encoding one video and streaming it was not the bottleneck though. It is the image tile delivery from a WMS time. So if one could use low-level code and access the rest of the data with indexes directly, I believe the on-the-fly delivery could be the way to go. So now more to the story. In 2012, LNK was involved with the POSM model. I remember talking to a colleague about this exciting new HTML5 technology, video and canvas describing how these elements behave like images in many ways. In the conversation, the idea of using video tiles came up. Robert coined the phrase longitudinal compression. A few hours of progressive working later, I had crafted some map tiles, some video map tiles from the POSM model raster data and combined some on a small grid in my browser, playing the videos at the same time. The tech got some interest, however mostly used my free time. When I presented in WMSV in 2013, I was talking to the open-layers guys and Tim Schaub came up with the post-compose idea and building it as an extension. There was motivation to present again here today. Now to standards. I have a long and skeptical journey with standards. It began when my boss gave me the immense job of reading the WMS spec and later punished me with the even more convoluted WFS2 standard. At the time, as you can imagine, I did not appreciate standard much. However, having spent time with WFS3 at a hackathon and getting annoyed yet again with different power plugs in different countries, I realized that standards are important. So I spent a bit of WMS time, pun intended. I hacked it behind my WMSV and learned to appreciate the iso-time notation standards since they are even implemented in JavaScript. I had a look at the time range spec in WMS time and decided to implement that in my dynamic WMSV. And here is the time spec. You can find the details and how to interpret them online. More that could be standardized. Frame rate, video formats, encoding options, filter spec, all this as extensions or as core. So more on streams. If you are not asleep yet, this might have helped. More on the story. Finding good example data is more difficult than I thought. I was not happy with what I had to show, so I found the Himawari satellite imagery, which you can see in a minute. As an aside, you could imagine that you could make a zoomable, panable, NK video possible with this tech. Satellite video feeds could provide live coverage of world events such as natural disasters, weather forecasts are shown in everyday TV. All this and more could be visualized with remotely sensed or model raster data in WMSV. So here the promised Himawari footage as a web app and using video tiles. Getting them to synchronizing like this was a major challenge. And here a 1080p slice of it. As you can see, the weather is nicely visible here. Just a couple of days ago, I got a dynamic version of the Himawari videos going where I simply cropped a high resolution video on the fly. It basically projects a portion of the video for display and could stream at five times the speed of the video. This possibly works. I don't know. Looks like it might be working. So here you see that it's dynamically generated because the seconds are advancing. There's no escape button. Anyway, you can try it out. I don't have the address on screen today. It doesn't matter. It's okay. So coming to the conclusions. Video streaming is fast and image fetching is slow. Tiles are good for servers and dynamic is better for clients. Finding data without errors is still hard. Hybrid caching seems the way to go. Composing, cropping, existing videos on the server works well. What was the problem statement from the beginning? Change. It's what we want to see. Change is what WMSV can show. So consider WMSV as something on your radar. Thank you. And I open for questions. Next, Tim. Really interesting. Questions from the audience. Hi. Is this like, have you already implemented that and is that accessible at the moment through open layers or something like that? Yeah, I thought this question might come up. It's not accessible at the moment, but I'm planning to make it accessible at least on my account. And sort of what are you targeting a version of open layers? I'm assuming it's incorporating open layers. Is that right or is it going to be separate? It's going to be an extension of open layers, if anything. I already created an extension, but I wouldn't say I'm not proud of that code because it's all messy. So yeah, I have to clean it up a bit to publish it. Any other questions? Over here. Yeah, hi. Thanks. Great presentation. I'm just interested to know, and I might be showing my ignorance here a little bit, but how are you generating the video tiles on the server? Yeah. How are you generating the video? So is it tiles or it's more of a video? So I did many different ways. So first, I built a pyramid from just the JPEG images or the imagery, the image files. So first, I built a pyramid with a map server and map cache. And then I simply had different times, just different layers. And then I just concatenated those to videos and just basically hacked map server to serve up videos instead of images, with a proxy in front of it. And then with the dynamic WMS v, I did implement like an FFM pack filter that's so that there's a filter spec on the FFM pack website. You can look it up. It's a bit complicated, but it's actually okay. Yeah. So and then that was the dynamic. And then I did the sort of hybrid approach with the caching and also dynamic generation by first I tried composing the video tiles, but that had a problem. So but it was just a superficial problem, I think, that could, you know, just me misinterpreting the documentation. But I think, but the speed was actually good to compose them the different tiles onto one video. So you could actually stream one video to the client quite nicely better than trying to synchronize on the client. Okay. Cool. Any other questions? Hi, sorry. So the video that you showed before was that like in the like you could drag that that global whatever it is around. What was that video of? Or was it like a 3D model that you overlaid those videos? It was just a video of the images. And I created a 2500 by 2500 video. And then I just simply crop it on the server on the fly. And that actually works quite quickly. But then you deliver less data. And it's, it still comes as a stream. And you know, the browser then handles all the decoding and playing. So for example, in that video, could you like drag it around? For example, like the cube, like would that be usable in that way? Well, not at this stage, actually. I tried doing that in the last couple of days. I had the tiling idea first, because I thought it would be too costly to generate on the fly. But actually the on the fly works quite well with the hybrid approach. Cool. Very good.
 Hi, I'm Tim. I work for Monarchifino-Alenka Research in New Zealand. Let me take you  on a journey with geographical videos and longitudinal compression and a story about  my experience with this exciting new technology. What is the problem to solve? Spatial data  does not only present a constant in space, it includes the thing that is fundamental  to our very nature, change. How? How can we show change? To summarize, I want to briefly  explain the motivation for WMSV. Then I'd like to show you a few proof of concept applications,  the first of which was the visualization of our national possum model. Furthermore, I  attempted visualizing temperature change in the Antarctic in 2015. The challenges of that data  led me to a visually nicer dataset. The Himawari satellite takes the photo every 10 minutes,  which I downloaded sample of and concatenated to videos. There are more challenges that I'll  get into. Meanwhile, I'll try to summarize the story of WMSV and briefly cover some concepts  and considerations. And finally, propose one or even more standards. First, I want to ask why.  Visualization of temporal data over time, such as weather patterns, forecasts, push fires,  storms, or different properties such as soil profiles, traveling through depths in the ground,  water or animal bodies could really help us see what is going on in very interesting ways across  different domains. In general, with what we perceive as time, we can visualize at least one  additional dimension of data, whether this is in 2D or in 3D, or on a screen or potentially in virtual  reality. For now, let's start with 2D maps and data over time in an interactive way. WMS time is a  standard in the web mapping service framework. It is a standard which enables open source maps to  consider time as simple map tiles. Here you can see an example using WMS time and PNG images. As  you may be able to see, we are making 110 requests here for one layer and one extent. I don't think  I need to explain why this is a problem. There is no transition between frames and no server side  filtering between images possible. Not only latency, but also decoding compressed images is  costly. So why not have a stream of images coming in as a video and being decoded as a stream?  Video technology has been researched and improved upon over decades now, and we have very efficient  video formats both in terms of speed and file size. Why visualize RESTor data? A lot of data is  sourced as RESTor, aerial, satellite, or UV imagery to name just a few. Models can be less  costly to process using RESTors, since some operations are costly to calculate with big data  and those vectors. Let me show you the national possible model. I won't go into too much detail  of the model itself. The focus here is on the video map technology. As you can see, there is a  navigable map showing the videos versus map tiles. The extent and positions of video tiles are  shown accurately on screen. Regional council areas and their statistics are shown. There is a  slider to advance time in a live preview. How did we do this? Initially with the possible model,  we replaced all the image elements with HTML5 video elements. In OpenLayers 2 at the time,  DOM rendering was used, which was the only approach possible then. DOM rendering is expensive and  was later replaced by Canvas and WebGL rendering, both of which are hardware accelerated. An HTML  Canvas element enables developers to do filtering and advanced image manipulations, which are not  having to worry about all the graphics challenges. My namesake, Tim Schaub, put together a proof of  concept which I learned was inspired by a Mac box post previously. This was after we did the  national possible model. Here is the example he put together. You can see how this kind of  visual could not be achieved with downloading separately compressed images as frames or with  DOM rendering with this kind of performance and at an angle like that. I will let you appreciate  this for a few seconds. I tried other data. They are remotely sensed and Arctic surface  temperature from 2015. However, the data was not really suitable. So WMSV is not great for  visualizing data that has little commonality. Also, some caching should be done. Video  tiles and compositing them on the server may be the way to go. Copying, composing, and padding  works. FFMPEG and complex filters make this possible at fast speeds. Challenges. Transparent  videos are not fully supported at the time I checked. With the Antarctic data, I wrote a  client-side filter that uses the green channel as transparency, which of course only works if  you are not using a channel in your data visualization. Layers on top of video layer are  problematic without DOM rendering. Synchronizing visual tiles is really a great challenge. The  Bars APIs are not reliable enough for this purpose. I tried a lot of things. Having the  videos precomputed and sitting on the server as simple files was best for now. Downloading  partial content then gets handled by the web server. I spent a lot of time optimizing the  synchronization. In the end, double buffering and frame caching seems to work best and surprisingly  also seems okay in terms of memory use. Performance and stability are challenging. However, having  tabs open for a while can crash the browser. At FostroG 2013, my demo failed because I  had opened it in the wrong browser. I have been implementing a dynamic WS MSP that worked  with any bounding box so that videos can be encoded on the fly. This solves the synchronization  problem but increases the load on the server and server-side filtering such as blending  can slow things down and not to forget image loading. FFMPEG supports GPU rendering but  it makes the video quality worse. There may be faster video encoding options, libraries  or similar that can solve these problems. Encoding one video and streaming it was not  the bottleneck though. It is the image tile delivery from a WMS time. So if one could  use low-level code and access the rest of the data with indexes directly, I believe the  on-the-fly delivery could be the way to go. So now more to the story. In 2012, LNK was  involved with the POSM model. I remember talking to a colleague about this exciting  new HTML5 technology, video and canvas describing how these elements behave like images in many  ways. In the conversation, the idea of using video tiles came up. Robert coined the phrase  longitudinal compression. A few hours of progressive working later, I had crafted some map tiles,  some video map tiles from the POSM model raster data and combined some on a small grid in  my browser, playing the videos at the same time. The tech got some interest, however mostly  used my free time. When I presented in WMSV in 2013, I was talking to the open-layers  guys and Tim Schaub came up with the post-compose idea and building it as an extension. There  was motivation to present again here today. Now to standards. I have a long and skeptical  journey with standards. It began when my boss gave me the immense job of reading the WMS  spec and later punished me with the even more convoluted WFS2 standard. At the time, as  you can imagine, I did not appreciate standard much. However, having spent time with WFS3  at a hackathon and getting annoyed yet again with different power plugs in different countries,  I realized that standards are important. So I spent a bit of WMS time, pun intended. I  hacked it behind my WMSV and learned to appreciate the iso-time notation standards since they  are even implemented in JavaScript. I had a look at the time range spec in WMS time and  decided to implement that in my dynamic WMSV. And here is the time spec. You can find the  details and how to interpret them online. More that could be standardized. Frame rate,  video formats, encoding options, filter spec, all this as extensions or as core. So more  on streams. If you are not asleep yet, this might have helped. More on the story. Finding  good example data is more difficult than I thought. I was not happy with what I had  to show, so I found the Himawari satellite imagery, which you can see in a minute. As  an aside, you could imagine that you could make a zoomable, panable, NK video possible  with this tech. Satellite video feeds could provide live coverage of world events such  as natural disasters, weather forecasts are shown in everyday TV. All this and more could  be visualized with remotely sensed or model raster data in WMSV.  So here the promised Himawari footage as a web app and using video tiles. Getting them  to synchronizing like this was a major challenge. And here a 1080p slice of it. As you can see,  the weather is nicely visible here. Just a couple of days ago, I got a dynamic version  of the Himawari videos going where I simply cropped a high resolution video on the fly.  It basically projects a portion of the video for display and could stream at five times  the speed of the video. This possibly works. I don't know. Looks like it might be working.  So here you see that it's dynamically generated because the seconds are advancing. There's  no escape button. Anyway, you can try it out. I don't have the address on screen today.  It doesn't matter. It's okay. So coming to the conclusions. Video streaming is fast and  image fetching is slow. Tiles are good for servers and dynamic is better for clients.  Finding data without errors is still hard. Hybrid caching seems the way to go. Composing,  cropping, existing videos on the server works well. What was the problem statement from  the beginning? Change. It's what we want to see. Change is what WMSV can show. So consider  WMSV as something on your radar. Thank you. And I open for questions.  Next, Tim. Really interesting. Questions from the audience.  Hi. Is this like, have you already implemented that and is that accessible at the moment  through open layers or something like that? Yeah, I thought this question might come up.  It's not accessible at the moment, but I'm planning to make it accessible at least on  my account. And sort of what are you targeting a version of open layers? I'm assuming it's  incorporating open layers. Is that right or is it going to be separate?  It's going to be an extension of open layers, if anything. I already created an extension,  but I wouldn't say I'm not proud of that code because it's all messy. So yeah, I have to  clean it up a bit to publish it. Any other questions?  Over here. Yeah, hi. Thanks. Great presentation. I'm  just interested to know, and I might be showing my ignorance here a little bit, but how are  you generating the video tiles on the server? Yeah.  How are you generating the video? So is it tiles or it's more of a video?  So I did many different ways. So first, I built a pyramid from just the JPEG images  or the imagery, the image files. So first, I built a pyramid with a map server and map  cache. And then I simply had different times, just different layers. And then I just concatenated  those to videos and just basically hacked map server to serve up videos instead of images,  with a proxy in front of it. And then with the dynamic WMS v, I did implement like an  FFM pack filter that's so that there's a filter spec on the FFM pack website. You can look  it up. It's a bit complicated, but it's actually okay. Yeah. So and then that was the dynamic.  And then I did the sort of hybrid approach with the caching and also dynamic generation  by first I tried composing the video tiles, but that had a problem. So but it was just  a superficial problem, I think, that could, you know, just me misinterpreting the documentation.  But I think, but the speed was actually good to compose them the different tiles onto one  video. So you could actually stream one video to the client quite nicely better than trying  to synchronize on the client.  Okay. Cool. Any other questions?  Hi, sorry. So the video that you showed before was that like in the like you could drag that  that global whatever it is around. What was that video of? Or was it like a 3D model that  you overlaid those videos?  It was just a video of the images. And I created a 2500 by 2500 video. And then I just simply  crop it on the server on the fly. And that actually works quite quickly. But then you  deliver less data. And it's, it still comes as a stream. And you know, the browser then  handles all the decoding and playing.  So for example, in that video, could you like drag it around? For example, like the cube,  like would that be usable in that way?  Well, not at this stage, actually. I tried doing that in the last couple of days. I had  the tiling idea first, because I thought it would be too costly to generate on the fly.  But actually the on the fly works quite well with the hybrid approach.  Cool. Very good.
Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_0_59.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_60_1847.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_1848_3491.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_3492_5507.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_5508_6419.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_6420_7367.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_7368_7439.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_7440_7655.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_7656_7799.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_7800_12299.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_12300_15191.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_15192_16943.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_16944_17555.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_17556_18443.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_18444_19115.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_19116_20135.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_20136_20327.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_20328_20483.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_20484_28535.png Tim-Hinnerk_Heuer__WMS-V_-_Video_Map_Tiles_for_the_3rd_or_4th_Dimension_fDz8H_a_Y9s_28536_28585.png
Hi, I'm Tim. I work for Monarchifino-Alenka Research in New Zealand. Let me take you on a journey with geographical videos and longitudinal compression and a story about my experience with this exciting new technology. What is the problem to solve? Spatial data does not only present a constant in space, it includes the thing that is fundamental to our very nature, change. How? How can we show change? To summarize, I want to briefly explain the motivation for WMSV. Then I'd like to show you a few proof of concept applications, the first of which was the visualization of our national possum model. Furthermore, I attempted visualizing temperature change in the Antarctic in 2015. The challenges of that data led me to a visually nicer dataset. The Himawari satellite takes the photo every 10 minutes, which I downloaded sample of and concatenated to videos. There are more challenges that I'll get into. Meanwhile, I'll try to summarize the story of WMSV and briefly cover some concepts and considerations. And finally, propose one or even more standards. First, I want to ask why. Visualization of temporal data over time, such as weather patterns, forecasts, push fires, storms, or different properties such as soil profiles, traveling through depths in the ground, water or animal bodies could really help us see what is going on in very interesting ways across different domains. In general, with what we perceive as time, we can visualize at least one additional dimension of data, whether this is in 2D or in 3D, or on a screen or potentially in virtual reality. For now, let's start with 2D maps and data over time in an interactive way. WMS time is a standard in the web mapping service framework. It is a standard which enables open source maps to consider time as simple map tiles. Here you can see an example using WMS time and PNG images. As you may be able to see, we are making 110 requests here for one layer and one extent. I don't think I need to explain why this is a problem. There is no transition between frames and no server side filtering between images possible. Not only latency, but also decoding compressed images is costly. So why not have a stream of images coming in as a video and being decoded as a stream? Video technology has been researched and improved upon over decades now, and we have very efficient video formats both in terms of speed and file size. Why visualize RESTor data? A lot of data is sourced as RESTor, aerial, satellite, or UV imagery to name just a few. Models can be less costly to process using RESTors, since some operations are costly to calculate with big data and those vectors. Let me show you the national possible model. I won't go into too much detail of the model itself. The focus here is on the video map technology. As you can see, there is a navigable map showing the videos versus map tiles. The extent and positions of video tiles are shown accurately on screen. Regional council areas and their statistics are shown. There is a slider to advance time in a live preview. How did we do this? Initially with the possible model, we replaced all the image elements with HTML5 video elements. In OpenLayers 2 at the time, DOM rendering was used, which was the only approach possible then. DOM rendering is expensive and was later replaced by Canvas and WebGL rendering, both of which are hardware accelerated. An HTML Canvas element enables developers to do filtering and advanced image manipulations, which are not having to worry about all the graphics challenges. My namesake, Tim Schaub, put together a proof of concept which I learned was inspired by a Mac box post previously. This was after we did the national possible model. Here is the example he put together. You can see how this kind of visual could not be achieved with downloading separately compressed images as frames or with DOM rendering with this kind of performance and at an angle like that. I will let you appreciate this for a few seconds. I tried other data. They are remotely sensed and Arctic surface temperature from 2015. However, the data was not really suitable. So WMSV is not great for visualizing data that has little commonality. Also, some caching should be done. Video tiles and compositing them on the server may be the way to go. Copying, composing, and padding works. FFMPEG and complex filters make this possible at fast speeds. Challenges. Transparent videos are not fully supported at the time I checked. With the Antarctic data, I wrote a client-side filter that uses the green channel as transparency, which of course only works if you are not using a channel in your data visualization. Layers on top of video layer are problematic without DOM rendering. Synchronizing visual tiles is really a great challenge. The Bars APIs are not reliable enough for this purpose. I tried a lot of things. Having the videos precomputed and sitting on the server as simple files was best for now. Downloading partial content then gets handled by the web server. I spent a lot of time optimizing the synchronization. In the end, double buffering and frame caching seems to work best and surprisingly also seems okay in terms of memory use. Performance and stability are challenging. However, having tabs open for a while can crash the browser. At FostroG 2013, my demo failed because I had opened it in the wrong browser. I have been implementing a dynamic WS MSP that worked with any bounding box so that videos can be encoded on the fly. This solves the synchronization problem but increases the load on the server and server-side filtering such as blending can slow things down and not to forget image loading. FFMPEG supports GPU rendering but it makes the video quality worse. There may be faster video encoding options, libraries or similar that can solve these problems. Encoding one video and streaming it was not the bottleneck though. It is the image tile delivery from a WMS time. So if one could use low-level code and access the rest of the data with indexes directly, I believe the on-the-fly delivery could be the way to go. So now more to the story. In 2012, LNK was involved with the POSM model. I remember talking to a colleague about this exciting new HTML5 technology, video and canvas describing how these elements behave like images in many ways. In the conversation, the idea of using video tiles came up. Robert coined the phrase longitudinal compression. A few hours of progressive working later, I had crafted some map tiles, some video map tiles from the POSM model raster data and combined some on a small grid in my browser, playing the videos at the same time. The tech got some interest, however mostly used my free time. When I presented in WMSV in 2013, I was talking to the open-layers guys and Tim Schaub came up with the post-compose idea and building it as an extension. There was motivation to present again here today. Now to standards. I have a long and skeptical journey with standards. It began when my boss gave me the immense job of reading the WMS spec and later punished me with the even more convoluted WFS2 standard. At the time, as you can imagine, I did not appreciate standard much. However, having spent time with WFS3 at a hackathon and getting annoyed yet again with different power plugs in different countries, I realized that standards are important. So I spent a bit of WMS time, pun intended. I hacked it behind my WMSV and learned to appreciate the iso-time notation standards since they are even implemented in JavaScript. I had a look at the time range spec in WMS time and decided to implement that in my dynamic WMSV. And here is the time spec. You can find the details and how to interpret them online. More that could be standardized. Frame rate, video formats, encoding options, filter spec, all this as extensions or as core. So more on streams. If you are not asleep yet, this might have helped. More on the story. Finding good example data is more difficult than I thought. I was not happy with what I had to show, so I found the Himawari satellite imagery, which you can see in a minute. As an aside, you could imagine that you could make a zoomable, panable, NK video possible with this tech. Satellite video feeds could provide live coverage of world events such as natural disasters, weather forecasts are shown in everyday TV. All this and more could be visualized with remotely sensed or model raster data in WMSV. So here the promised Himawari footage as a web app and using video tiles. Getting them to synchronizing like this was a major challenge. And here a 1080p slice of it. As you can see, the weather is nicely visible here. Just a couple of days ago, I got a dynamic version of the Himawari videos going where I simply cropped a high resolution video on the fly. It basically projects a portion of the video for display and could stream at five times the speed of the video. This possibly works. I don't know. Looks like it might be working. So here you see that it's dynamically generated because the seconds are advancing. There's no escape button. Anyway, you can try it out. I don't have the address on screen today. It doesn't matter. It's okay. So coming to the conclusions. Video streaming is fast and image fetching is slow. Tiles are good for servers and dynamic is better for clients. Finding data without errors is still hard. Hybrid caching seems the way to go. Composing, cropping, existing videos on the server works well. What was the problem statement from the beginning? Change. It's what we want to see. Change is what WMSV can show. So consider WMSV as something on your radar. Thank you. And I open for questions. Next, Tim. Really interesting. Questions from the audience. Hi. Is this like, have you already implemented that and is that accessible at the moment through open layers or something like that? Yeah, I thought this question might come up. It's not accessible at the moment, but I'm planning to make it accessible at least on my account. And sort of what are you targeting a version of open layers? I'm assuming it's incorporating open layers. Is that right or is it going to be separate? It's going to be an extension of open layers, if anything. I already created an extension, but I wouldn't say I'm not proud of that code because it's all messy. So yeah, I have to clean it up a bit to publish it. Any other questions? Over here. Yeah, hi. Thanks. Great presentation. I'm just interested to know, and I might be showing my ignorance here a little bit, but how are you generating the video tiles on the server? Yeah. How are you generating the video? So is it tiles or it's more of a video? So I did many different ways. So first, I built a pyramid from just the JPEG images or the imagery, the image files. So first, I built a pyramid with a map server and map cache. And then I simply had different times, just different layers. And then I just concatenated those to videos and just basically hacked map server to serve up videos instead of images, with a proxy in front of it. And then with the dynamic WMS v, I did implement like an FFM pack filter that's so that there's a filter spec on the FFM pack website. You can look it up. It's a bit complicated, but it's actually okay. Yeah. So and then that was the dynamic. And then I did the sort of hybrid approach with the caching and also dynamic generation by first I tried composing the video tiles, but that had a problem. So but it was just a superficial problem, I think, that could, you know, just me misinterpreting the documentation. But I think, but the speed was actually good to compose them the different tiles onto one video. So you could actually stream one video to the client quite nicely better than trying to synchronize on the client. Okay. Cool. Any other questions? Hi, sorry. So the video that you showed before was that like in the like you could drag that that global whatever it is around. What was that video of? Or was it like a 3D model that you overlaid those videos? It was just a video of the images. And I created a 2500 by 2500 video. And then I just simply crop it on the server on the fly. And that actually works quite quickly. But then you deliver less data. And it's, it still comes as a stream. And you know, the browser then handles all the decoding and playing. So for example, in that video, could you like drag it around? For example, like the cube, like would that be usable in that way? Well, not at this stage, actually. I tried doing that in the last couple of days. I had the tiling idea first, because I thought it would be too costly to generate on the fly. But actually the on the fly works quite well with the hybrid approach. Cool. Very good.
briefly explain the motivation for WMSV. Then I'd faster video encoding options, libraries or