Marine Data enhanced Virtual Laboratory DEVL #1 - June 18
Proctor, Roger
Roger Proctor from Australian Ocean Data Network, Uni of Tasmania (1.6.18) In June & July's TechTalk events, representatives from each of the DEVLs will introduce their projects from a developers' perspective: the problem the project is trying to solve, the tech stacks deployed and to be developed, the approaches of their software development and community engagement while developing tools and applications. The demonstration/discussion of 8 projects are scheduled as follows: DEVL #1 June (1st): Astronomy (Robert Shen), Marine Science (Roger Proctor), Terrestrial Ecology (Gerhard Weis), and Climate (Clare Richards).
 Right, okay, so our virtual laboratory, this data enhanced virtual laboratory is about building information infrastructure in support of the National Ocean Modelling System. I'm giving the presentation, but two of the guys who are listed below, Angus on the right there and Hugo on the left, and other people involved in my group here in Imos are Guillen, Gallibara, and Sebastian Mancini and the other members of the project are folk in the Bureau of Meteorology, CSIRO, NCI, players involved a bit, South Australia Research Development Institute, Tasmanian Partnership for Advanced Computing, University of New South Wales, University of Western Australia, and in this case, Metocean, New Zealand. So what are we actually trying to do here? And so our project aim here is to further develop the marine virtual laboratory to support the long-term goal of the ocean research community, namely developing the Australian National Shelsea Reanalysis, or ANSA as we call it. So what is a real analysis? This is a technical term for a product which combines the optimum combination of model simulation and observations, maximising the value of the observational set through data assimilation. Now this project that we're doing isn't going to do the reanalysis. Here the emphasis is on building on our capabilities to assemble the necessary observational data and to construct services to enable model ingestion of these data into the data assimilation schemes to be used in the analysis. The actual project to do the reanalysis is probably about a three to four year project to actually complete it. So given that we've only got money till December, it's not the realistic goal we even think about doing the reanalysis at this stage. So why do we want to do this? So the growth of the blue economy is predicted to be large by 2025 and it's predicted to grow three times faster than the Australian GDP. So there is a burgeoning interest in the marine environment and arguments for this national modelling service system have been around for quite some time and in fact it's a recommendation of the National Marine Science Plan 2015 to 2025. Earlier this year the Bureau of Meteorology commenced a plan for operational coastal modelling service and to enable that to really get going then certain groundwork has to be done to provide for such an operational system and this is really what this project is about. It's doing the groundwork to provide the observations and the service to enable a reanalysis program to be done. So reanalysis is quite simple in concept. It combines combining observations in a model state through the simulation process to produce an updated model state which essentially will bring the model generally speaking closer to the observations and thereby providing a far more realistic simulation than sometimes do come out of model simulations. So there are various ways you can do data assimilation and there are various methods that you can use. There are ensemble methods such as optimal interpolation and using a Kalman filter. There are variational methods either in three dimensions or in four dimensions, I think three dimensions in time but each method actually requires a way of reducing the observations that are available in the community down to a number that are manageable and this manageable number of observations are called superobs and you want to do this in such a way that you don't lose the information content of the larger data collection and then when you actually want to do the analysis there are various ways in which you can do it and on the right here you can see four different approaches to doing assimilation combining the observations into an analysis, running a model and you can see you get step changes, this is the top version, you can do it sequentially, continuous assimilation and again you've got a continuous solution but there are step changes in this and then there is non-sequential intermittent assimilation when you put the information in when you actually have it available or the bottom one which is what we really would like to aim for which is the non-sequential continuous assimilation to end up with a smooth solution from the results of the assimilation process. So in setting up coastal models and this is what the marine virtual laboratory actually does, the marine virtual laboratory as it stands at the moment is a tool for accelerating the development of coastal modelling studies so in the marine virtual laboratory you can go through a step through a menu which enables you to select different community open source models, you can choose from a menu of those, excuse me I've got a bit of a cold still and you can choose that, you can define your grid and region of interest from a map, you can then select and you can play around with that to get resolution right and things like this that you can then select from another menu to choose from data sets for atmospheric forcing from initial conditions for ocean boundary conditions and you can tune some of the parameters and at the end of the process you can either submit your configuration into the cloud to do a simulation with a particular model or you can actually bundle the data sets that you've collected constructed from the particular model of interest and take them away and use them on a computer somewhere else but in all those processes there are certain things that you need to do and you need some you need information about the geography about coastlines, you need information about the sea depths, the bathymetry, you need an initial condition, you need boundary condition, boundary forcing for the ocean and you need surface forcing for meteorology to drive the ocean circulation in the process and you also need some other information particularly in coastal areas. In Marvel at the moment once you've chosen your region of interest in your time period then a web processing service delivers observations that you can use for validation of the simulation and what we're moving on to now is thinking about how you use the observations available in data assimilation processes. Sorry? Three minutes. Okay plenty of time so what you actually want in a coastal ocean reanalysis is a variable grid and this example here which I've chosen is the Syro Compass model for Storm Bay which is just down here in Hobart shows the kind of resolution that realistically you want to do if you're going to do a sensible reanalysis of coastal processes so here you've got grid sizes that range between five kilometers down to a hundred meters. So our project here intends to acquire and assemble the necessary data to do this reanalysis between 1992 and 2016, a compiling observational atmospheric forcing, ocean forcing, part of the project is providing a coastal discharge set which doesn't exist for Australia at the moment to enable us to do this because fresh water inflow into coastal regions is an extremely important forcing function and we need to assemble the necessary bathymetry and information about tides because tides in coastal areas have to be taken account of when you're doing data assimilation. So the second part of this is to develop the services to prepare these model ready observations, the super-obs, for use in the intended schemes and the schemes that we're going to use assemble stuff for a ensemble Kalman filter, ensemble optimal interpolation and 4d bar and there's remote sensing data, satellite information, the long track data, there's in situ data, combination of profile data, mooring data and glider data and the intention is to assemble all of this information on NCI and to run a service to process those observations to produce the super-obs outputs for the necessary inputs to assimilation guns. So we're in the process of looking at designing a GUI user interface which will take a limited number of parameters, it will take the grid that you want your observations gridded to, take the time period of interest and the data cycle time that you want to adopt, you'll then choose a particular assimilation scheme, you'll choose observation parameters from a list that we have and then the observations will then be processed and that's our job, that's the Australian Ocean Data Network, using either web feature services or web processing services, we're still in the process of deciding how the best way approach to go, then all this then runs on NCI using code that's been developed by Australian data assimilation scientists, one code for ensembles, one code for 4d bar and then your output from the end is these super-obs formatted for the chosen scheme that you will then use to run your particular model. We're aiming for this service to be open and this is something we're negotiating with NCI at the moment as to how this can actually work because a lot of these data sets are under different projects in NCI, we want to bring them together and make them available to the community. Some of the key questions we've still to resolve, because of the involvement of New Zealand in this space, then we're contemplating we have to extend this region of interest over to New Zealand, we still as I said mentioned we're looking at the best way of delivering the observations in terms of which sort of service would be appropriate and in general looking at how efficient can we make this production service for the users. And then the latter parts of the project say two things really, one is about demonstrating the utility of these services through test cases which our modelling teams are in the process of defining at the moment. So it may be they will run different simulation schemes to the one they currently use in their work to look at the comparison between the results but also look at how efficient it is to construct these super-obs compared to the way they do it manually at the moment. And then when we've piled all those data sets for the 25-year simulation then we'll look at how we add these data sets into Marvel as it stands at the moment which will make the utility of this service much more attractive to our users. Okay thank you Roger.
 Right, okay, so our virtual laboratory, this data enhanced virtual laboratory is about  building information infrastructure in support of the National Ocean Modelling System.  I'm giving the presentation, but two of the guys who are listed below, Angus on the right  there and Hugo on the left, and other people involved in my group here in Imos are Guillen,  Gallibara, and Sebastian Mancini and the other members of the project are folk in the Bureau  of Meteorology, CSIRO, NCI, players involved a bit, South Australia Research Development Institute,  Tasmanian Partnership for Advanced Computing, University of New South Wales, University of  Western Australia, and in this case, Metocean, New Zealand. So what are we actually trying to do here?  And so our project aim here is to further develop the marine virtual laboratory to support the  long-term goal of the ocean research community, namely developing the Australian National  Shelsea Reanalysis, or ANSA as we call it. So what is a real analysis? This is a technical term  for a product which combines the optimum combination of model simulation and observations,  maximising the value of the observational set through data assimilation. Now this project  that we're doing isn't going to do the reanalysis. Here the emphasis is on building on our capabilities  to assemble the necessary observational data and to construct services to enable model ingestion  of these data into the data assimilation schemes to be used in the analysis. The actual project  to do the reanalysis is probably about a three to four year project to actually complete it. So  given that we've only got money till December, it's not the realistic goal we even think about  doing the reanalysis at this stage. So why do we want to do this? So the growth of the blue economy  is predicted to be large by 2025 and it's predicted to grow three times faster than the  Australian GDP. So there is a burgeoning interest in the marine environment and  arguments for this national modelling service system have been around for quite some time and  in fact it's a recommendation of the National Marine Science Plan 2015 to 2025. Earlier this year  the Bureau of Meteorology commenced a plan for operational coastal modelling service  and to enable that to really get going then certain groundwork has to be done to provide  for such an operational system and this is really what this project is about. It's doing the groundwork  to provide the observations and the service to enable a reanalysis program to be done.  So reanalysis is quite simple in concept. It combines combining observations in a model state  through the simulation process to produce an updated model state which essentially will bring  the model generally speaking closer to the observations and thereby providing a far more  realistic simulation than sometimes do come out of model simulations. So there are various ways  you can do data assimilation and there are various methods that you can use. There are  ensemble methods such as optimal interpolation and using a Kalman filter. There are variational  methods either in three dimensions or in four dimensions, I think three dimensions in time  but each method actually requires a way of reducing the observations that are available in the community  down to a number that are manageable and this manageable number of observations are called  superobs and you want to do this in such a way that you don't lose the information content of  the larger data collection and then when you actually want to do the analysis there are  various ways in which you can do it and on the right here you can see four different approaches  to doing assimilation combining the observations into an analysis, running a model and you can  see you get step changes, this is the top version, you can do it sequentially, continuous assimilation  and again you've got a continuous solution but there are step changes in this and then there  is non-sequential intermittent assimilation when you put the information in when you actually have  it available or the bottom one which is what we really would like to aim for which is the  non-sequential continuous assimilation to end up with a smooth solution from the results of the  assimilation process. So in setting up coastal models and this is what the marine virtual  laboratory actually does, the marine virtual laboratory as it stands at the moment is a tool for  accelerating the development of coastal modelling studies so in the marine virtual laboratory  you can go through a step through a menu which enables you to select different community open  source models, you can choose from a menu of those, excuse me I've got a bit of a cold still  and you can choose that, you can define your grid and region of interest from a map, you can then  select and you can play around with that to get resolution right and things like this that you  can then select from another menu to choose from data sets for atmospheric forcing from  initial conditions for ocean boundary conditions and you can tune some of the parameters and at  the end of the process you can either submit your configuration into the cloud to do a simulation  with a particular model or you can actually bundle the data sets that you've collected  constructed from the particular model of interest and take them away and use them on a computer  somewhere else but in all those processes there are certain things that you need to do and you  need some you need information about the geography about coastlines, you need information about the  sea depths, the bathymetry, you need an initial condition, you need boundary condition, boundary  forcing for the ocean and you need surface forcing for meteorology to drive the ocean circulation  in the process and you also need some other information particularly in coastal areas.  In Marvel at the moment once you've chosen your region of interest in your time period then a  web processing service delivers observations that you can use for validation of the simulation  and what we're moving on to now is thinking about how you use the observations available  in data assimilation processes.  Sorry?  Three minutes.  Okay plenty of time so what you actually want in a coastal ocean reanalysis is a variable grid  and this example here which I've chosen is the Syro Compass model for Storm Bay which is just down  here in Hobart shows the kind of resolution that realistically you want to do if you're going to  do a sensible reanalysis of coastal processes so here you've got grid sizes that range between  five kilometers down to a hundred meters. So our project here intends to acquire and assemble  the necessary data to do this reanalysis between 1992 and 2016, a compiling observational  atmospheric forcing, ocean forcing, part of the project is providing a coastal discharge set  which doesn't exist for Australia at the moment to enable us to do this because fresh water  inflow into coastal regions is an extremely important forcing function and we need to assemble the  necessary bathymetry and information about tides because tides in coastal areas have to be taken  account of when you're doing data assimilation. So the second part of this is to develop the  services to prepare these model ready observations, the super-obs, for use in the intended schemes and  the schemes that we're going to use assemble stuff for a ensemble Kalman filter, ensemble  optimal interpolation and 4d bar and there's remote sensing data, satellite information,  the long track data, there's in situ data, combination of profile data, mooring data and  glider data and the intention is to assemble all of this information on NCI and to run a service  to process those observations to produce the super-obs outputs for the necessary  inputs to assimilation guns. So we're in the process of looking at designing a GUI user interface  which will take a limited number of parameters, it will take the grid that you want your observations  gridded to, take the time period of interest and the data cycle time that you want to adopt,  you'll then choose a particular assimilation scheme, you'll choose observation parameters from  a list that we have and then the observations will then be processed and that's our job,  that's the Australian Ocean Data Network, using either web feature services or web processing  services, we're still in the process of deciding how the best way approach to go, then all this  then runs on NCI using code that's been developed by Australian data assimilation scientists,  one code for ensembles, one code for 4d bar and then your output from the end is these super-obs  formatted for the chosen scheme that you will then use to run your particular model. We're aiming  for this service to be open and this is something we're negotiating with NCI at the moment as to  how this can actually work because a lot of these data sets are under different projects in NCI,  we want to bring them together and make them available to the community. Some of the key  questions we've still to resolve, because of the involvement of New Zealand in this space,  then we're contemplating we have to extend this region of interest over to New Zealand,  we still as I said mentioned we're looking at the best way of delivering the observations in  terms of which sort of service would be appropriate and in general looking at how efficient can we  make this production service for the users. And then the latter parts of the project say  two things really, one is about demonstrating the utility of these services through test cases  which our modelling teams are in the process of defining at the moment. So it may be they will run  different simulation schemes to the one they currently use in their work to look at the  comparison between the results but also look at how efficient it is to construct these super-obs  compared to the way they do it manually at the moment. And then when we've piled all those data  sets for the 25-year simulation then we'll look at how we add these data sets into Marvel as it  stands at the moment which will make the utility of this service much more attractive to our users.  Okay thank you Roger.
Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_0_1451.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_1452_3155.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_3156_4655.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_4656_5351.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_5352_7559.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_7560_10703.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_10704_11495.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_11496_11831.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_11832_12011.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_12012_12083.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_12084_12191.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_12192_12371.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_12372_12491.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_12492_13367.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_13368_15587.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_15588_16271.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_16272_17447.png Marine_Virtual_Laboratory_DEVL_-_Tech_Talk_-_June_18_uBDLEwW7wec_17448_17536.png
Right, okay, so our virtual laboratory, this data enhanced virtual laboratory is about building information infrastructure in support of the National Ocean Modelling System. I'm giving the presentation, but two of the guys who are listed below, Angus on the right there and Hugo on the left, and other people involved in my group here in Imos are Guillen, Gallibara, and Sebastian Mancini and the other members of the project are folk in the Bureau of Meteorology, CSIRO, NCI, players involved a bit, South Australia Research Development Institute, Tasmanian Partnership for Advanced Computing, University of New South Wales, University of Western Australia, and in this case, Metocean, New Zealand. So what are we actually trying to do here? And so our project aim here is to further develop the marine virtual laboratory to support the long-term goal of the ocean research community, namely developing the Australian National Shelsea Reanalysis, or ANSA as we call it. So what is a real analysis? This is a technical term for a product which combines the optimum combination of model simulation and observations, maximising the value of the observational set through data assimilation. Now this project that we're doing isn't going to do the reanalysis. Here the emphasis is on building on our capabilities to assemble the necessary observational data and to construct services to enable model ingestion of these data into the data assimilation schemes to be used in the analysis. The actual project to do the reanalysis is probably about a three to four year project to actually complete it. So given that we've only got money till December, it's not the realistic goal we even think about doing the reanalysis at this stage. So why do we want to do this? So the growth of the blue economy is predicted to be large by 2025 and it's predicted to grow three times faster than the Australian GDP. So there is a burgeoning interest in the marine environment and arguments for this national modelling service system have been around for quite some time and in fact it's a recommendation of the National Marine Science Plan 2015 to 2025. Earlier this year the Bureau of Meteorology commenced a plan for operational coastal modelling service and to enable that to really get going then certain groundwork has to be done to provide for such an operational system and this is really what this project is about. It's doing the groundwork to provide the observations and the service to enable a reanalysis program to be done. So reanalysis is quite simple in concept. It combines combining observations in a model state through the simulation process to produce an updated model state which essentially will bring the model generally speaking closer to the observations and thereby providing a far more realistic simulation than sometimes do come out of model simulations. So there are various ways you can do data assimilation and there are various methods that you can use. There are ensemble methods such as optimal interpolation and using a Kalman filter. There are variational methods either in three dimensions or in four dimensions, I think three dimensions in time but each method actually requires a way of reducing the observations that are available in the community down to a number that are manageable and this manageable number of observations are called superobs and you want to do this in such a way that you don't lose the information content of the larger data collection and then when you actually want to do the analysis there are various ways in which you can do it and on the right here you can see four different approaches to doing assimilation combining the observations into an analysis, running a model and you can see you get step changes, this is the top version, you can do it sequentially, continuous assimilation and again you've got a continuous solution but there are step changes in this and then there is non-sequential intermittent assimilation when you put the information in when you actually have it available or the bottom one which is what we really would like to aim for which is the non-sequential continuous assimilation to end up with a smooth solution from the results of the assimilation process. So in setting up coastal models and this is what the marine virtual laboratory actually does, the marine virtual laboratory as it stands at the moment is a tool for accelerating the development of coastal modelling studies so in the marine virtual laboratory you can go through a step through a menu which enables you to select different community open source models, you can choose from a menu of those, excuse me I've got a bit of a cold still and you can choose that, you can define your grid and region of interest from a map, you can then select and you can play around with that to get resolution right and things like this that you can then select from another menu to choose from data sets for atmospheric forcing from initial conditions for ocean boundary conditions and you can tune some of the parameters and at the end of the process you can either submit your configuration into the cloud to do a simulation with a particular model or you can actually bundle the data sets that you've collected constructed from the particular model of interest and take them away and use them on a computer somewhere else but in all those processes there are certain things that you need to do and you need some you need information about the geography about coastlines, you need information about the sea depths, the bathymetry, you need an initial condition, you need boundary condition, boundary forcing for the ocean and you need surface forcing for meteorology to drive the ocean circulation in the process and you also need some other information particularly in coastal areas. In Marvel at the moment once you've chosen your region of interest in your time period then a web processing service delivers observations that you can use for validation of the simulation and what we're moving on to now is thinking about how you use the observations available in data assimilation processes. Sorry? Three minutes. Okay plenty of time so what you actually want in a coastal ocean reanalysis is a variable grid and this example here which I've chosen is the Syro Compass model for Storm Bay which is just down here in Hobart shows the kind of resolution that realistically you want to do if you're going to do a sensible reanalysis of coastal processes so here you've got grid sizes that range between five kilometers down to a hundred meters. So our project here intends to acquire and assemble the necessary data to do this reanalysis between 1992 and 2016, a compiling observational atmospheric forcing, ocean forcing, part of the project is providing a coastal discharge set which doesn't exist for Australia at the moment to enable us to do this because fresh water inflow into coastal regions is an extremely important forcing function and we need to assemble the necessary bathymetry and information about tides because tides in coastal areas have to be taken account of when you're doing data assimilation. So the second part of this is to develop the services to prepare these model ready observations, the super-obs, for use in the intended schemes and the schemes that we're going to use assemble stuff for a ensemble Kalman filter, ensemble optimal interpolation and 4d bar and there's remote sensing data, satellite information, the long track data, there's in situ data, combination of profile data, mooring data and glider data and the intention is to assemble all of this information on NCI and to run a service to process those observations to produce the super-obs outputs for the necessary inputs to assimilation guns. So we're in the process of looking at designing a GUI user interface which will take a limited number of parameters, it will take the grid that you want your observations gridded to, take the time period of interest and the data cycle time that you want to adopt, you'll then choose a particular assimilation scheme, you'll choose observation parameters from a list that we have and then the observations will then be processed and that's our job, that's the Australian Ocean Data Network, using either web feature services or web processing services, we're still in the process of deciding how the best way approach to go, then all this then runs on NCI using code that's been developed by Australian data assimilation scientists, one code for ensembles, one code for 4d bar and then your output from the end is these super-obs formatted for the chosen scheme that you will then use to run your particular model. We're aiming for this service to be open and this is something we're negotiating with NCI at the moment as to how this can actually work because a lot of these data sets are under different projects in NCI, we want to bring them together and make them available to the community. Some of the key questions we've still to resolve, because of the involvement of New Zealand in this space, then we're contemplating we have to extend this region of interest over to New Zealand, we still as I said mentioned we're looking at the best way of delivering the observations in terms of which sort of service would be appropriate and in general looking at how efficient can we make this production service for the users. And then the latter parts of the project say two things really, one is about demonstrating the utility of these services through test cases which our modelling teams are in the process of defining at the moment. So it may be they will run different simulation schemes to the one they currently use in their work to look at the comparison between the results but also look at how efficient it is to construct these super-obs compared to the way they do it manually at the moment. And then when we've piled all those data sets for the 25-year simulation then we'll look at how we add these data sets into Marvel as it stands at the moment which will make the utility of this service much more attractive to our users. Okay thank you Roger.
actually trying to do here? And so our project ocean research community, namely developing the observational data and to construct services to data assimilation schemes to be used in the Marine Science Plan 2015 to 2025. Earlier this ocean forcing, part of the project is providing a the Australian Ocean Data Network, using either sets are under different projects in NCI, we want Roger.