Road Less Travelled: IAGs Journey To An Open Source Geospatial Platform
Haverkamp, Ed
The Road Less Travelled: A corporation’s journey to an open source enterprise geospatial platform
 Thank you everybody. First of all I think I'd like to thank Paul for playing me in very nicely into the hero's journey this morning in his plenary. I assure you no money changed hands but I'm very thankful for it. So the words of Joseph Campbell himself, a hero ventures forth from the world of common day into a region of supernatural wonder. Fabulous forces are there encountered and a decisive victory is won. The hero comes back from this mysterious adventure with the powers to bestow boons on his fellow man. So I'd actually like to also thank my colleague Elise McKenna who's not here for actually suggesting the hero's journey as a way of telling this story and then I started researching it and I found out how wonderful it actually was. So the hero's journey operates as a circular story structure really meaning the hero's journey will start and end where it began. Though their journey will leave them forever changed. So the aim of my talk today is to explain the how, the why and the ways of our leaps forward in our capability, our challenges and how we overcame them to hopefully motivate other organisations to adopt ever increasing amounts of amazing open source geospatial data, data engineering and data science talent and tools to get better at what they do, to put location first and foremost into critical decision making. So the introduction to the protagonists world in the hero's journey. So all about IAG. IAG is Australia's largest general insurer. We have a revenues approaching $11 billion and we insure over $2 trillion in assets protecting millions of homes and cars and hundreds of thousands of businesses. Our brands date back almost 160 years but you didn't really come here to listen about our annual report so I'll just skip over the rest of it and go straight into it. IAG is not new to geospatial by any means. During the Woollongong floods that you can see here back in 1998 approximately 400 millimetres of rain fell in a three hour period. As a result refused claims and coverage put the insurance industry into turmoil. Geocoding and mapping started to become a central focus as part of any solution moving forward for insurance companies. So the key to it all, I'm a bit biased, geocoding. At the heart of change towards more granular knowledge of risk we needed world class knowledge of geocoding and something that remains very core to us today in increasing efforts and understanding across the business to get a grips to garbage in, garbage out, data remediation and data quality. We're only ever as good as the data that we have in our systems. Accurately located risks meant accurate modelling of risk. At the property level models are continually updated and refined by the information that we receive. There's a level of maths in here that I don't even begin to understand, I leave that to the analysts but needless to say that location pays a huge part in this and it's always been the job of the geospatial teams to maintain what we term geo factors. Geospatially relevant factors around location that can make a difference. Not just perils for bushfire, flood, storm but also non-peril factors. The ability to communicate to stakeholders on many projects became a core function, helped us to grow the geospatial teams across the business and the spatial capabilities start seeing in areas such as BI and portfolio management where there was custom maps for stakeholders showing such things as historical claims impact. Situational awareness around major events response and providing the inherent ability to visualise and represent risks in a context that everyone would understand and in an easy way to communicate that. Helping business decision making with spatial analysis and inclusion into such things as board presentations seems pretty rudimentary to us as GIS people and pretty simple and quite frankly a little bit of an ugly map but it holds massive influence to those who live in a world that's dominated by spreadsheets and charts. We started early mapping applications for portfolio management and visualisation that blew their minds. Early on postcode was about the only common field throughout many of our disparate systems that we could rely on and it was a method where we could link them and at least a first attempt into getting a notion of what spatial views of data look like and something's better than nothing, right? In that area alone on the evangelism of correct ways of visualisation and spatial information we had plenty of work left ahead of us to undo the wrongs of the postcode past if you will and get the business to learn about the quality of the data that they had and available to us and better ways of displaying with it and working with it. All the previous slides won't be anything new to anyone in this room. We've all been there in whatever organisation we've worked on and you may be representing at a time and we certainly dabbled in a variety of different technology over that time. We started with Map Info, moved on to Manifold.net because for the first time we were able to actually integrate with a real live database. We then set to developing our full Esri stack first based on SQL Server and then in 2014 cutting our back end databases over to Postgres with PostGIS. Our call to action in the hero's journey represents an event, ultimately disrupts the comfort of the hero's ordinary world and represents a challenge or a quest that must be undertaken. In 2014 IAG established a whole new division called Information Analytics with the intent that IAG's competitive advantage was founded on information. We needed to maximise the opportunities that we had and be prepared for the age of the customer. It was at that point of IAG's journey that I was fortunate enough to join the team and this was our call to action. To start enacting on this vision and opportunity, IAG embarked on creating a data lake, bringing together many systems, 230 source systems in all through many years of just growing a company out by acquisition. These services and operating businesses land the data into a single store and of an enterprise including raw copies of source system data and transformed copies of that data to be used for reporting, visualisation, analytics and machine learning. As part of this buildout Geospatial was seen as an inherent need of the new system so a proof of concept was established to see how well spatial data could work inside our Green Plum environment. By virtue of the fact that under the hood, Green Plum was inherently Postgres and albeit a fair few versions behind had Postgres. At the same time, as the technology investment, was a change to the organisation around the data engineering and data science fields. We needed to cultivate a community of data science and data engineering talent. We needed to communicate data's true value proposition, create new data driven work practices and also be able to scale those work practices. In the hero's journey, this is the point where the hero crosses into the field of adventure, leaving the known limits of his world and venturing into the unknown and dangerous realm where the rules and limits are unknown. In the new information analytics world, we were establishing a culture of innovation, working on newly acquired and geo-enabling sources from the data lake to sit along those that hadn't been yet ingested. I still strongly remember my first days at IAG and wondering why Hugh would have a picture of Justin Timberlake posted up in his cubicle, but hey, I don't judge. It was a reminder to us that under Project Timberlake, we were unashamedly trying to bring sexy back and how we communicated the value of data back to the business and that led us down to the road of innovation and experiments with cesium, Node.js, many multiples maps, Flask, Geoserver, Hexgrids and then 4D Hexgrids. With our first set of geo-enabled cross-baring claims data from the data lake, we were able to analyse lodgement and closure of claims over time using our newfound love of hexagons. And during an internal challenge on reducing claim settlement time, this is what we produced. Our ability to communicate this data in this manner whilst it may seem a little showy and not overly operational was a great way for us to deliver the message about the power of the information stores, our technology, our skills and the road we were undertaking to take a unified view of data and its possibilities. In the hero's journey, the mentor is a guide, a protector, a magical helper that offers knowledge, confidence, advice to aid the junior. Within this person, this was Hugh Selmans. No stranger to anybody in this room, I'm sure, better recognised, maybe by the orange pushpin or his minus 34 Twitter handle. A strong proponent of the open data and open source movements within Geospatial Australia. A confirmed geo-hipster, geo-rabble founder and well respected within IAG. Hugh was there at the beginning of the push for us to take those Geospatial functions and get them to work within our new data lake environment. The first challenge that we had to overcome was we had to present, you know, the hero must overcome a challenge presented on the journey towards their ultimate goal. The first challenge for us was emerging of all the disparate Geospatial teams into a single entity, bringing together the spatial minds within the business during what was quite significant organisational change and a way of working that wasn't going to be easy. The team of talented people and the new location engineering team was here to make a difference. And within a short time, we had our first list of strategic goals established. We started sharing out on a new postgres space infrastructure. We started ingesting data from the data lake to processes and geo-enable rather than old static SAS data sets. Adoption, changing the ways of working and growing talent within a team in areas such as product management, agile practices and a firm belief and a willingness by all of continuous learning. Never stop learning. A change in operating not only a GIS department, to stop thinking about ourselves as a GIS department or being GIS people, to a team of spatial data engineers and data scientists. So our initial state was to establish a postgres-driven back-end database powering both the existing ESRI systems and new parallel development on Python and Geo-Server capabilities. In the beginning of our transition, all existing applications and new ones that we were starting to build on leaflet-based front-ends. And on top of this, we started developing out our capability to work more closely with data from the data lake rather than legacy data warehouse options and other ad hoc files. Temptation in the hero's journey. In this step, a hero faces temptations that may lead him or her to abandon or stray from their quest. The seeker of the life beyond must press forward, surpass the temptations and the temptations call and soar to the immaculate either beyond. Of course, the temptation was there throughout the journey to stick with the tried and true. But through things like data security, data governance and cost, along with a firm belief from within the entire organisation that we should be embracing what open source meant to IAG and what we could give back to the community and to challenge ourselves and our skill sets by continuous learning and development. For the most part, there's always been an alternative solution for us in the open source domain and we're very thankful to the community that's out there and many of you are in this room. The dark moment in the hero's journey described as uncomfortable, but in many ways is exactly what you wanted or needed to move forward. And as such, we identified a few key issues that we definitely needed to tackle. Application using bespoke JavaScript front-ends, leaflet, open layers, was difficult for a lot of the team to upskill on easily. And maybe to the astonishment of many in this room, not every space-ally-minded person is longing to dig into JavaScript. Luckily, we recruited Stephen Leed from full extent to help us in this regard and we need heavily upon his boot leaf framework that he worked out. So bootstrapping leaflet into a way that we could just configure maps by updating a config file. He's giving a talk later on and I strongly suggest you go to that one. Demand grew, more systems were acquired into the data lake, suddenly the data lake infrastructure that had been planned out and yes, we will scale, all of a sudden couldn't handle everything that was coming in and we were told to get out. So that led us to continue to operate and siloed infrastructure for much longer than what we would have hoped, but what it did was it put a stronger emphasis on our ETL processes. How will we still to get data in and out of the data lake rather than operating in the data lake? One of the other things we found is that an MPP environment ain't that great for spatial anyway. The GIST index doesn't really chart all that well, so you're probably better off doing it in your own postgres and then just shipping the data back. So existing custom data sets often created many years ago and legacy SAS objects had to be recreated. So there's a lot of information there that needed to be ingested. So we needed to be able to handle the mashup of legacy data as well as those from the data lake source. In the final conflict, the hero is pushed to the brink of death and loses and faces the greatest challenge. So the decision was made to go off Esri support. This was our final conflict and so began our life of decommissioning. Creation of a new open stack environment allowing us to spin up cloud on premise, gearing up for that cloud movement later on. Moving buildings as well forced our timeline, but alongside this the data lake infrastructure had a multiple fold increase in performance. So they kind of led us back into the sandpit. Increased data flows to and from the data lake meant automation, monitoring and QA. Existing BAU work that took up large chunks of time needed to be reduced or automated so that we had more time to concentrate on the future. Creation of weather alerting and self service apps to replace existing event processing services needed to be written. Heavy reliance on for on-call support in GSbational during a major event when you're fielding at second slip on your Saturday cricket team is not an ideal phone call to get. We also needed to stop thinking like GIS people and start thinking like developers. Code creation and deployment through GitHub. We actually won a best failure award at our division awards. Four hours, four lines of code, but finally we got everybody up to speed and working with Git. The hero brings with them the object of their quest and they use better than ordinary world in some way whether it's through the knowledge or a cure or some form of protection, the return home. So this is our current state stack which I won't bore you with but we have moved all our instances within the same physical space as the data lake into our larger enterprise data hub to be able to leverage it directly, move data in and out fast and Python and GS server now serve the majority of our services, leaflet, the majority of our web maps, but we're still open and we have Tableau running right throughout the organisation as our reporting framework but we support analysts with our shiny and so the key for us is to make sure that whatever we put out, make it open, everyone can use it. So we're looking back at all the elements we've brought together over the last few years, the list is long, but Sarah here is some of the most important ones. The power of Python has been a huge help, Python frameworks that can be built and be reusable, Python server, Flask and Gunicorn, don't know where we would be without it, pandas for doing a wide variety of tasks including on the fly analysis and integrating Python underneath the Postgres install, one thing that we figured out later on had to change that so to allow Postgres to use the underlying Python functions that were available to it. No, I post this, no, I post this, pause here, ST, there's some key things, ST subdivide, cluster your gist index, tune your Postgres, truncate an insert, set your fill factor as 50 when you're doing faster updates and materialise views and foreign data wrappers for the win and lots and lots of functions that we've written. And keep calm and get. This is not so more about the actual software itself, but more about how we do things, our mindset as developers rather than GIS people now, a lot of automation and preparing for a world for us on continuous integration and continuous delivery. What good is a bestowment of gifts if you can't share out? So our public GitHub repo where we've been able to reshare out some of the goodness that we've been able to build internally and obviously there's a lot of IP so not everything is shared but we try to share what we can and hopefully that this continues. As for the future, what it holds for us, AI and machine learning, Kafka, allowing us to stream real-time data, I'm really excited about leaving behind near real-time as in twice a day to real-time. That's a huge challenge for us and how Kafka can work with us in terms of spatial objects. Apache Airflow, Spark, Snowflake, Kubernetes, we don't stand still. We can't stand still. Our data infrastructure is changing. We will have to move with it. Lastly, in summing up, we've all been through an amazing journey. We're certainly not at the end. So my key comment to all of you, I guess, is to experiment, to innovate, fail fast, don't ever settle, use Python, use PostGIS, set up a hack time to get the team to play with the new technologies. We're a huge proponent of this. What wasn't right for you before, make sure you revisit it because it's changing as well. What wasn't right for you two years ago, especially with the amount of code changes that are going on, maybe it's worth another look. The tech radar is a little bit more fluid than just dismissing it and leaving it out there. Thank you very much. E.T.L. A lot of our E.T.L. processes, we actually just wrote using things like foreign data wrappers and Python. So a lot of the monitoring systems and everything, it's pretty easy for us to either use Python or foreign data wrappers to move data around. We kind of flick between the two depending on what it is. If we're doing bulk migrations, we do just entire dumps out and then just move those over. If we're doing table refreshes, the refresh materialised view is the best way that we can just sync the data between two datasets. It's fast, it's efficient, it can be done concurrently, it doesn't table lock, so it's really good for what we do. So yeah, there's no tool as such, we just pretty much wrote our own.
 Thank you everybody. First of all I think I'd like to thank Paul for playing me in very  nicely into the hero's journey this morning in his plenary. I assure you no money changed  hands but I'm very thankful for it. So the words of Joseph Campbell himself, a hero ventures  forth from the world of common day into a region of supernatural wonder. Fabulous forces  are there encountered and a decisive victory is won. The hero comes back from this mysterious  adventure with the powers to bestow boons on his fellow man. So I'd actually like to also  thank my colleague Elise McKenna who's not here for actually suggesting the hero's journey  as a way of telling this story and then I started researching it and I found out how  wonderful it actually was. So the hero's journey operates as a circular story structure really  meaning the hero's journey will start and end where it began. Though their journey will  leave them forever changed. So the aim of my talk today is to explain the how, the why  and the ways of our leaps forward in our capability, our challenges and how we overcame them to  hopefully motivate other organisations to adopt ever increasing amounts of amazing open source  geospatial data, data engineering and data science talent and tools to get better at  what they do, to put location first and foremost into critical decision making.  So the introduction to the protagonists world in the hero's journey. So all about IAG. IAG  is Australia's largest general insurer. We have a revenues approaching $11 billion and  we insure over $2 trillion in assets protecting millions of homes and cars and hundreds of  thousands of businesses. Our brands date back almost 160 years but you didn't really come  here to listen about our annual report so I'll just skip over the rest of it and go  straight into it. IAG is not new to geospatial by any means. During the Woollongong floods  that you can see here back in 1998 approximately 400 millimetres of rain fell in a three hour  period. As a result refused claims and coverage put the insurance industry into turmoil. Geocoding  and mapping started to become a central focus as part of any solution moving forward for  insurance companies. So the key to it all, I'm a bit biased, geocoding. At the heart  of change towards more granular knowledge of risk we needed world class knowledge of  geocoding and something that remains very core to us today in increasing efforts and  understanding across the business to get a grips to garbage in, garbage out, data remediation  and data quality. We're only ever as good as the data that we have in our systems. Accurately  located risks meant accurate modelling of risk. At the property level models are continually  updated and refined by the information that we receive. There's a level of maths in here  that I don't even begin to understand, I leave that to the analysts but needless to  say that location pays a huge part in this and it's always been the job of the geospatial  teams to maintain what we term geo factors. Geospatially relevant factors around location  that can make a difference. Not just perils for bushfire, flood, storm but also non-peril  factors. The ability to communicate to stakeholders on many projects became a core function, helped  us to grow the geospatial teams across the business and the spatial capabilities start  seeing in areas such as BI and portfolio management where there was custom maps for  stakeholders showing such things as historical claims impact. Situational awareness around  major events response and providing the inherent ability to visualise and represent risks in  a context that everyone would understand and in an easy way to communicate that. Helping  business decision making with spatial analysis and inclusion into such things as board presentations  seems pretty rudimentary to us as GIS people and pretty simple and quite frankly a little  bit of an ugly map but it holds massive influence to those who live in a world that's dominated  by spreadsheets and charts. We started early mapping applications for portfolio management  and visualisation that blew their minds. Early on postcode was about the only common field  throughout many of our disparate systems that we could rely on and it was a method where  we could link them and at least a first attempt into getting a notion of what spatial views  of data look like and something's better than nothing, right? In that area alone on the  evangelism of correct ways of visualisation and spatial information we had plenty of work  left ahead of us to undo the wrongs of the postcode past if you will and get the business  to learn about the quality of the data that they had and available to us and better ways  of displaying with it and working with it. All the previous slides won't be anything  new to anyone in this room. We've all been there in whatever organisation we've worked  on and you may be representing at a time and we certainly dabbled in a variety of different  technology over that time. We started with Map Info, moved on to Manifold.net because  for the first time we were able to actually integrate with a real live database. We then  set to developing our full Esri stack first based on SQL Server and then in 2014 cutting  our back end databases over to Postgres with PostGIS. Our call to action in the hero's  journey represents an event, ultimately disrupts the comfort of the hero's ordinary world and  represents a challenge or a quest that must be undertaken. In 2014 IAG established a whole  new division called Information Analytics with the intent that IAG's competitive advantage  was founded on information. We needed to maximise the opportunities that we had and be prepared  for the age of the customer. It was at that point of IAG's journey that I was fortunate  enough to join the team and this was our call to action. To start enacting on this vision  and opportunity, IAG embarked on creating a data lake, bringing together many systems,  230 source systems in all through many years of just growing a company out by acquisition.  These services and operating businesses land the data into a single store and of an enterprise  including raw copies of source system data and transformed copies of that data to be  used for reporting, visualisation, analytics and machine learning. As part of this buildout  Geospatial was seen as an inherent need of the new system so a proof of concept was established  to see how well spatial data could work inside our Green Plum environment. By virtue of the  fact that under the hood, Green Plum was inherently Postgres and albeit a fair few versions behind  had Postgres. At the same time, as the technology investment, was a change to the organisation  around the data engineering and data science fields. We needed to cultivate a community  of data science and data engineering talent. We needed to communicate data's true value  proposition, create new data driven work practices and also be able to scale those work practices.  In the hero's journey, this is the point where the hero crosses into the field of adventure,  leaving the known limits of his world and venturing into the unknown and dangerous realm  where the rules and limits are unknown. In the new information analytics world, we were  establishing a culture of innovation, working on newly acquired and geo-enabling sources  from the data lake to sit along those that hadn't been yet ingested. I still strongly  remember my first days at IAG and wondering why Hugh would have a picture of Justin Timberlake  posted up in his cubicle, but hey, I don't judge. It was a reminder to us that under  Project Timberlake, we were unashamedly trying to bring sexy back and how we communicated  the value of data back to the business and that led us down to the road of innovation  and experiments with cesium, Node.js, many multiples maps, Flask, Geoserver, Hexgrids  and then 4D Hexgrids. With our first set of geo-enabled cross-baring claims data from  the data lake, we were able to analyse lodgement and closure of claims over time using our  newfound love of hexagons. And during an internal challenge on reducing claim settlement time,  this is what we produced. Our ability to communicate this data in this manner whilst  it may seem a little showy and not overly operational was a great way for us to deliver  the message about the power of the information stores, our technology, our skills and the  road we were undertaking to take a unified view of data and its possibilities. In the  hero's journey, the mentor is a guide, a protector, a magical helper that offers knowledge,  confidence, advice to aid the junior. Within this person, this was Hugh Selmans. No stranger  to anybody in this room, I'm sure, better recognised, maybe by the orange pushpin or  his minus 34 Twitter handle. A strong proponent of the open data and open source movements  within Geospatial Australia. A confirmed geo-hipster, geo-rabble founder and well respected within  IAG. Hugh was there at the beginning of the push for us to take those Geospatial functions  and get them to work within our new data lake environment.  The first challenge that we had to overcome was we had to present, you know, the hero  must overcome a challenge presented on the journey towards their ultimate goal. The first  challenge for us was emerging of all the disparate Geospatial teams into a single entity, bringing  together the spatial minds within the business during what was quite significant organisational  change and a way of working that wasn't going to be easy. The team of talented people and  the new location engineering team was here to make a difference. And within a short time,  we had our first list of strategic goals established. We started sharing out on a new postgres  space infrastructure. We started ingesting data from the data lake to processes and geo-enable  rather than old static SAS data sets. Adoption, changing the ways of working and growing talent  within a team in areas such as product management, agile practices and a firm belief and a willingness  by all of continuous learning. Never stop learning. A change in operating not only a  GIS department, to stop thinking about ourselves as a GIS department or being GIS people, to  a team of spatial data engineers and data scientists. So our initial state was to establish  a postgres-driven back-end database powering both the existing ESRI systems and new parallel  development on Python and Geo-Server capabilities. In the beginning of our transition, all existing  applications and new ones that we were starting to build on leaflet-based front-ends. And  on top of this, we started developing out our capability to work more closely with data  from the data lake rather than legacy data warehouse options and other ad hoc files.  Temptation in the hero's journey. In this step, a hero faces temptations that may lead  him or her to abandon or stray from their quest. The seeker of the life beyond must  press forward, surpass the temptations and the temptations call and soar to the immaculate  either beyond. Of course, the temptation was there throughout the journey to stick with  the tried and true. But through things like data security, data governance and cost, along  with a firm belief from within the entire organisation that we should be embracing what  open source meant to IAG and what we could give back to the community and to challenge  ourselves and our skill sets by continuous learning and development. For the most part,  there's always been an alternative solution for us in the open source domain and we're  very thankful to the community that's out there and many of you are in this room. The  dark moment in the hero's journey described as uncomfortable, but in many ways is exactly  what you wanted or needed to move forward. And as such, we identified a few key issues  that we definitely needed to tackle. Application using bespoke JavaScript front-ends, leaflet,  open layers, was difficult for a lot of the team to upskill on easily. And maybe to the  astonishment of many in this room, not every space-ally-minded person is longing to dig  into JavaScript. Luckily, we recruited Stephen Leed from full extent to help us in this regard  and we need heavily upon his boot leaf framework that he worked out. So bootstrapping leaflet  into a way that we could just configure maps by updating a config file. He's giving a  talk later on and I strongly suggest you go to that one. Demand grew, more systems were  acquired into the data lake, suddenly the data lake infrastructure that had been planned  out and yes, we will scale, all of a sudden couldn't handle everything that was coming  in and we were told to get out. So that led us to continue to operate and siloed infrastructure  for much longer than what we would have hoped, but what it did was it put a stronger emphasis  on our ETL processes. How will we still to get data in and out of the data lake rather  than operating in the data lake? One of the other things we found is that an MPP environment  ain't that great for spatial anyway. The GIST index doesn't really chart all that well,  so you're probably better off doing it in your own postgres and then just shipping the  data back. So existing custom data sets often created many years ago and legacy SAS objects  had to be recreated. So there's a lot of information there that needed to be ingested. So we needed  to be able to handle the mashup of legacy data as well as those from the data lake source.  In the final conflict, the hero is pushed to the brink of death and loses and faces  the greatest challenge. So the decision was made to go off Esri support. This was our  final conflict and so began our life of decommissioning. Creation of a new open stack environment allowing  us to spin up cloud on premise, gearing up for that cloud movement later on. Moving buildings  as well forced our timeline, but alongside this the data lake infrastructure had a multiple  fold increase in performance. So they kind of led us back into the sandpit. Increased  data flows to and from the data lake meant automation, monitoring and QA. Existing BAU  work that took up large chunks of time needed to be reduced or automated so that we had  more time to concentrate on the future. Creation of weather alerting and self service apps to  replace existing event processing services needed to be written. Heavy reliance on for  on-call support in GSbational during a major event when you're fielding at second slip  on your Saturday cricket team is not an ideal phone call to get. We also needed to stop  thinking like GIS people and start thinking like developers. Code creation and deployment  through GitHub. We actually won a best failure award at our division awards. Four hours,  four lines of code, but finally we got everybody up to speed and working with Git. The hero  brings with them the object of their quest and they use better than ordinary world in  some way whether it's through the knowledge or a cure or some form of protection, the  return home. So this is our current state stack which I won't bore you with but we have  moved all our instances within the same physical space as the data lake into our larger enterprise  data hub to be able to leverage it directly, move data in and out fast and Python and GS  server now serve the majority of our services, leaflet, the majority of our web maps, but  we're still open and we have Tableau running right throughout the organisation as our reporting  framework but we support analysts with our shiny and so the key for us is to make sure  that whatever we put out, make it open, everyone can use it. So we're looking back at all  the elements we've brought together over the last few years, the list is long, but  Sarah here is some of the most important ones. The power of Python has been a huge help,  Python frameworks that can be built and be reusable, Python server, Flask and Gunicorn,  don't know where we would be without it, pandas for doing a wide variety of tasks including  on the fly analysis and integrating Python underneath the Postgres install, one thing  that we figured out later on had to change that so to allow Postgres to use the underlying  Python functions that were available to it. No, I post this, no, I post this, pause here,  ST, there's some key things, ST subdivide, cluster your gist index, tune your Postgres,  truncate an insert, set your fill factor as 50 when you're doing faster updates and materialise  views and foreign data wrappers for the win and lots and lots of functions that we've  written. And keep calm and get. This is not so more about the actual software itself,  but more about how we do things, our mindset as developers rather than GIS people now,  a lot of automation and preparing for a world for us on continuous integration and continuous  delivery. What good is a bestowment of gifts if you can't share out? So our public GitHub  repo where we've been able to reshare out some of the goodness that we've been able  to build internally and obviously there's a lot of IP so not everything is shared but  we try to share what we can and hopefully that this continues. As for the future, what  it holds for us, AI and machine learning, Kafka, allowing us to stream real-time data,  I'm really excited about leaving behind near real-time as in twice a day to real-time.  That's a huge challenge for us and how Kafka can work with us in terms of spatial objects.  Apache Airflow, Spark, Snowflake, Kubernetes, we don't stand still. We can't stand still.  Our data infrastructure is changing. We will have to move with it. Lastly, in summing up,  we've all been through an amazing journey. We're certainly not at the end. So my key  comment to all of you, I guess, is to experiment, to innovate, fail fast, don't ever settle,  use Python, use PostGIS, set up a hack time to get the team to play with the new technologies.  We're a huge proponent of this. What wasn't right for you before, make sure you revisit  it because it's changing as well. What wasn't right for you two years ago, especially with  the amount of code changes that are going on, maybe it's worth another look. The tech  radar is a little bit more fluid than just dismissing it and leaving it out there. Thank  you very much.  E.T.L. A lot of our E.T.L. processes, we actually just wrote using things like foreign data  wrappers and Python. So a lot of the monitoring systems and everything, it's pretty easy for  us to either use Python or foreign data wrappers to move data around. We kind of flick between  the two depending on what it is. If we're doing bulk migrations, we do just entire dumps  out and then just move those over. If we're doing table refreshes, the refresh materialised  view is the best way that we can just sync the data between two datasets. It's fast,  it's efficient, it can be done concurrently, it doesn't table lock, so it's really good  for what we do. So yeah, there's no tool as such, we just pretty much wrote our own.
Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_0_47.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_48_215.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_216_827.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_828_4211.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_4212_7643.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_7644_8327.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_8328_12107.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_12108_12923.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_12924_14927.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_14928_17495.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_17496_21095.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_21096_26459.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_26460_26939.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_26940_27515.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_27516_27599.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_27600_28007.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_28008_28295.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_28296_29483.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_29484_29651.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_29652_29843.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_29844_29903.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_29904_30707.png Ed_Haverkamp__The_Road_Less_Travelled__A_corporation_s_journey____AQKJL08M1M4_30708_30781.png
Thank you everybody. First of all I think I'd like to thank Paul for playing me in very nicely into the hero's journey this morning in his plenary. I assure you no money changed hands but I'm very thankful for it. So the words of Joseph Campbell himself, a hero ventures forth from the world of common day into a region of supernatural wonder. Fabulous forces are there encountered and a decisive victory is won. The hero comes back from this mysterious adventure with the powers to bestow boons on his fellow man. So I'd actually like to also thank my colleague Elise McKenna who's not here for actually suggesting the hero's journey as a way of telling this story and then I started researching it and I found out how wonderful it actually was. So the hero's journey operates as a circular story structure really meaning the hero's journey will start and end where it began. Though their journey will leave them forever changed. So the aim of my talk today is to explain the how, the why and the ways of our leaps forward in our capability, our challenges and how we overcame them to hopefully motivate other organisations to adopt ever increasing amounts of amazing open source geospatial data, data engineering and data science talent and tools to get better at what they do, to put location first and foremost into critical decision making. So the introduction to the protagonists world in the hero's journey. So all about IAG. IAG is Australia's largest general insurer. We have a revenues approaching $11 billion and we insure over $2 trillion in assets protecting millions of homes and cars and hundreds of thousands of businesses. Our brands date back almost 160 years but you didn't really come here to listen about our annual report so I'll just skip over the rest of it and go straight into it. IAG is not new to geospatial by any means. During the Woollongong floods that you can see here back in 1998 approximately 400 millimetres of rain fell in a three hour period. As a result refused claims and coverage put the insurance industry into turmoil. Geocoding and mapping started to become a central focus as part of any solution moving forward for insurance companies. So the key to it all, I'm a bit biased, geocoding. At the heart of change towards more granular knowledge of risk we needed world class knowledge of geocoding and something that remains very core to us today in increasing efforts and understanding across the business to get a grips to garbage in, garbage out, data remediation and data quality. We're only ever as good as the data that we have in our systems. Accurately located risks meant accurate modelling of risk. At the property level models are continually updated and refined by the information that we receive. There's a level of maths in here that I don't even begin to understand, I leave that to the analysts but needless to say that location pays a huge part in this and it's always been the job of the geospatial teams to maintain what we term geo factors. Geospatially relevant factors around location that can make a difference. Not just perils for bushfire, flood, storm but also non-peril factors. The ability to communicate to stakeholders on many projects became a core function, helped us to grow the geospatial teams across the business and the spatial capabilities start seeing in areas such as BI and portfolio management where there was custom maps for stakeholders showing such things as historical claims impact. Situational awareness around major events response and providing the inherent ability to visualise and represent risks in a context that everyone would understand and in an easy way to communicate that. Helping business decision making with spatial analysis and inclusion into such things as board presentations seems pretty rudimentary to us as GIS people and pretty simple and quite frankly a little bit of an ugly map but it holds massive influence to those who live in a world that's dominated by spreadsheets and charts. We started early mapping applications for portfolio management and visualisation that blew their minds. Early on postcode was about the only common field throughout many of our disparate systems that we could rely on and it was a method where we could link them and at least a first attempt into getting a notion of what spatial views of data look like and something's better than nothing, right? In that area alone on the evangelism of correct ways of visualisation and spatial information we had plenty of work left ahead of us to undo the wrongs of the postcode past if you will and get the business to learn about the quality of the data that they had and available to us and better ways of displaying with it and working with it. All the previous slides won't be anything new to anyone in this room. We've all been there in whatever organisation we've worked on and you may be representing at a time and we certainly dabbled in a variety of different technology over that time. We started with Map Info, moved on to Manifold.net because for the first time we were able to actually integrate with a real live database. We then set to developing our full Esri stack first based on SQL Server and then in 2014 cutting our back end databases over to Postgres with PostGIS. Our call to action in the hero's journey represents an event, ultimately disrupts the comfort of the hero's ordinary world and represents a challenge or a quest that must be undertaken. In 2014 IAG established a whole new division called Information Analytics with the intent that IAG's competitive advantage was founded on information. We needed to maximise the opportunities that we had and be prepared for the age of the customer. It was at that point of IAG's journey that I was fortunate enough to join the team and this was our call to action. To start enacting on this vision and opportunity, IAG embarked on creating a data lake, bringing together many systems, 230 source systems in all through many years of just growing a company out by acquisition. These services and operating businesses land the data into a single store and of an enterprise including raw copies of source system data and transformed copies of that data to be used for reporting, visualisation, analytics and machine learning. As part of this buildout Geospatial was seen as an inherent need of the new system so a proof of concept was established to see how well spatial data could work inside our Green Plum environment. By virtue of the fact that under the hood, Green Plum was inherently Postgres and albeit a fair few versions behind had Postgres. At the same time, as the technology investment, was a change to the organisation around the data engineering and data science fields. We needed to cultivate a community of data science and data engineering talent. We needed to communicate data's true value proposition, create new data driven work practices and also be able to scale those work practices. In the hero's journey, this is the point where the hero crosses into the field of adventure, leaving the known limits of his world and venturing into the unknown and dangerous realm where the rules and limits are unknown. In the new information analytics world, we were establishing a culture of innovation, working on newly acquired and geo-enabling sources from the data lake to sit along those that hadn't been yet ingested. I still strongly remember my first days at IAG and wondering why Hugh would have a picture of Justin Timberlake posted up in his cubicle, but hey, I don't judge. It was a reminder to us that under Project Timberlake, we were unashamedly trying to bring sexy back and how we communicated the value of data back to the business and that led us down to the road of innovation and experiments with cesium, Node.js, many multiples maps, Flask, Geoserver, Hexgrids and then 4D Hexgrids. With our first set of geo-enabled cross-baring claims data from the data lake, we were able to analyse lodgement and closure of claims over time using our newfound love of hexagons. And during an internal challenge on reducing claim settlement time, this is what we produced. Our ability to communicate this data in this manner whilst it may seem a little showy and not overly operational was a great way for us to deliver the message about the power of the information stores, our technology, our skills and the road we were undertaking to take a unified view of data and its possibilities. In the hero's journey, the mentor is a guide, a protector, a magical helper that offers knowledge, confidence, advice to aid the junior. Within this person, this was Hugh Selmans. No stranger to anybody in this room, I'm sure, better recognised, maybe by the orange pushpin or his minus 34 Twitter handle. A strong proponent of the open data and open source movements within Geospatial Australia. A confirmed geo-hipster, geo-rabble founder and well respected within IAG. Hugh was there at the beginning of the push for us to take those Geospatial functions and get them to work within our new data lake environment. The first challenge that we had to overcome was we had to present, you know, the hero must overcome a challenge presented on the journey towards their ultimate goal. The first challenge for us was emerging of all the disparate Geospatial teams into a single entity, bringing together the spatial minds within the business during what was quite significant organisational change and a way of working that wasn't going to be easy. The team of talented people and the new location engineering team was here to make a difference. And within a short time, we had our first list of strategic goals established. We started sharing out on a new postgres space infrastructure. We started ingesting data from the data lake to processes and geo-enable rather than old static SAS data sets. Adoption, changing the ways of working and growing talent within a team in areas such as product management, agile practices and a firm belief and a willingness by all of continuous learning. Never stop learning. A change in operating not only a GIS department, to stop thinking about ourselves as a GIS department or being GIS people, to a team of spatial data engineers and data scientists. So our initial state was to establish a postgres-driven back-end database powering both the existing ESRI systems and new parallel development on Python and Geo-Server capabilities. In the beginning of our transition, all existing applications and new ones that we were starting to build on leaflet-based front-ends. And on top of this, we started developing out our capability to work more closely with data from the data lake rather than legacy data warehouse options and other ad hoc files. Temptation in the hero's journey. In this step, a hero faces temptations that may lead him or her to abandon or stray from their quest. The seeker of the life beyond must press forward, surpass the temptations and the temptations call and soar to the immaculate either beyond. Of course, the temptation was there throughout the journey to stick with the tried and true. But through things like data security, data governance and cost, along with a firm belief from within the entire organisation that we should be embracing what open source meant to IAG and what we could give back to the community and to challenge ourselves and our skill sets by continuous learning and development. For the most part, there's always been an alternative solution for us in the open source domain and we're very thankful to the community that's out there and many of you are in this room. The dark moment in the hero's journey described as uncomfortable, but in many ways is exactly what you wanted or needed to move forward. And as such, we identified a few key issues that we definitely needed to tackle. Application using bespoke JavaScript front-ends, leaflet, open layers, was difficult for a lot of the team to upskill on easily. And maybe to the astonishment of many in this room, not every space-ally-minded person is longing to dig into JavaScript. Luckily, we recruited Stephen Leed from full extent to help us in this regard and we need heavily upon his boot leaf framework that he worked out. So bootstrapping leaflet into a way that we could just configure maps by updating a config file. He's giving a talk later on and I strongly suggest you go to that one. Demand grew, more systems were acquired into the data lake, suddenly the data lake infrastructure that had been planned out and yes, we will scale, all of a sudden couldn't handle everything that was coming in and we were told to get out. So that led us to continue to operate and siloed infrastructure for much longer than what we would have hoped, but what it did was it put a stronger emphasis on our ETL processes. How will we still to get data in and out of the data lake rather than operating in the data lake? One of the other things we found is that an MPP environment ain't that great for spatial anyway. The GIST index doesn't really chart all that well, so you're probably better off doing it in your own postgres and then just shipping the data back. So existing custom data sets often created many years ago and legacy SAS objects had to be recreated. So there's a lot of information there that needed to be ingested. So we needed to be able to handle the mashup of legacy data as well as those from the data lake source. In the final conflict, the hero is pushed to the brink of death and loses and faces the greatest challenge. So the decision was made to go off Esri support. This was our final conflict and so began our life of decommissioning. Creation of a new open stack environment allowing us to spin up cloud on premise, gearing up for that cloud movement later on. Moving buildings as well forced our timeline, but alongside this the data lake infrastructure had a multiple fold increase in performance. So they kind of led us back into the sandpit. Increased data flows to and from the data lake meant automation, monitoring and QA. Existing BAU work that took up large chunks of time needed to be reduced or automated so that we had more time to concentrate on the future. Creation of weather alerting and self service apps to replace existing event processing services needed to be written. Heavy reliance on for on-call support in GSbational during a major event when you're fielding at second slip on your Saturday cricket team is not an ideal phone call to get. We also needed to stop thinking like GIS people and start thinking like developers. Code creation and deployment through GitHub. We actually won a best failure award at our division awards. Four hours, four lines of code, but finally we got everybody up to speed and working with Git. The hero brings with them the object of their quest and they use better than ordinary world in some way whether it's through the knowledge or a cure or some form of protection, the return home. So this is our current state stack which I won't bore you with but we have moved all our instances within the same physical space as the data lake into our larger enterprise data hub to be able to leverage it directly, move data in and out fast and Python and GS server now serve the majority of our services, leaflet, the majority of our web maps, but we're still open and we have Tableau running right throughout the organisation as our reporting framework but we support analysts with our shiny and so the key for us is to make sure that whatever we put out, make it open, everyone can use it. So we're looking back at all the elements we've brought together over the last few years, the list is long, but Sarah here is some of the most important ones. The power of Python has been a huge help, Python frameworks that can be built and be reusable, Python server, Flask and Gunicorn, don't know where we would be without it, pandas for doing a wide variety of tasks including on the fly analysis and integrating Python underneath the Postgres install, one thing that we figured out later on had to change that so to allow Postgres to use the underlying Python functions that were available to it. No, I post this, no, I post this, pause here, ST, there's some key things, ST subdivide, cluster your gist index, tune your Postgres, truncate an insert, set your fill factor as 50 when you're doing faster updates and materialise views and foreign data wrappers for the win and lots and lots of functions that we've written. And keep calm and get. This is not so more about the actual software itself, but more about how we do things, our mindset as developers rather than GIS people now, a lot of automation and preparing for a world for us on continuous integration and continuous delivery. What good is a bestowment of gifts if you can't share out? So our public GitHub repo where we've been able to reshare out some of the goodness that we've been able to build internally and obviously there's a lot of IP so not everything is shared but we try to share what we can and hopefully that this continues. As for the future, what it holds for us, AI and machine learning, Kafka, allowing us to stream real-time data, I'm really excited about leaving behind near real-time as in twice a day to real-time. That's a huge challenge for us and how Kafka can work with us in terms of spatial objects. Apache Airflow, Spark, Snowflake, Kubernetes, we don't stand still. We can't stand still. Our data infrastructure is changing. We will have to move with it. Lastly, in summing up, we've all been through an amazing journey. We're certainly not at the end. So my key comment to all of you, I guess, is to experiment, to innovate, fail fast, don't ever settle, use Python, use PostGIS, set up a hack time to get the team to play with the new technologies. We're a huge proponent of this. What wasn't right for you before, make sure you revisit it because it's changing as well. What wasn't right for you two years ago, especially with the amount of code changes that are going on, maybe it's worth another look. The tech radar is a little bit more fluid than just dismissing it and leaving it out there. Thank you very much. E.T.L. A lot of our E.T.L. processes, we actually just wrote using things like foreign data wrappers and Python. So a lot of the monitoring systems and everything, it's pretty easy for us to either use Python or foreign data wrappers to move data around. We kind of flick between the two depending on what it is. If we're doing bulk migrations, we do just entire dumps out and then just move those over. If we're doing table refreshes, the refresh materialised view is the best way that we can just sync the data between two datasets. It's fast, it's efficient, it can be done concurrently, it doesn't table lock, so it's really good for what we do. So yeah, there's no tool as such, we just pretty much wrote our own.
to action in the hero's journey represents an open source movements within Geospatial