Spatial is not special: architecting for high performance geo
Varley, Mark (AddressCloud)
Spatial is not special. Enlightenment comes from the realisation that a spatial index is just an index, a very cool one yes, but an index all the same. Gone are the days where the spatial database was the domain of a few large vendors and the GIS department. Spatial databases are everywhere now with almost every RDBMS and NoSQL DB supporting some kind of spatial index. In this presentation we delve into the realm of microservice architectures and containers and how applying the right tool for each job and how pushing geo-processing down the stack can cut response times from seconds to milliseconds and unlock the holy grail of IT architecture: unlimited scalability
 Okay, so we are going to start our last session. And so our speaker is Mark Folli and he's a developer but he's also an entrepreneur. So he's the founder of address cloud, which is a geo coding engine. Today is going to speak a little bit about high performance architecture and cloud. So let's welcome our speaker. Just test this out, you can hear me. Okay, great. That's a good start. So this is my third Phosphor G. And I don't know, I've got a bit of a theory. I'm not sure if it's true or not. I'd like to test it briefly. So I think that my theory is that people here fall into, generally fall into one of two camps. I think there are geo people who've fallen into IT and I think there are IT people who've fallen into geo. So just a brief show of hands, I know the audience. So who would class themselves as a geo person who has had to learn IT? Okay, and for the second camp, who's an IT person who's fallen into geo? Okay, that's quite a good split, kind of 50-50. So yeah, I'm in the latter camp. So I've worked for the last 15 years as a solution architect. My pretty much working exclusively in financial services and quite a lot of that time in insurance. And I've fallen into geo. I fell into geo about 10 years ago and I've worked on and off with geo projects ever since. And what I'm going to present today is it's nothing groundbreaking, it's not rocket science, it's just an example of something that I've worked on in it and I'm going to take a very simple use case and I'm going to kind of work through how that's evolved over the years and I think where it's heading next. I think it's quite a parallel, there's a few parallels that can be drawn to GIS and solution software architecture in general. So hopefully you'll gain something out of it. So first of all, the title, it's deliberately controversial. Of course, spatial is special. We wouldn't be here if it wasn't. As the introduction just now, I've built a business over the last year that has spatial technology at the heart of it and I love spatial technology. So it's not special, it is special, but it's not always special. So from an architectural perspective, what I struggled with when I first started working with spatial is there's so many new things to learn, there's so many new buzzwords and it's like, okay, you have to have this big platform and you need to do this. And I'm really looking to kind of peel away the layers of the onion and really get down to the core of what's the requirement, what we're actually trying to solve. And I think by doing that, if you keep it simple and if you can keep it simple, then your performance problems will really take care of themselves. So that's my take away message and I'm going to be coming back to that throughout the presentation. So yeah, having worked in insurance, well insurers worry a lot. That's what they're paid to do. Essentially they're professional gamblers. They're gambling with, they're gambling that our houses aren't going to burn down. They're gambling that we're not going to flood, that the building's not going to fall in the grounds and that's what they're doing. They're basically trying to make a profit for their shareholders based upon making good decisions and yeah, and if they generate revenue, then that's great. That's what they're there for. And obviously for us as consumers, we want to know that if that one of these terrible things does happen to us, that we're going to get paid. So if we look at the four pictures there, I mean the first two, a little bit more difficult to predict. The top two, obviously there, there's massive implications for geography and it's no surprise that one of the kind of biggest purchases of GIS technology is the insurance industry. So for us, I come from the UK where we have a fantastic postcode system. It's very, very detailed. It's down to normally down to like a couple of blocks or a block, 50 houses, 100 houses. But that's served us fine for a long time. So actuaries have been working for years with postcode tables and taking data from lots of data sources, looking at the history of their claims to try and work out where the bad hotspots are. And that was good and that was doing okay, but the problem is that postcodes don't flood or burn down, buildings burn down. So as great as the postcodes have been, actually what the insurers realized is that they could gain competitive advantage by looking beyond the postcode and saying, well, if we can get down to building level and our competitors are still doing things at postcode level, then that's great. It means we can give, imagine we have a street like that with the bottom of the street, so the river, the top of the street is not. You want to give the guy at the bottom of the hill a really bad price or say, forget about you, I'm not insuring you at all. But the customer at the top is a good customer. We want to give them a good price. So actually, if we can get down to building level, then we're already gaining competitive advantage. So what we're going to do, let's hire a GIS team. There's some smart people out there. We're going to employ a bunch of grads out of university. We're going to get some fantastic maps. We're going to go and invest and buy lots of really expensive and really great data. And we're going to build up models. So we'll build up, going back to the picture before, we can build up a flood model. We can build a subsidence model. We can build a crime layer. We can potentially look at arson and work out where we think buildings might burn down. And there's a whole bunch of other cool stuff that we can do with that. So yeah, that's what insurance companies were doing probably 15 years ago, I guess, when I entered the arena. And we've got some pretty happy business users. And they're saying, that's great. Suddenly now, we're not working blind and we can start providing better prices for our customers. But the problem was the GIS teams were becoming so busy answering the business inquiries that they didn't have any time left to do any geography in GIS. So these guys are getting maxed out. There's normally quite small teams and they're having to answer these questions all the time. So well, another solution is we can build a web portal. So let's take the desktop experience and what the GIS team were doing on the desktop. Let's push it onto a server. OK. And we push it onto a server that business users can help themselves. Fantastic problem solved. But at the time, we didn't have any experience. We knew how the desktop works, but we didn't have any experience of delivering these kind of applications to the server. And it's geo, right? It's really, really complicated. So we better call the experts in. So we engaged with a very large purveyor of GIS solutions who people may know of. And we invited them in and they said, well, this is what you need. You need a three tier architecture. And you need this GIS application server in there. That's really important. You need a service layer. You need to put a data gateway. And at the bottom, you obviously still have your databases and your files. But if you do that, that's fantastic. And then the users will be able to look up and serve themselves. Yeah, we've got fantastic. The users are suddenly happy. They can serve themselves. The GIS guys are free again. And they can start working on geography. Life's good. But we've got another problem. So the problem now is we're having to, the business users doing it themselves, but they have a customer on the phone and they're saying, right, OK. So we're going to give you a quotation, but we still need to go and look up and see where you are on the map. And OK, right, you look like you're in a bad flat area, but you're quite good. And so this process was still quite cumbersome and quite long. And the majority of the business that was being written was not going through this process. It was just not possible. So let's build the GIS platform. So let's take what we already had, which was essentially a web version of the desktop. OK, and then let's integrate that with our front line systems. So if you're a customer at home and you're getting your web quotation, you obviously, you never see a map. You're not going to go and choose your building on a map. But behind the scenes, what the solution is actually doing is it is doing a geographic lookup. So it's GIS, but without the maps. But behind the scenes is going through and interrogating all of those different layers to give you as the customer a price that reflects your house and not your neighbor's house. So we're doing OK. So we will get the experts in. Obviously now the requirements become a bit more complicated. So we need a much, much more complicated architecture. So now we need a big tier at the top. We need our integration layer. We need to be able to plug the solution into all of the front line systems. Yeah, well, we need security. We need to have all of this stuff sitting in the middle. And then obviously at the bottom, we have our desktop tier so we can push up. So we've got our all singing, all dancing, amazing GIS platform. So yeah, we're really happy. OK, that's great now. So our customers are getting good prices and life's looking good. We're writing less bad risks. We're getting less big claims. And life's getting, you know, life's looking pretty good for their insurance company. But the problem is most people, I don't know, just again, another show of hands. So if you get your insurance, how many people would just go to, I don't know, in the UK we have money supermarket, we have aggregators, we have compare the market. Who when they go and get their insurance basically just goes onto an aggregator website, looks all the prices, gets the best one. Yeah, OK, exactly. And that's probably around similar. So most people now they don't go and sit there and look at every single different insurance companies. They go onto one of these great aggregators and it's OK, this is where I live, bang, bang, bang, bang, bang, take the cheapest. But the problem is that this fantastic GIS platform that we've invested in, the data that we've invested in, the teams that we've invested in, all of that stuff's working in our data center. Great. And it doesn't work for our aggregator business and that's probably 60, 70% of our business now. So actually we've invested a huge amount of money and technology into essentially giving a good price for 30% of our business. The other 70%, who knows? We're still taking a gamble and we're still working back on the post codes. So what we're going to do, well we can take that great enterprise GIS platform that we built in-house and let's just scale it out. It gets a little bit tricky. The problem is what works well on a couple of servers when you're scaling out and you're looking at having to deliver a sub-second performance, you're looking to be able to process thousands of transactions per second instead of 20, 30 transactions a second, the thing doesn't scale anymore. And if it does scale, someone's going to be unhappy. Basically the CFO is not going to be very pleased. So really what do we do? I mean, where do we go from here? Okay, we don't have a scalable solution. We've got all of this very expensive proprietary technology sitting in it. Where do we go? What we can go to the open source community? They've got some fantastic offerings that have come out of probably some of the people in this room. Absolutely and definitely the way to go. But actually at this time we need to maybe take a step back and think about what was the business requirement? What will we try and to solve? Do we need to have all of these layers in our architecture? And actually that's just really, if we think about the requirement, we just want to know the risk of bad stuff happening at a location. Do we need a map? Not really, no, not for this. We don't have the luxury of time for a map. We just need somewhere that we can provide an assessment, a building level of various different risk layers. And we want to do that as quickly as possible for as many people as possible. So actually we could probably just do something like this. So instead of having this big platform, actually all we probably need is an API that speaks and sends out GeoJSON or JSON if we're not using a map. That could just be a simple little Python script or a little job calling PostGIS directly. And do you know what? We don't need to have a GIS application server. We don't need all of these other layers. If we took it down to the one function it needs to perform, which is give me an assessment of these layers at this location, that's all we need. And by breaking it down to something small like that, actually now in terms of scaling that, you know, life becomes a little bit easier. So we don't have to even necessarily implement it in our... So if we were using.NET we could go and use Python or we could use Node.js. We could carry on using SQL Server if we wanted to or we could go and look at an open source software in such as Spatial Light or PostGIS. And suddenly our unit of scale has reduced and now to be able to scale up something like this is a lot easier. But we think we could probably go a step further, actually. So even like this we're still being able to perform. Even if you strip away the onion and get rid of all of the complexity in the mid tier, you're still having to hit the database to querying, raster layers, vector layers, having to do quite a few lookups, get the response back and get that back crucially has to be sub-second. So actually we kind of scratched our heads and thought, well, do we need to be doing all of this spatial stuff every time we do a lookup? Well, sometimes yes, but is there a way that we could pre-calculate the answers? If we knew the answers in advance, we wouldn't have to calculate them for the customer. So we kind of scratched our heads and had a bit of a think about it and then we had a bit of a eureka moment. We know where all the addresses are because we buy the data. So it's a national address dataset. We know the location of 37 million addresses in the UK and we know exactly where they all are. And our customers are probably only going to be looking to get an answer at one of those 37 million locations. And the other thing as well is actually at the moment, all we're doing, I mentioned at the beginning that post codes don't flood, buildings flood. Well, at the moment, what we've gone from is post codes to points. So at the moment, if we have a flood layer that cuts through the middle of a building and the entrance is over here, but the layer is over here and the majority of the flood is happening the rest of the building, actually we're giving an incorrect answer. But because of the time constraints and having to do the GIS at a point of query, we don't have the luxury of doing more complicated analysis. But if we go with this approach, we could potentially pre-cash them and answer all those questions up front. So what do we do? We've got a unique, in the UK now, we've just open sourced, a year and a half ago, they just open sourced a unique property reference number. So every single building in the UK, every single address has a unique reference identifier. It's open source and it's available. But it's going to take forever to analyze all of that data. How are we going to do that? That's going to, also every time the data changes, it's going to take a long time. Not at all now, so we can split the problem up into chunks, we could use a cluster, if we wanted to, we could use a cloud-based cluster and essentially we could answer that question, what is the risk for all of those buildings? And we could probably get that down to hours rather than days or weeks. But the question comes up, but we don't always ensure buildings. So part of insurance is I'm a property developer, I've just bought this piece of land over there. I want to ensure it, so that obviously is not going to have a unique identifier and we're going to have to use the geographic approach. That's no problem at all, so let's keep the old solution for the 1%. So we end up with something that potentially could look like this. So we've got our address, we want to describe an address. For 99% of the cases, we can just do it on the identifier, but for the small percentage of the cases where we can't do that, we go ahead and do it on the spatial database. No problem at all. And kind of what we've got here inadvertently, without wanting to bring out our buzzword, is we found ourselves here now with a microservices architecture. We didn't go around doing it, we didn't look on the line and see this fantastic thing and jump on the bandwagon and say, oh, we need to do microservices. We've just kind of fallen into it because we are just by simplifying and solving one problem. And what we've got now is we've got something actually on the left here, because there's no spatial in here at all, we could use pretty much any technology that we want to. We still got our spatial over here on the right that's powering the spatial queries, but on the left there, we just really just got a key value store. So we can scale out, we can cluster, and suddenly we're able to meet, not sub-second, but here we're probably talking sub-100 milliseconds, sub-50 milliseconds. We've got something that's super, super, super fast. So actually, another advantage of this is when our customers go online and get their quotation and bang, bang, bang, actually the guys who respond the quickest will normally end up at the top. So not only of that, we end up coming further up the list. And the other hidden benefit as well for this, I don't know, has anyone heard of or experimented with serverless architectures? Quite a new thing. Yeah. Not one for today, but maybe one for another talk. But this opens the door for a serverless architecture. So suddenly now we have a simple function that does one thing. It calls a key value store and returns some data. Over on the left there for people who aren't AWS customers, I'm a big fan of AWS, Amazon. We have a Lambda service and we have a DynamoDB store. And that thing can sit there, and if it's not running, it doesn't cost us any money. It only costs us money when we use it. So suddenly we have not only have we eliminated complexity, we've got a faster service, we can pretty much run it for next to nothing. So the message really is that you're free to use the right tools for the job. If you're interested a bit more in these architectures, there's a couple of links there to some videos and YouTube clips. And we're happy. We're happy developers. We've got happy business users. Obviously, this isn't silver bullets. It's not going to help you with all of your really complex geospatial problems. If you wanted to take your desktop architecture and be able to publish services very easy, then great, go and use a GIS platform. But just when you go back and you're looking at this, find yourself questioning, do I really need all of these layers? Do I need a full function in GIS platform? Do I need to be able to support every single standard, or do I just need something that does one thing really well? So to summarize, I would say that spatial is not always really special. You should optimize for your requirements. Beware the platform and approach the platform with caution and really question the use of the platform. Cash it if you can. So if you're able to answer the question in advance, don't incur all that overhead at query time. But above all, just keep it simple, and it will scale. And that's my message for the day. Okay, so I think we have time for a few questions. I'm sure they're going to be some. No, so all we I'll just flick back to the sliding question. So okay, I've lost the slides, but that's fine. Maybe the data that we receive, I don't know whereabouts you come from? From DC, okay. So the question was going back to the slide where I showed the picture of the UK in chunks and how we distributed the processing. Well the way the data is actually delivered, so the data we typically use comes from the Ordnance Survey in the UK, and it actually is delivered in geographic chunks. So that grid layout, that's the way the data is actually delivered. So it's kind of already ready. So it's already that positioning is already done for us. So we just take that grid, apply the same to the addresses, so we'll take just all the addresses from within that grid. We would snapshot out, for example, our flood model just for that grid, and have that queued up and running on a machine. And we can run as many of those in parallel as we've got the money for. Okay, no problem. Yeah, one question on the back. Not very well, no, sorry. Well I'm taking, so this is from my perspective now, so essentially I'm running a, this is a SaaS company now. So the business already looking to, or business already looking to offload any kind of use cases that are not secure to SaaS providers. And this one is kind of perfect. There's nothing financial here, there's nothing that would really worry a big insurance company. Essentially this is just around providing geographic assessments. So I'm lucky that I didn't have to have the conversation. So yeah, this is my take on it as a SaaS provider who are free to use whatever technology I want to. And I'm seeing, I've certainly seen clients that I've worked with in the past before I was running my own company who are totally embracing the cloud, totally embracing microservices. So yeah, I think this is, I think it's the future, I think it's the shape of things to come. Hi. The more personal question. Yeah. Yes. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Just the barrier to end, so what I found, I can tell you, I remember probably one of the first conversations that I had as an IT person that having a conversation with a GIS person was our interpretation of what was a map. You might have had similar things. So we were probably sitting in a meeting for maybe an hour and I'm talking to this guy and we've been talking about this map and for me, the map was the sum of the parts. For me, a map is what you see on Google at the time or if I had my Ordnance Survey map at home. And for him, he was thinking of the map, well, were you talking about the base map or the layers or the, I was like, what's a base map? Just that kind of barrier to end, I found it really, really hard. And I think I probably made decisions and gave advice earlier on when I started off looking at this, just thinking as an outsider, well, you must have to have a GIS platform. I mean, that's absolutely, how can you do a GIS without a GIS platform? So yeah, I think it takes a while to actually break it down and simplify and actually what we're doing necessarily is not necessarily that complicated. Does that answer the question? Yeah. One question. So from your perspective, with your experience, what would you think would be the biggest challenge if you want to transform like a monolithic application, let's say, into microservices? So I mean, you remove the complexity out of the small blocks, but then you have like the orchestration of the... Yeah, absolutely. It's definitely not a silver bullet and you need to approach it with caution. I haven't had to really take something massive and break it up. From what I've read and what I've understood is that the hardest thing is how you partition. And what do you do? So for a microservices architecture, really the kind of guiding principle is that you should never share data. You should never share databases. So you would have each function or each piece of the application would look at its own data store. So that's quite a good way of segregating it. But if you make a bad decision earlier on, especially if you're just getting into this, if you make a bad decision, you could end up adding huge layers of additional complexity. So I think that would be a bit of a challenge. There's a bunch of approaches. If you do have an old monolithic application that you want to kind of break up, it's quite good to think about the services, build a service layer on top of it first, get that to a level where you're happy with it, even if it's just calling the monolith. And then after that, then that's the point where you can start partitioning off. So does that make sense? Is that answer a question? Yeah, that's a very good advice. Okay. Thank you. So any more questions? Okay, so let's thank the speaker and also the other speakers. And I think this will be the last session, the last talk from this session. Thank you.
 Okay, so we are going to start our last session.  And so our speaker is Mark Folli and he's a developer but he's also an entrepreneur.  So he's the founder of address cloud, which is a geo coding engine.  Today is going to speak a little bit about high performance architecture and cloud.  So let's welcome our speaker.  Just test this out, you can hear me.  Okay, great.  That's a good start.  So this is my third Phosphor G.  And I don't know, I've got a bit of a theory.  I'm not sure if it's true or not.  I'd like to test it briefly.  So I think that my theory is that people here fall into, generally fall into one of two  camps.  I think there are geo people who've fallen into IT and I think there are IT people who've  fallen into geo.  So just a brief show of hands, I know the audience.  So who would class themselves as a geo person who has had to learn IT?  Okay, and for the second camp, who's an IT person who's fallen into geo?  Okay, that's quite a good split, kind of 50-50.  So yeah, I'm in the latter camp.  So I've worked for the last 15 years as a solution architect.  My pretty much working exclusively in financial services and quite a lot of that time in insurance.  And I've fallen into geo.  I fell into geo about 10 years ago and I've worked on and off with geo projects ever since.  And what I'm going to present today is it's nothing groundbreaking, it's not rocket science,  it's just an example of something that I've worked on in it and I'm going to take a very  simple use case and I'm going to kind of work through how that's evolved over the years  and I think where it's heading next.  I think it's quite a parallel, there's a few parallels that can be drawn to GIS and solution  software architecture in general.  So hopefully you'll gain something out of it.  So first of all, the title, it's deliberately controversial.  Of course, spatial is special.  We wouldn't be here if it wasn't.  As the introduction just now, I've built a business over the last year that has spatial  technology at the heart of it and I love spatial technology.  So it's not special, it is special, but it's not always special.  So from an architectural perspective, what I struggled with when I first started working  with spatial is there's so many new things to learn, there's so many new buzzwords and  it's like, okay, you have to have this big platform and you need to do this.  And I'm really looking to kind of peel away the layers of the onion and really get down  to the core of what's the requirement, what we're actually trying to solve.  And I think by doing that, if you keep it simple and if you can keep it simple, then  your performance problems will really take care of themselves.  So that's my take away message and I'm going to be coming back to that throughout the presentation.  So yeah, having worked in insurance, well insurers worry a lot.  That's what they're paid to do.  Essentially they're professional gamblers.  They're gambling with, they're gambling that our houses aren't going to burn down.  They're gambling that we're not going to flood, that the building's not going to fall in the  grounds and that's what they're doing.  They're basically trying to make a profit for their shareholders based upon making good  decisions and yeah, and if they generate revenue, then that's great.  That's what they're there for.  And obviously for us as consumers, we want to know that if that one of these terrible  things does happen to us, that we're going to get paid.  So if we look at the four pictures there, I mean the first two, a little bit more difficult  to predict.  The top two, obviously there, there's massive implications for geography and it's no surprise  that one of the kind of biggest purchases of GIS technology is the insurance industry.  So for us, I come from the UK where we have a fantastic postcode system.  It's very, very detailed.  It's down to normally down to like a couple of blocks or a block, 50 houses, 100 houses.  But that's served us fine for a long time.  So actuaries have been working for years with postcode tables and taking data from lots  of data sources, looking at the history of their claims to try and work out where the  bad hotspots are.  And that was good and that was doing okay, but the problem is that postcodes don't flood  or burn down, buildings burn down.  So as great as the postcodes have been, actually what the insurers realized is that they could  gain competitive advantage by looking beyond the postcode and saying, well, if we can get  down to building level and our competitors are still doing things at postcode level,  then that's great.  It means we can give, imagine we have a street like that with the bottom of the street, so  the river, the top of the street is not.  You want to give the guy at the bottom of the hill a really bad price or say, forget  about you, I'm not insuring you at all.  But the customer at the top is a good customer.  We want to give them a good price.  So actually, if we can get down to building level, then we're already gaining competitive  advantage.  So what we're going to do, let's hire a GIS team.  There's some smart people out there.  We're going to employ a bunch of grads out of university.  We're going to get some fantastic maps.  We're going to go and invest and buy lots of really expensive and really great data.  And we're going to build up models.  So we'll build up, going back to the picture before, we can build up a flood model.  We can build a subsidence model.  We can build a crime layer.  We can potentially look at arson and work out where we think buildings might burn down.  And there's a whole bunch of other cool stuff that we can do with that.  So yeah, that's what insurance companies were doing probably 15 years ago, I guess, when  I entered the arena.  And we've got some pretty happy business users.  And they're saying, that's great.  Suddenly now, we're not working blind and we can start providing better prices for our  customers.  But the problem was the GIS teams were becoming so busy answering the business inquiries that  they didn't have any time left to do any geography in GIS.  So these guys are getting maxed out.  There's normally quite small teams and they're having to answer these questions all the time.  So well, another solution is we can build a web portal.  So let's take the desktop experience and what the GIS team were doing on the desktop.  Let's push it onto a server.  OK.  And we push it onto a server that business users can help themselves.  Fantastic problem solved.  But at the time, we didn't have any experience.  We knew how the desktop works, but we didn't have any experience of delivering these kind  of applications to the server.  And it's geo, right?  It's really, really complicated.  So we better call the experts in.  So we engaged with a very large purveyor of GIS solutions who people may know of.  And we invited them in and they said, well, this is what you need.  You need a three tier architecture.  And you need this GIS application server in there.  That's really important.  You need a service layer.  You need to put a data gateway.  And at the bottom, you obviously still have your databases and your files.  But if you do that, that's fantastic.  And then the users will be able to look up and serve themselves.  Yeah, we've got fantastic.  The users are suddenly happy.  They can serve themselves.  The GIS guys are free again.  And they can start working on geography.  Life's good.  But we've got another problem.  So the problem now is we're having to, the business users doing it themselves, but they  have a customer on the phone and they're saying, right, OK.  So we're going to give you a quotation, but we still need to go and look up and see where  you are on the map.  And OK, right, you look like you're in a bad flat area, but you're quite good.  And so this process was still quite cumbersome and quite long.  And the majority of the business that was being written was not going through this process.  It was just not possible.  So let's build the GIS platform.  So let's take what we already had, which was essentially a web version of the desktop.  OK, and then let's integrate that with our front line systems.  So if you're a customer at home and you're getting your web quotation, you obviously,  you never see a map.  You're not going to go and choose your building on a map.  But behind the scenes, what the solution is actually doing is it is doing a geographic  lookup.  So it's GIS, but without the maps.  But behind the scenes is going through and interrogating all of those different layers  to give you as the customer a price that reflects your house and not your neighbor's house.  So we're doing OK.  So we will get the experts in.  Obviously now the requirements become a bit more complicated.  So we need a much, much more complicated architecture.  So now we need a big tier at the top.  We need our integration layer.  We need to be able to plug the solution into all of the front line systems.  Yeah, well, we need security.  We need to have all of this stuff sitting in the middle.  And then obviously at the bottom, we have our desktop tier so we can push up.  So we've got our all singing, all dancing, amazing GIS platform.  So yeah, we're really happy.  OK, that's great now.  So our customers are getting good prices and life's looking good.  We're writing less bad risks.  We're getting less big claims.  And life's getting, you know, life's looking pretty good for their insurance company.  But the problem is most people, I don't know, just again, another show of hands.  So if you get your insurance, how many people would just go to, I don't know, in the UK  we have money supermarket, we have aggregators, we have compare the market.  Who when they go and get their insurance basically just goes onto an aggregator website, looks  all the prices, gets the best one.  Yeah, OK, exactly.  And that's probably around similar.  So most people now they don't go and sit there and look at every single different insurance  companies.  They go onto one of these great aggregators and it's OK, this is where I live, bang, bang,  bang, bang, bang, take the cheapest.  But the problem is that this fantastic GIS platform that we've invested in, the data  that we've invested in, the teams that we've invested in, all of that stuff's working in  our data center.  Great.  And it doesn't work for our aggregator business and that's probably 60, 70% of our business  now.  So actually we've invested a huge amount of money and technology into essentially giving  a good price for 30% of our business.  The other 70%, who knows?  We're still taking a gamble and we're still working back on the post codes.  So what we're going to do, well we can take that great enterprise GIS platform that we  built in-house and let's just scale it out.  It gets a little bit tricky.  The problem is what works well on a couple of servers when you're scaling out and you're  looking at having to deliver a sub-second performance, you're looking to be able to process thousands  of transactions per second instead of 20, 30 transactions a second, the thing doesn't  scale anymore.  And if it does scale, someone's going to be unhappy.  Basically the CFO is not going to be very pleased.  So really what do we do?  I mean, where do we go from here?  Okay, we don't have a scalable solution.  We've got all of this very expensive proprietary technology sitting in it.  Where do we go?  What we can go to the open source community?  They've got some fantastic offerings that have come out of probably some of the people  in this room.  Absolutely and definitely the way to go.  But actually at this time we need to maybe take a step back and think about what was the  business requirement?  What will we try and to solve?  Do we need to have all of these layers in our architecture?  And actually that's just really, if we think about the requirement, we just want to know  the risk of bad stuff happening at a location.  Do we need a map?  Not really, no, not for this.  We don't have the luxury of time for a map.  We just need somewhere that we can provide an assessment, a building level of various  different risk layers.  And we want to do that as quickly as possible for as many people as possible.  So actually we could probably just do something like this.  So instead of having this big platform, actually all we probably need is an API that speaks  and sends out GeoJSON or JSON if we're not using a map.  That could just be a simple little Python script or a little job calling PostGIS directly.  And do you know what?  We don't need to have a GIS application server.  We don't need all of these other layers.  If we took it down to the one function it needs to perform, which is give me an assessment  of these layers at this location, that's all we need.  And by breaking it down to something small like that, actually now in terms of scaling  that, you know, life becomes a little bit easier.  So we don't have to even necessarily implement it in our...  So if we were using.NET we could go and use Python or we could use Node.js.  We could carry on using SQL Server if we wanted to or we could go and look at an open source  software in such as Spatial Light or PostGIS.  And suddenly our unit of scale has reduced and now to be able to scale up something like  this is a lot easier.  But we think we could probably go a step further, actually.  So even like this we're still being able to perform.  Even if you strip away the onion and get rid of all of the complexity in the mid tier,  you're still having to hit the database to querying, raster layers, vector layers, having  to do quite a few lookups, get the response back and get that back crucially has to be  sub-second.  So actually we kind of scratched our heads and thought, well, do we need to be doing  all of this spatial stuff every time we do a lookup?  Well, sometimes yes, but is there a way that we could pre-calculate the answers?  If we knew the answers in advance, we wouldn't have to calculate them for the customer.  So we kind of scratched our heads and had a bit of a think about it and then we had  a bit of a eureka moment.  We know where all the addresses are because we buy the data.  So it's a national address dataset.  We know the location of 37 million addresses in the UK and we know exactly where they all  are.  And our customers are probably only going to be looking to get an answer at one of those  37 million locations.  And the other thing as well is actually at the moment, all we're doing, I mentioned at  the beginning that post codes don't flood, buildings flood.  Well, at the moment, what we've gone from is post codes to points.  So at the moment, if we have a flood layer that cuts through the middle of a building  and the entrance is over here, but the layer is over here and the majority of the flood  is happening the rest of the building, actually we're giving an incorrect answer.  But because of the time constraints and having to do the GIS at a point of query, we don't  have the luxury of doing more complicated analysis.  But if we go with this approach, we could potentially pre-cash them and answer all those  questions up front.  So what do we do?  We've got a unique, in the UK now, we've just open sourced, a year and a half ago, they  just open sourced a unique property reference number.  So every single building in the UK, every single address has a unique reference identifier.  It's open source and it's available.  But it's going to take forever to analyze all of that data.  How are we going to do that?  That's going to, also every time the data changes, it's going to take a long time.  Not at all now, so we can split the problem up into chunks, we could use a cluster, if  we wanted to, we could use a cloud-based cluster and essentially we could answer that question,  what is the risk for all of those buildings?  And we could probably get that down to hours rather than days or weeks.  But the question comes up, but we don't always ensure buildings.  So part of insurance is I'm a property developer, I've just bought this piece of land over there.  I want to ensure it, so that obviously is not going to have a unique identifier and we're  going to have to use the geographic approach.  That's no problem at all, so let's keep the old solution for the 1%.  So we end up with something that potentially could look like this.  So we've got our address, we want to describe an address.  For 99% of the cases, we can just do it on the identifier, but for the small percentage  of the cases where we can't do that, we go ahead and do it on the spatial database.  No problem at all.  And kind of what we've got here inadvertently, without wanting to bring out our buzzword,  is we found ourselves here now with a microservices architecture.  We didn't go around doing it, we didn't look on the line and see this fantastic thing and  jump on the bandwagon and say, oh, we need to do microservices.  We've just kind of fallen into it because we are just by simplifying and solving one  problem.  And what we've got now is we've got something actually on the left here, because there's  no spatial in here at all, we could use pretty much any technology that we want to.  We still got our spatial over here on the right that's powering the spatial queries,  but on the left there, we just really just got a key value store.  So we can scale out, we can cluster, and suddenly we're able to meet, not sub-second, but here  we're probably talking sub-100 milliseconds, sub-50 milliseconds.  We've got something that's super, super, super fast.  So actually, another advantage of this is when our customers go online and get their  quotation and bang, bang, bang, actually the guys who respond the quickest will normally  end up at the top.  So not only of that, we end up coming further up the list.  And the other hidden benefit as well for this, I don't know, has anyone heard of or  experimented with serverless architectures?  Quite a new thing.  Yeah.  Not one for today, but maybe one for another talk.  But this opens the door for a serverless architecture.  So suddenly now we have a simple function that does one thing.  It calls a key value store and returns some data.  Over on the left there for people who aren't AWS customers, I'm a big fan of AWS, Amazon.  We have a Lambda service and we have a DynamoDB store.  And that thing can sit there, and if it's not running, it doesn't cost us any money.  It only costs us money when we use it.  So suddenly we have not only have we eliminated complexity, we've got a faster service, we  can pretty much run it for next to nothing.  So the message really is that you're free to use the right tools for the job.  If you're interested a bit more in these architectures, there's a couple of links there to some videos  and YouTube clips.  And we're happy.  We're happy developers.  We've got happy business users.  Obviously, this isn't silver bullets.  It's not going to help you with all of your really complex geospatial problems.  If you wanted to take your desktop architecture and be able to publish services very easy,  then great, go and use a GIS platform.  But just when you go back and you're looking at this, find yourself questioning, do I really  need all of these layers?  Do I need a full function in GIS platform?  Do I need to be able to support every single standard, or do I just need something that  does one thing really well?  So to summarize, I would say that spatial is not always really special.  You should optimize for your requirements.  Beware the platform and approach the platform with caution and really question the use of  the platform.  Cash it if you can.  So if you're able to answer the question in advance, don't incur all that overhead at  query time.  But above all, just keep it simple, and it will scale.  And that's my message for the day.  Okay, so I think we have time for a few questions.  I'm sure they're going to be some.  No, so all we I'll just flick back to the sliding question.  So okay, I've lost the slides, but that's fine.  Maybe the data that we receive, I don't know whereabouts you come from?  From DC, okay.  So the question was going back to the slide where I showed the picture of the UK in chunks  and how we distributed the processing.  Well the way the data is actually delivered, so the data we typically use comes from the  Ordnance Survey in the UK, and it actually is delivered in geographic chunks.  So that grid layout, that's the way the data is actually delivered.  So it's kind of already ready.  So it's already that positioning is already done for us.  So we just take that grid, apply the same to the addresses, so we'll take just all the  addresses from within that grid.  We would snapshot out, for example, our flood model just for that grid, and have that queued  up and running on a machine.  And we can run as many of those in parallel as we've got the money for.  Okay, no problem.  Yeah, one question on the back.  Not very well, no, sorry.  Well I'm taking, so this is from my perspective now, so essentially I'm running a, this is  a SaaS company now.  So the business already looking to, or business already looking to offload any kind of use  cases that are not secure to SaaS providers.  And this one is kind of perfect.  There's nothing financial here, there's nothing that would really worry a big insurance company.  Essentially this is just around providing geographic assessments.  So I'm lucky that I didn't have to have the conversation.  So yeah, this is my take on it as a SaaS provider who are free to use whatever technology  I want to.  And I'm seeing, I've certainly seen clients that I've worked with in the past before I  was running my own company who are totally embracing the cloud, totally embracing microservices.  So yeah, I think this is, I think it's the future, I think it's the shape of things to  come.  Hi.  The more personal question.  Yeah.  Yes.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Just the barrier to end, so what I found, I can tell you, I remember probably one of  the first conversations that I had as an IT person that having a conversation with a  GIS person was our interpretation of what was a map.  You might have had similar things.  So we were probably sitting in a meeting for maybe an hour and I'm talking to this guy  and we've been talking about this map and for me, the map was the sum of the parts.  For me, a map is what you see on Google at the time or if I had my Ordnance Survey map  at home.  And for him, he was thinking of the map, well, were you talking about the base map  or the layers or the, I was like, what's a base map?  Just that kind of barrier to end, I found it really, really hard.  And I think I probably made decisions and gave advice earlier on when I started off  looking at this, just thinking as an outsider, well, you must have to have a GIS platform.  I mean, that's absolutely, how can you do a GIS without a GIS platform?  So yeah, I think it takes a while to actually break it down and simplify and actually what  we're doing necessarily is not necessarily that complicated.  Does that answer the question?  Yeah.  One question.  So from your perspective, with your experience, what would you think would be the biggest  challenge if you want to transform like a monolithic application, let's say, into microservices?  So I mean, you remove the complexity out of the small blocks, but then you have like the  orchestration of the...  Yeah, absolutely.  It's definitely not a silver bullet and you need to approach it with caution.  I haven't had to really take something massive and break it up.  From what I've read and what I've understood is that the hardest thing is how you partition.  And what do you do?  So for a microservices architecture, really the kind of guiding principle is that you  should never share data.  You should never share databases.  So you would have each function or each piece of the application would look at its own data  store.  So that's quite a good way of segregating it.  But if you make a bad decision earlier on, especially if you're just getting into this,  if you make a bad decision, you could end up adding huge layers of additional complexity.  So I think that would be a bit of a challenge.  There's a bunch of approaches.  If you do have an old monolithic application that you want to kind of break up, it's quite  good to think about the services, build a service layer on top of it first, get that  to a level where you're happy with it, even if it's just calling the monolith.  And then after that, then that's the point where you can start partitioning off.  So does that make sense?  Is that answer a question?  Yeah, that's a very good advice.  Okay.  Thank you.  So any more questions?  Okay, so let's thank the speaker and also the other speakers.  And I think this will be the last session, the last talk from this session.  Thank you.
foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_0_215.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_216_299.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_300_4319.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_4320_5015.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_5016_5687.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_5688_6839.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_6840_8903.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_8904_10019.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_10020_12047.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_12048_15395.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_15396_17543.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_17544_22691.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_22692_24815.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_24816_29879.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_29880_30815.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_30816_31655.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_31656_32483.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_32484_32591.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_32592_32639.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_32640_32699.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_32700_32951.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_32952_33887.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_33888_34019.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_34020_35423.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_35424_35591.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_35592_35975.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_35976_36179.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_36180_36623.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_36624_37103.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_37104_37487.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_37488_37811.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_37812_37883.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_37884_38591.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_38592_39203.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_39204_39263.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_39264_39839.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_39840_40031.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_40032_41627.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_41628_41723.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_41724_42167.png foss4g-2016-1221-spatial_is_not_special_architecting_for_high_performance_geo-hd_42168_42319.png
Okay, so we are going to start our last session. And so our speaker is Mark Folli and he's a developer but he's also an entrepreneur. So he's the founder of address cloud, which is a geo coding engine. Today is going to speak a little bit about high performance architecture and cloud. So let's welcome our speaker. Just test this out, you can hear me. Okay, great. That's a good start. So this is my third Phosphor G. And I don't know, I've got a bit of a theory. I'm not sure if it's true or not. I'd like to test it briefly. So I think that my theory is that people here fall into, generally fall into one of two camps. I think there are geo people who've fallen into IT and I think there are IT people who've fallen into geo. So just a brief show of hands, I know the audience. So who would class themselves as a geo person who has had to learn IT? Okay, and for the second camp, who's an IT person who's fallen into geo? Okay, that's quite a good split, kind of 50-50. So yeah, I'm in the latter camp. So I've worked for the last 15 years as a solution architect. My pretty much working exclusively in financial services and quite a lot of that time in insurance. And I've fallen into geo. I fell into geo about 10 years ago and I've worked on and off with geo projects ever since. And what I'm going to present today is it's nothing groundbreaking, it's not rocket science, it's just an example of something that I've worked on in it and I'm going to take a very simple use case and I'm going to kind of work through how that's evolved over the years and I think where it's heading next. I think it's quite a parallel, there's a few parallels that can be drawn to GIS and solution software architecture in general. So hopefully you'll gain something out of it. So first of all, the title, it's deliberately controversial. Of course, spatial is special. We wouldn't be here if it wasn't. As the introduction just now, I've built a business over the last year that has spatial technology at the heart of it and I love spatial technology. So it's not special, it is special, but it's not always special. So from an architectural perspective, what I struggled with when I first started working with spatial is there's so many new things to learn, there's so many new buzzwords and it's like, okay, you have to have this big platform and you need to do this. And I'm really looking to kind of peel away the layers of the onion and really get down to the core of what's the requirement, what we're actually trying to solve. And I think by doing that, if you keep it simple and if you can keep it simple, then your performance problems will really take care of themselves. So that's my take away message and I'm going to be coming back to that throughout the presentation. So yeah, having worked in insurance, well insurers worry a lot. That's what they're paid to do. Essentially they're professional gamblers. They're gambling with, they're gambling that our houses aren't going to burn down. They're gambling that we're not going to flood, that the building's not going to fall in the grounds and that's what they're doing. They're basically trying to make a profit for their shareholders based upon making good decisions and yeah, and if they generate revenue, then that's great. That's what they're there for. And obviously for us as consumers, we want to know that if that one of these terrible things does happen to us, that we're going to get paid. So if we look at the four pictures there, I mean the first two, a little bit more difficult to predict. The top two, obviously there, there's massive implications for geography and it's no surprise that one of the kind of biggest purchases of GIS technology is the insurance industry. So for us, I come from the UK where we have a fantastic postcode system. It's very, very detailed. It's down to normally down to like a couple of blocks or a block, 50 houses, 100 houses. But that's served us fine for a long time. So actuaries have been working for years with postcode tables and taking data from lots of data sources, looking at the history of their claims to try and work out where the bad hotspots are. And that was good and that was doing okay, but the problem is that postcodes don't flood or burn down, buildings burn down. So as great as the postcodes have been, actually what the insurers realized is that they could gain competitive advantage by looking beyond the postcode and saying, well, if we can get down to building level and our competitors are still doing things at postcode level, then that's great. It means we can give, imagine we have a street like that with the bottom of the street, so the river, the top of the street is not. You want to give the guy at the bottom of the hill a really bad price or say, forget about you, I'm not insuring you at all. But the customer at the top is a good customer. We want to give them a good price. So actually, if we can get down to building level, then we're already gaining competitive advantage. So what we're going to do, let's hire a GIS team. There's some smart people out there. We're going to employ a bunch of grads out of university. We're going to get some fantastic maps. We're going to go and invest and buy lots of really expensive and really great data. And we're going to build up models. So we'll build up, going back to the picture before, we can build up a flood model. We can build a subsidence model. We can build a crime layer. We can potentially look at arson and work out where we think buildings might burn down. And there's a whole bunch of other cool stuff that we can do with that. So yeah, that's what insurance companies were doing probably 15 years ago, I guess, when I entered the arena. And we've got some pretty happy business users. And they're saying, that's great. Suddenly now, we're not working blind and we can start providing better prices for our customers. But the problem was the GIS teams were becoming so busy answering the business inquiries that they didn't have any time left to do any geography in GIS. So these guys are getting maxed out. There's normally quite small teams and they're having to answer these questions all the time. So well, another solution is we can build a web portal. So let's take the desktop experience and what the GIS team were doing on the desktop. Let's push it onto a server. OK. And we push it onto a server that business users can help themselves. Fantastic problem solved. But at the time, we didn't have any experience. We knew how the desktop works, but we didn't have any experience of delivering these kind of applications to the server. And it's geo, right? It's really, really complicated. So we better call the experts in. So we engaged with a very large purveyor of GIS solutions who people may know of. And we invited them in and they said, well, this is what you need. You need a three tier architecture. And you need this GIS application server in there. That's really important. You need a service layer. You need to put a data gateway. And at the bottom, you obviously still have your databases and your files. But if you do that, that's fantastic. And then the users will be able to look up and serve themselves. Yeah, we've got fantastic. The users are suddenly happy. They can serve themselves. The GIS guys are free again. And they can start working on geography. Life's good. But we've got another problem. So the problem now is we're having to, the business users doing it themselves, but they have a customer on the phone and they're saying, right, OK. So we're going to give you a quotation, but we still need to go and look up and see where you are on the map. And OK, right, you look like you're in a bad flat area, but you're quite good. And so this process was still quite cumbersome and quite long. And the majority of the business that was being written was not going through this process. It was just not possible. So let's build the GIS platform. So let's take what we already had, which was essentially a web version of the desktop. OK, and then let's integrate that with our front line systems. So if you're a customer at home and you're getting your web quotation, you obviously, you never see a map. You're not going to go and choose your building on a map. But behind the scenes, what the solution is actually doing is it is doing a geographic lookup. So it's GIS, but without the maps. But behind the scenes is going through and interrogating all of those different layers to give you as the customer a price that reflects your house and not your neighbor's house. So we're doing OK. So we will get the experts in. Obviously now the requirements become a bit more complicated. So we need a much, much more complicated architecture. So now we need a big tier at the top. We need our integration layer. We need to be able to plug the solution into all of the front line systems. Yeah, well, we need security. We need to have all of this stuff sitting in the middle. And then obviously at the bottom, we have our desktop tier so we can push up. So we've got our all singing, all dancing, amazing GIS platform. So yeah, we're really happy. OK, that's great now. So our customers are getting good prices and life's looking good. We're writing less bad risks. We're getting less big claims. And life's getting, you know, life's looking pretty good for their insurance company. But the problem is most people, I don't know, just again, another show of hands. So if you get your insurance, how many people would just go to, I don't know, in the UK we have money supermarket, we have aggregators, we have compare the market. Who when they go and get their insurance basically just goes onto an aggregator website, looks all the prices, gets the best one. Yeah, OK, exactly. And that's probably around similar. So most people now they don't go and sit there and look at every single different insurance companies. They go onto one of these great aggregators and it's OK, this is where I live, bang, bang, bang, bang, bang, take the cheapest. But the problem is that this fantastic GIS platform that we've invested in, the data that we've invested in, the teams that we've invested in, all of that stuff's working in our data center. Great. And it doesn't work for our aggregator business and that's probably 60, 70% of our business now. So actually we've invested a huge amount of money and technology into essentially giving a good price for 30% of our business. The other 70%, who knows? We're still taking a gamble and we're still working back on the post codes. So what we're going to do, well we can take that great enterprise GIS platform that we built in-house and let's just scale it out. It gets a little bit tricky. The problem is what works well on a couple of servers when you're scaling out and you're looking at having to deliver a sub-second performance, you're looking to be able to process thousands of transactions per second instead of 20, 30 transactions a second, the thing doesn't scale anymore. And if it does scale, someone's going to be unhappy. Basically the CFO is not going to be very pleased. So really what do we do? I mean, where do we go from here? Okay, we don't have a scalable solution. We've got all of this very expensive proprietary technology sitting in it. Where do we go? What we can go to the open source community? They've got some fantastic offerings that have come out of probably some of the people in this room. Absolutely and definitely the way to go. But actually at this time we need to maybe take a step back and think about what was the business requirement? What will we try and to solve? Do we need to have all of these layers in our architecture? And actually that's just really, if we think about the requirement, we just want to know the risk of bad stuff happening at a location. Do we need a map? Not really, no, not for this. We don't have the luxury of time for a map. We just need somewhere that we can provide an assessment, a building level of various different risk layers. And we want to do that as quickly as possible for as many people as possible. So actually we could probably just do something like this. So instead of having this big platform, actually all we probably need is an API that speaks and sends out GeoJSON or JSON if we're not using a map. That could just be a simple little Python script or a little job calling PostGIS directly. And do you know what? We don't need to have a GIS application server. We don't need all of these other layers. If we took it down to the one function it needs to perform, which is give me an assessment of these layers at this location, that's all we need. And by breaking it down to something small like that, actually now in terms of scaling that, you know, life becomes a little bit easier. So we don't have to even necessarily implement it in our... So if we were using.NET we could go and use Python or we could use Node.js. We could carry on using SQL Server if we wanted to or we could go and look at an open source software in such as Spatial Light or PostGIS. And suddenly our unit of scale has reduced and now to be able to scale up something like this is a lot easier. But we think we could probably go a step further, actually. So even like this we're still being able to perform. Even if you strip away the onion and get rid of all of the complexity in the mid tier, you're still having to hit the database to querying, raster layers, vector layers, having to do quite a few lookups, get the response back and get that back crucially has to be sub-second. So actually we kind of scratched our heads and thought, well, do we need to be doing all of this spatial stuff every time we do a lookup? Well, sometimes yes, but is there a way that we could pre-calculate the answers? If we knew the answers in advance, we wouldn't have to calculate them for the customer. So we kind of scratched our heads and had a bit of a think about it and then we had a bit of a eureka moment. We know where all the addresses are because we buy the data. So it's a national address dataset. We know the location of 37 million addresses in the UK and we know exactly where they all are. And our customers are probably only going to be looking to get an answer at one of those 37 million locations. And the other thing as well is actually at the moment, all we're doing, I mentioned at the beginning that post codes don't flood, buildings flood. Well, at the moment, what we've gone from is post codes to points. So at the moment, if we have a flood layer that cuts through the middle of a building and the entrance is over here, but the layer is over here and the majority of the flood is happening the rest of the building, actually we're giving an incorrect answer. But because of the time constraints and having to do the GIS at a point of query, we don't have the luxury of doing more complicated analysis. But if we go with this approach, we could potentially pre-cash them and answer all those questions up front. So what do we do? We've got a unique, in the UK now, we've just open sourced, a year and a half ago, they just open sourced a unique property reference number. So every single building in the UK, every single address has a unique reference identifier. It's open source and it's available. But it's going to take forever to analyze all of that data. How are we going to do that? That's going to, also every time the data changes, it's going to take a long time. Not at all now, so we can split the problem up into chunks, we could use a cluster, if we wanted to, we could use a cloud-based cluster and essentially we could answer that question, what is the risk for all of those buildings? And we could probably get that down to hours rather than days or weeks. But the question comes up, but we don't always ensure buildings. So part of insurance is I'm a property developer, I've just bought this piece of land over there. I want to ensure it, so that obviously is not going to have a unique identifier and we're going to have to use the geographic approach. That's no problem at all, so let's keep the old solution for the 1%. So we end up with something that potentially could look like this. So we've got our address, we want to describe an address. For 99% of the cases, we can just do it on the identifier, but for the small percentage of the cases where we can't do that, we go ahead and do it on the spatial database. No problem at all. And kind of what we've got here inadvertently, without wanting to bring out our buzzword, is we found ourselves here now with a microservices architecture. We didn't go around doing it, we didn't look on the line and see this fantastic thing and jump on the bandwagon and say, oh, we need to do microservices. We've just kind of fallen into it because we are just by simplifying and solving one problem. And what we've got now is we've got something actually on the left here, because there's no spatial in here at all, we could use pretty much any technology that we want to. We still got our spatial over here on the right that's powering the spatial queries, but on the left there, we just really just got a key value store. So we can scale out, we can cluster, and suddenly we're able to meet, not sub-second, but here we're probably talking sub-100 milliseconds, sub-50 milliseconds. We've got something that's super, super, super fast. So actually, another advantage of this is when our customers go online and get their quotation and bang, bang, bang, actually the guys who respond the quickest will normally end up at the top. So not only of that, we end up coming further up the list. And the other hidden benefit as well for this, I don't know, has anyone heard of or experimented with serverless architectures? Quite a new thing. Yeah. Not one for today, but maybe one for another talk. But this opens the door for a serverless architecture. So suddenly now we have a simple function that does one thing. It calls a key value store and returns some data. Over on the left there for people who aren't AWS customers, I'm a big fan of AWS, Amazon. We have a Lambda service and we have a DynamoDB store. And that thing can sit there, and if it's not running, it doesn't cost us any money. It only costs us money when we use it. So suddenly we have not only have we eliminated complexity, we've got a faster service, we can pretty much run it for next to nothing. So the message really is that you're free to use the right tools for the job. If you're interested a bit more in these architectures, there's a couple of links there to some videos and YouTube clips. And we're happy. We're happy developers. We've got happy business users. Obviously, this isn't silver bullets. It's not going to help you with all of your really complex geospatial problems. If you wanted to take your desktop architecture and be able to publish services very easy, then great, go and use a GIS platform. But just when you go back and you're looking at this, find yourself questioning, do I really need all of these layers? Do I need a full function in GIS platform? Do I need to be able to support every single standard, or do I just need something that does one thing really well? So to summarize, I would say that spatial is not always really special. You should optimize for your requirements. Beware the platform and approach the platform with caution and really question the use of the platform. Cash it if you can. So if you're able to answer the question in advance, don't incur all that overhead at query time. But above all, just keep it simple, and it will scale. And that's my message for the day. Okay, so I think we have time for a few questions. I'm sure they're going to be some. No, so all we I'll just flick back to the sliding question. So okay, I've lost the slides, but that's fine. Maybe the data that we receive, I don't know whereabouts you come from? From DC, okay. So the question was going back to the slide where I showed the picture of the UK in chunks and how we distributed the processing. Well the way the data is actually delivered, so the data we typically use comes from the Ordnance Survey in the UK, and it actually is delivered in geographic chunks. So that grid layout, that's the way the data is actually delivered. So it's kind of already ready. So it's already that positioning is already done for us. So we just take that grid, apply the same to the addresses, so we'll take just all the addresses from within that grid. We would snapshot out, for example, our flood model just for that grid, and have that queued up and running on a machine. And we can run as many of those in parallel as we've got the money for. Okay, no problem. Yeah, one question on the back. Not very well, no, sorry. Well I'm taking, so this is from my perspective now, so essentially I'm running a, this is a SaaS company now. So the business already looking to, or business already looking to offload any kind of use cases that are not secure to SaaS providers. And this one is kind of perfect. There's nothing financial here, there's nothing that would really worry a big insurance company. Essentially this is just around providing geographic assessments. So I'm lucky that I didn't have to have the conversation. So yeah, this is my take on it as a SaaS provider who are free to use whatever technology I want to. And I'm seeing, I've certainly seen clients that I've worked with in the past before I was running my own company who are totally embracing the cloud, totally embracing microservices. So yeah, I think this is, I think it's the future, I think it's the shape of things to come. Hi. The more personal question. Yeah. Yes. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Just the barrier to end, so what I found, I can tell you, I remember probably one of the first conversations that I had as an IT person that having a conversation with a GIS person was our interpretation of what was a map. You might have had similar things. So we were probably sitting in a meeting for maybe an hour and I'm talking to this guy and we've been talking about this map and for me, the map was the sum of the parts. For me, a map is what you see on Google at the time or if I had my Ordnance Survey map at home. And for him, he was thinking of the map, well, were you talking about the base map or the layers or the, I was like, what's a base map? Just that kind of barrier to end, I found it really, really hard. And I think I probably made decisions and gave advice earlier on when I started off looking at this, just thinking as an outsider, well, you must have to have a GIS platform. I mean, that's absolutely, how can you do a GIS without a GIS platform? So yeah, I think it takes a while to actually break it down and simplify and actually what we're doing necessarily is not necessarily that complicated. Does that answer the question? Yeah. One question. So from your perspective, with your experience, what would you think would be the biggest challenge if you want to transform like a monolithic application, let's say, into microservices? So I mean, you remove the complexity out of the small blocks, but then you have like the orchestration of the... Yeah, absolutely. It's definitely not a silver bullet and you need to approach it with caution. I haven't had to really take something massive and break it up. From what I've read and what I've understood is that the hardest thing is how you partition. And what do you do? So for a microservices architecture, really the kind of guiding principle is that you should never share data. You should never share databases. So you would have each function or each piece of the application would look at its own data store. So that's quite a good way of segregating it. But if you make a bad decision earlier on, especially if you're just getting into this, if you make a bad decision, you could end up adding huge layers of additional complexity. So I think that would be a bit of a challenge. There's a bunch of approaches. If you do have an old monolithic application that you want to kind of break up, it's quite good to think about the services, build a service layer on top of it first, get that to a level where you're happy with it, even if it's just calling the monolith. And then after that, then that's the point where you can start partitioning off. So does that make sense? Is that answer a question? Yeah, that's a very good advice. Okay. Thank you. So any more questions? Okay, so let's thank the speaker and also the other speakers. And I think this will be the last session, the last talk from this session. Thank you.
into IT and I think there are IT people who've it's just an example of something that I've a parallel, there's a few parallels that can be it's not special, it is special, but it's not prices for our customers. But the problem was the with a very large purveyor of GIS solutions who took it down to the one function it needs to sometimes yes, but is there a way that we could spatial database. No problem at all. And kind of ourselves here now with a microservices is that you're free to use the right tools for picture of the UK in chunks and how we use comes from the Ordnance Survey in the UK, and insurance company. Essentially this is just microservices architecture, really the kind of