Delivering a digital register of common land in Wales with the help of Open Source Geo
Marlow, Peter
Ratcliffe, Kathryn
The project showcases the benefits of enhancing commercial technologies and products with those available in the open source geo community. The Electronic Register of Common Land and Towns and Village Greens in Wales enables the Public to access register pages and maps freely online and local authorities will be able to, for the first time, update the registers and maps digitally. This saves time and cost for both the Public and LA’s and provides a unified register of Common Land. The application harnesses a range of open source technologies, including GeoServer, OpenLayers, .NET Core and Linux containers, working in tandem with more commercial products, such as SQL Server to provide a performant, flexible and secure application. All of which is hosted using Microsoft’s Azure cloud platform. The project helps showcase the possibilities and benefits of enhancing what would traditionally be an application comprised of commercial, closed source technologies and products, with those available in the open source geo community.
 Hello, I'm Peter Marlowe, I work for CGI, I've been the technical lead on this project that's been running for just over a year now. I'm joined me as well as Catherine Ratcliffe, who's the project manager, so she's also from CGI as well. We've been helping create a digital register of common land in Wales for the Welsh Government. Quite a bit to get through, so let's crack on. So the first bit is what is common land. So common land is land subject to rights enjoyed by one or more persons to take or use part of a piece of land or of the produce of a piece of land which is owned by someone else, so straight forward. What I want to touch on here is just a few of the kind of references and key terms that you're going to hear me use in this presentation that's related to common land and try and explain those. So you know what I'm talking about when I come to mention them. So the first one is register unit. So register unit is the official term for a piece of common land, so within Wales there's lots of common land and the term to refer to an individual piece of that is a register unit and that usually has a reference as well, so for example CL120 that would be the unique reference for a piece of common land within Wales. Second point here is dominant tenement, so not to be mistaken for the famous actor who played Doctor Who and so dominant tenements or usually refer to them just as DTs. So these are the people that own the rights, so they own the specific rights over common land and they typically tend to be farmers. And then lastly we've got the rights of common, so these are these detail the specifics of the rights, so for example grazing 10 sheep. So kind of in conclusion, you have dominant tenements who have one or more rights of common and those rights of common apply to a register unit which is a piece of common land. So in terms of the background around the project itself, so obviously the register of common land exists at the moment in its paper format. Each local authority in Wales has its own version of their register that they maintain and that's open to the public at the moment but obviously in order for people to come and see it they have to book in with their local authority a range of time, physically come to the offices and open up the register and find the bit that they want to take a look at and so ties in a little bit with point two there which is time consuming, so obviously that isn't a quick process, you know in this day and age people expect to see you know information after a couple of clicks on the internet. So part of this making it digital will allow the public for the first time to be able to access all the information to do with common land in Wales, you know online, you know no restrictions they don't need to go to their local authority and request time, they can see it all online which is fantastic and again on the second point there for the time consuming where there's disputes in common land where there's rights apply and the details of those rights but often they end up resulting in kind of legal cases and that takes a long time to resolve especially having to go back to look at the physical registers and trying to determine who changed them and what change they made and that feeds into the third point there which is the register maintenance and so by making it digital it's not just allowing the public to see that information but actually all the local authorities will be able to maintain that register digitally going forward and that will be the official legal register and as part of that this system is recording all of the changes that are made to this full audit history so we're recording who made the change, why they made the change and what they actually changed and that's really important in these legal cases to determine what action should be taken. So the fourth point on there which is about the physical state so as I mentioned they are paper based at the moment, they're quite old and so you know they're starting to deteriorate and we'll have a look at a minute that's come to the state of the maps and the kind of damage that's happened to them obviously by digitising it we can make it last forever essentially, we don't have those issues of deteriorating over time and the last bit on there is the data sharing so it's really critical that when we digitise this data we can make it available to other departments within Welsh Government or other organisations outside of Welsh Government and obviously at the moment that's almost impossible to do because each local authority has their own register, you know it's paper based if you wanted to get a Wales century view of common land in Wales, it's really trickle to do at the moment because you have to call off to lots of different local authorities. So just a quick look at some of the numbers here, so there's 22 local authorities in Wales as I mentioned each one's got their own register, most of these registers are kind of more than 50 years old so they're old documents in terms of the quantity of maps, you know we're looking at about 12,000 map pages, 25,000 register pages around 70,000 land and right entry pages so that's a lot but I think the point I want to make here is it isn't really a big data project we're not talking about millions of records that we need to process here but the challenge is more in the quality of the data that we're trying to transform so on this next slide here hopefully you can see that in some detail so this is some example of register maps as they are now in the paper based register and you can kind of see hopefully from this that some of the quality is not fantastic and obviously these maps have been drawn on, hand drawn to trying to note the different boundaries and where edits have been made sometimes they're kind of on top of each other, pretty difficult to determine where the boundary starts and where the boundary ends and the same is true of the register pages so here we've got a couple of examples so this is the textual content of the register so the rights we can see here that they are handwritten if they get voided then obviously they just get crossed out and in a lot of cases normally in pencil where they've done an update kind of in line is quite tricky to decipher the kind of changes that have been made here and what's the kind of real legal version of it and the hope is that by digitizing this we can standardize that so there's just one way to create a right one way to record common land in Wales. So without further ado let's jump into the application so I'll come out of here. So hopefully you can see this on my screen okay so what we've got here is the application is in two parts so this part I'm going to show you first is search the register there's also manage the register so this is the screen that the public will be able to see so anybody can come on and start using this service and we can see here there's a couple of options that we've got to find this common land data because obviously we're phosphogy and we all have maps we're going to use the by location search so I'm just going to search for my favorite village in Wales or favorite after starting this project and what we can see here is that's brought me up brought up a map for me and I sent it on that location in Wales and we can start to see some of the register units that are in green and the dominant tenements that are in red and we've got some of the textual information around those on the side here so I'm just going to go into full screen because bigger maps are always better and at this point obviously in Wales and I can just kind of zoom around and I'm free to go and look around and look at common land and dominant tenements kind of anywhere in Wales at this point I'm not restricted by that initial search and this is test data so this is on our test system but a lot of it's indicative of real data so I'm going to come back here now and just come back to CL120 and if I hover over this I get this pop up and it allows me to drill in and see more details for this register unit so if I select that I'm now on the second map page now this is a page that's kind of centered on this specific register unit so we can still see that register unit in the bottom right here and now we can see a couple of dominant tenements and the important thing about this page is it's focused on this register unit and just showing who has rights over this register unit so it's only showing the dominant tenements that have rights so if I scroll down a little bit you can see some of the tabs here and you can see there an example right of common that exists so this is in the kind of formatted standardized version of this so we can see they've got the right to graze as the type and we've got the subjects that they can graze. We've also got just as a temporary measure here we can see the original text as it was in the register before it was transformed so these are the original five columns from the register with their original data so I think the bit that's interesting here is column four which contains the details of the rights so we can see here that's basically just a big kind of block of text you know paragraph of text is difficult to query that it's difficult to break it down and do a lot with that whereas in the new format you know it's split out we spit out the type of right which is graze and we spit out the numbers and the subjects of that right which means you know going forward it's much easier to query that and get an idea of you know how many grades and rights are there in Wales which you know you just couldn't do in the paper form or even just taking the paper form in its current format and putting it digitally. So if I come back up to this map here I'm not constrained again I can start zooming around as well here if I want to get it slightly closer in and we can see some of these purple polygons boundaries so those are deregistered areas so they were bits of commonland and for one reason or another they're no longer commonland and the last bit I want to share you on this public side is just this notion of what we've kind of termed as hopping so at the moment obviously I'm centred on this register unit CL120 but if I want to change my context and say actually for example this DT30 where else does that DT have rights which other bits of commonland does it have rights and I can actually select that from here and the context is updated the maps updated not sure how good the video quality is there in terms of the frames per second but it's a smooth transition that's brought that map out there and it's now showing me that actually this dominant tenement has rights also over CL30 that we can see up in the corner here and the actual list of rights has been updated to be specific to that dominant tenement and that I'm not stopped there I can carry on doing this kind of hopping so I can say okay who's got rights over CL13 and I can select that as well and we can see in this case quite a lot of people have rights over CL13 so it's just a really good way of you know initially placing yourself somewhere looking at a certain bit of commonland and I've just been able to kind of continue your investigation continue your search into it and you know keep jumping and looking at rights and looking at different bits of commonland. Okay so I'm just going to quickly go into the admin side of the portal so this is the local authorities that would sign in to make changes to the register. I'm just going to sign in with my government gateway account and this is a test government gateway service so there's no need to try and write down my access code it's not a real access code and it's definitely not my pin number so now I'm in we can see now that the screen is talking about managing the register and what I can do from here is I can select update the register so there's quite a few steps here different paths through I'm going to kind of skip most of these because they're not really relevant to what we want to show here and get to the bit where we can see some maps so I'm going to select the register unit that we saw previously and I'm going to select to create a new rights entry so I'm going to keep these details the same as default and I'm going to start seeing this in here so now that we've kind of normalized it standardize this data it's meant that we can select stuff for drop down to just much easier to obviously create rights with so I'm just going to say that they can graze 10 cattle you know I could keep adding rights of common there as well I'll just add that one for now so now I need to put in the rights description so this is a description of the dominant tenement so who are the rights attached to and what's important here is I have to put it in English and in Welsh so obviously the system itself labels the screen it's obviously available in Welsh but the actual register content as well will also be available in Welsh and that's a big plus so the register as it is at the moment it's paper format it's only available in English but the digital version will be available in both English and Welsh so that's a massive bonus for Welsh speakers so I'm just going to put in the dominant tenement name in this case the name of a farm and I'm going to select to digitize this dominant tenement so here we've got a map again and we can see my register unit if I zoom in a bit we can see we're getting master map data back as well and I've got these extra controls on the side here that are going to allow me to digitize this dominant tenement so the first thing I want to do is just turn on this epoch 2 data so the dominant tenements a lot of the time their descriptions are just and field numbers and obviously in modern versions of master map data for modern survey those field numbers aren't included anymore and so what we've done to aid the digitization processes we've processed the epoch 2 data and we've loaded that into into geoserver and we're serving that up basically to allow them to overlay that over the up-to-date master map data and to allow them to capture dominant tenements based on the field numbers so I'm going to select to create here I'm just going to tweak transparency a little bit and I was going to select to snap so this is snapping to the the underlying master map coming from modern survey so not obviously to the epoch data because that's just raster and the master map data here is actually a it's a vector feed WFS coming directly from via Europa which is really nice so we don't have to host that or manage that it just you know comes to us and that's because the Welsh government got a public sector mapping agreement that allows them to have access to that which is very nice I want to quickly show you these other tools that we've got here so we've got this reduced polygon and combined and basically what these let us do in a lot of cases it's necessary to kind of cut holes out of the geometry and if they're bits that aren't common land you know they've installed an electricity pile on or or such and there's cases as well where they would need to kind of do the opposite and fill them back in so they've got the tools to be able to do that and lastly I just wanted to show very quickly if I kind of edit this now and move that boundary so it's over the register unit which is the green boundary I get this warning come up telling me that you know the dominant tenement boundary intersects a registered area and so what we do is we put kind of validation rules in and there's lots of these for the other layers as well it kind of help the user guide them to be like okay the update you're making might not be strictly correct now in this case it's only a warning because the data that we're transforming into the digital register some of that data is already going to have these inconsistencies so we don't want to be too strict because the person making that update they might not be able to you know make a decision as to which is the right way to go but we've got a raft of those kind of updates those validation rules to help them you got about five minutes Peter okay thanks and so that's showing me digitizing the dominant tenement so that's fine I've done that I come back to here there's lots of extra information I could put in here but it's not really relevant to what we want to show today so I'm just going to submit that so that's gone in I can view that update and as we can see I'm back to this page again and hopefully you know you can see that DT that I created is there and the right that I put in is also there as well cool okay I'm just going to jump back into the presentation quickly so in terms of what's what the benefits that open source geo has given to this project that we've put together so obviously we've mentioned before we're using geo server that's super fast WMS so one of the things that love that geo server is the performance of that so we've kind of seen that on the big map there bring about those registered units bring about the dominant tenements you know really performant the OTC standards that has out of the box so we touched on it earlier as well it's critical that although this register of common land it's in charge of managing that data within Wales but it needs to be able to share that data with other systems and other departments and hopefully you caught the data map Wales presentation from my colleague Pascal earlier today and that's a great example of a system in Welsh government that will be able to harness this data and display within their system as well so though they won't be the owner of it they won't be able to update it but in a read-only capacity they'll be able to get you know common land geometry and display it in their portal which is really critical and the last bit a little bit more technical but because it's open source you can kind of get under the hood of it and diagnose any issues you can get into the code you can debug it so we didn't have any issues on our project which is great but just knowing that that option is there is a great place to be so the second part is open layers that we're using so obviously that's a very powerful comprehensive web mapping library and in our case you know we could tailor that to the specific application needs so some of those geometry validations some of the geometry alterations that we were making and the kind of zooming in and out changing the context you know open layers allowed us to do all of that you know pretty seamlessly and the last bit there we're using that in combination with JSTS which is the library that allows us to perform those fairly complex geometry operations you know cutting holes out and we could have used turf.js as well you know those two are pretty interchangeable. So just want to highlight one of the challenges we had because you know we always have challenges and we were using obviously we were using SQL Server as the database management system that was the preferred option from Welsh Government mainly down to security but obviously that isn't often the first choice of database when you're working with Spatial usually it's Postgres, PostJS so that creates some issues just because it's more difficult to find to help articles and documentation on that and we did have some bugs when it's put some work around in you know just because working with Spatial SQL Server is just a little bit less mature than some of the other alternatives but in the end the effort was justified obviously we have it working now and what that's really given us is the ability to use SQL Server as your SQL in particular which you know if you host your application in Azure you know SQL Server has got the best support, the best level of functionality, the different services to do with scaling, you know security, availability all of that stuff you know SQL Server within Azure is the best at that so it's given the application access to that which is which is fantastic. Okay the last bit one on just top Sean so we are deploying this application into Azure you'll probably do a half hour talk just on that but I just wanted to pick up two points about how we are doing that that is maybe slightly different to how other people are so the first bit is how we are building this so the application is really in two parts so it's the portal application so that's written in.NET Core and then obviously we've got Geo Server that's serving up the GIS information that's obviously in Java you know two different languages, two pretty different applications but in this case they're both built with Docker and they're both deployed in containers using Ubuntu and that gives us a bit of parity essentially in that build process even though they're both very different and then how do we actually deploy that up into the cloud so we're using within Azure there's an app service and that's their kind of PAS platform as a service and within there there's the option to use web app for containers which is what we're using and that basically allows you to take those containers and deploy them up into Azure as web apps and that is obviously managed service from Microsoft from Azure and that helps with things like availability scaling and security particularly on the SSL side where I don't know if anybody's tried to do SSL and Geo Server before sometimes that isn't straightforward can be a bit of a nightmare so anything that helps with that you know is really good. Okay so that was it really I know that was a little bit of a whistle stop tour going through it but I'm hoping I covered you know most of the stuff and certainly how the open source Geo elements have really enhanced the solution. Thanks very much Peter that's great yeah we've got a few questions that have come in so I'm going to go straight on to those and run through those we've got eight to nine minutes I guess. So first of all from Victor about base mapping saying asking what base map did you use and was can open street map attribute data be included in the database is there any element of crowd sourcing in this project? So yeah good question so there's no element of crowd sourcing so in terms of the base map that we're using so it's the standard audit survey master map and the base map as well so that all comes from this far Europa service that we have access to so that provides us with a base map but also provides us with the vector geometry for master map which is what we start to on the admins range. Cool thanks and the next one is from David says hi Pete fantastic project and presentation the question is have you used these technologies in other projects and if so what are the main issues you've faced with the technologies I know you mentioned some issues with SQL server for example. Yeah so that was certainly one of the bigger ones and just because a lot of that was unfamiliarity so yeah a lot of our other projects that we've run is usually post you know Postgres or PostGIS if you're working in the spatial domain so SQL server did provide a few issues there probably the other big change for this project is it's all it's all in Azure and it was using web apps so that's slightly different perhaps different to what's normally done and especially in terms of Geo server so taking Geo server and hosting that in a web app there was some stuff that we had to do configuration wise to get that and to work if you've got a little bit less control when you deploy it into a web app so although obviously Microsoft takes care of some of that scaling and you know availability but in terms of getting getting behind the scenes looking at what Geo servers doing there's a little bit more configuration you have to do there and so I think those are probably the two main points especially on the SQL server side the geometry so in native SQL we take that and we obviously transform it back into Geo JSON WKT so we can work with it in the browser with in open layers and that kind of transformation back and forwards there was a couple of that there's a couple of cultures there that we had to be careful of. Cool the next one is a comment really from Dave who says he had no idea that common land wasn't digitized in Wales so it's back in the day in 2004 in natural England so I'm not sure I've lost the end of my natural map to all common land in England so it's getting one up for England there I think. Yes so I mean the thing to mention there as well is that as far as we're aware certainly in the UK maybe even worldwide certainly in terms of taking common land and building a digital register Wales seemed to be the first in the UK at least to do this so we know in England obviously they don't have a digital register and the same with Scotland so they might have the common land geometries digitized but this is kind of a full service that allows them to completely manage their registers and basically get rid of the paper-based ones put them in the National Archive but they don't need to rely on those it's the legal thing anymore it's all totally digital so I think Wales are leading the way. So we've got one up for Wales in fact. And the next one so Tony asks aren't vector tiles generally simplified and so how does the SNAP2 vector data work with respect to that? So yes so at the moment the way that's working is the actual the raster imagery that comes back that still comes back as raster we still get that master map back as raster and what we're doing is we're loading WFS service for the vector data and we're kind of loading that in but we just don't show it so it kind of sits underneath underneath the raster and then with an open layers you can you can basically give it that kind of vector vector layer and say you know SNAP2 this and so obviously in this case because both of those sets of data are coming from the same provider they kept in sync so as far as their users concerned there's no you know there's no difference there the raster always represents the it's always in line with the vector but what it meant was we didn't need to we didn't need to manage that so we wanted to avoid having to take the master map data and you know put it into Geo server and do all the styling and that kind of stuff which is quite a lot. So it's not simplified at all the vector data? No. No okay cool. A question from Akin has Wales got a digitized brownfield map do you know? I do not no sorry. Okay somebody else might know. Okay a couple more questions they keep coming in this one from Ulrich not really a Geo question but I'm going to ask it anyway how was the textual content captured and how was the transition to the legal acceptance of the digital data from the paper maps managed? Do you want to answer that one? Yeah so essentially each page and each map was scanned in so we've got some scanners going around Wales literally scanning in the pages and then they use OCR so Optical Character Recognition to take the data they can and handwritten data is actually manually typed so a very very long long process which we're still undertaking to be honest and then basically they're linked to the maps with sort of you know unique keys which are always put into the database so that's done. That'll be going on for another year. So is it legally accepted now that the digital data is? So not yet it will go through the local authorities consultation which will happen over the next 12 months and then it will go to public consultation which is spring 2022 and once that's happened then it will become the legal register. Okay cool so I've just got one more comment here which is batting back for England so Sarah says rural payments agency England maintains the digital record of the common land register for the purposes of inspections which can be accessed by the public on request. So who knew? Just don't tell Wales. They're very proud of thinking they've beaten England.
 Hello, I'm Peter Marlowe, I work for CGI, I've been the technical lead on this project  that's been running for just over a year now.  I'm joined me as well as Catherine Ratcliffe, who's the project manager, so she's also  from CGI as well.  We've been helping create a digital register of common land in Wales for the Welsh Government.  Quite a bit to get through, so let's crack on.  So the first bit is what is common land.  So common land is land subject to rights enjoyed by one or more persons to take or use part  of a piece of land or of the produce of a piece of land which is owned by someone else,  so straight forward.  What I want to touch on here is just a few of the kind of references and key terms that  you're going to hear me use in this presentation that's related to common land and try and  explain those.  So you know what I'm talking about when I come to mention them.  So the first one is register unit.  So register unit is the official term for a piece of common land, so within Wales there's  lots of common land and the term to refer to an individual piece of that is a register  unit and that usually has a reference as well, so for example CL120 that would be the unique  reference for a piece of common land within Wales.  Second point here is dominant tenement, so not to be mistaken for the famous actor who  played Doctor Who and so dominant tenements or usually refer to them just as DTs.  So these are the people that own the rights, so they own the specific rights over common  land and they typically tend to be farmers.  And then lastly we've got the rights of common, so these are these detail the specifics of  the rights, so for example grazing 10 sheep.  So kind of in conclusion, you have dominant tenements who have one or more rights of common  and those rights of common apply to a register unit which is a piece of common land.  So in terms of the background around the project itself, so obviously the register  of common land exists at the moment in its paper format.  Each local authority in Wales has its own version of their register that they maintain  and that's open to the public at the moment but obviously in order for people to come  and see it they have to book in with their local authority a range of time, physically  come to the offices and open up the register and find the bit that they want to take a look  at and so ties in a little bit with point two there which is time consuming, so obviously  that isn't a quick process, you know in this day and age people expect to see you know  information after a couple of clicks on the internet.  So part of this making it digital will allow the public for the first time to be able to  access all the information to do with common land in Wales, you know online, you know no  restrictions they don't need to go to their local authority and request time, they can  see it all online which is fantastic and again on the second point there for the time consuming  where there's disputes in common land where there's rights apply and the details of those  rights but often they end up resulting in kind of legal cases and that takes a long  time to resolve especially having to go back to look at the physical registers and trying  to determine who changed them and what change they made and that feeds into the third point  there which is the register maintenance and so by making it digital it's not just allowing  the public to see that information but actually all the local authorities will be able to  maintain that register digitally going forward and that will be the official legal register  and as part of that this system is recording all of the changes that are made to this full  audit history so we're recording who made the change, why they made the change and what  they actually changed and that's really important in these legal cases to determine what action  should be taken.  So the fourth point on there which is about the physical state so as I mentioned they  are paper based at the moment, they're quite old and so you know they're starting to deteriorate  and we'll have a look at a minute that's come to the state of the maps and the kind of damage  that's happened to them obviously by digitising it we can make it last forever essentially,  we don't have those issues of deteriorating over time and the last bit on there is the  data sharing so it's really critical that when we digitise this data we can make it  available to other departments within Welsh Government or other organisations outside  of Welsh Government and obviously at the moment that's almost impossible to do because each  local authority has their own register, you know it's paper based if you wanted to get  a Wales century view of common land in Wales, it's really trickle to do at the moment because  you have to call off to lots of different local authorities.  So just a quick look at some of the numbers here, so there's 22 local authorities in Wales  as I mentioned each one's got their own register, most of these registers are kind of more than  50 years old so they're old documents in terms of the quantity of maps, you know we're looking  at about 12,000 map pages, 25,000 register pages around 70,000 land and right entry pages  so that's a lot but I think the point I want to make here is it isn't really a big data  project we're not talking about millions of records that we need to process here but  the challenge is more in the quality of the data that we're trying to transform so on  this next slide here hopefully you can see that in some detail so this is some example  of register maps as they are now in the paper based register and you can kind of see hopefully  from this that some of the quality is not fantastic and obviously these maps have been  drawn on, hand drawn to trying to note the different boundaries and where edits have  been made sometimes they're kind of on top of each other, pretty difficult to determine  where the boundary starts and where the boundary ends and the same is true of the register  pages so here we've got a couple of examples so this is the textual content of the register  so the rights we can see here that they are handwritten if they get voided then obviously  they just get crossed out and in a lot of cases normally in pencil where they've done  an update kind of in line is quite tricky to decipher the kind of changes that have  been made here and what's the kind of real legal version of it and the hope is that by  digitizing this we can standardize that so there's just one way to create a right one  way to record common land in Wales.  So without further ado let's jump into the application so I'll come out of here.  So hopefully you can see this on my screen okay so what we've got here is the application  is in two parts so this part I'm going to show you first is search the register there's  also manage the register so this is the screen that the public will be able to see so anybody  can come on and start using this service and we can see here there's a couple of options  that we've got to find this common land data because obviously we're phosphogy and we all  have maps we're going to use the by location search so I'm just going to search for my  favorite village in Wales or favorite after starting this project and what we can see  here is that's brought me up brought up a map for me and I sent it on that location in Wales  and we can start to see some of the register units that are in green and the dominant  tenements that are in red and we've got some of the textual information around those on  the side here so I'm just going to go into full screen because bigger maps are always  better and at this point obviously in Wales and I can just kind of zoom around and I'm  free to go and look around and look at common land and dominant tenements kind of anywhere  in Wales at this point I'm not restricted by that initial search and this is test data  so this is on our test system but a lot of it's indicative of real data so I'm going  to come back here now and just come back to CL120 and if I hover over this I get this  pop up and it allows me to drill in and see more details for this register unit so if  I select that I'm now on the second map page now this is a page that's kind of centered  on this specific register unit so we can still see that register unit in the bottom right  here and now we can see a couple of dominant tenements and the important thing about this  page is it's focused on this register unit and just showing who has rights over this  register unit so it's only showing the dominant tenements that have rights so if I scroll down  a little bit you can see some of the tabs here and you can see there an example right  of common that exists so this is in the kind of formatted standardized version of this  so we can see they've got the right to graze as the type and we've got the subjects that  they can graze. We've also got just as a temporary measure here we can see the original text  as it was in the register before it was transformed so these are the original five columns from  the register with their original data so I think the bit that's interesting here is column  four which contains the details of the rights so we can see here that's basically just a  big kind of block of text you know paragraph of text is difficult to query that it's difficult  to break it down and do a lot with that whereas in the new format you know it's split out  we spit out the type of right which is graze and we spit out the numbers and the subjects  of that right which means you know going forward it's much easier to query that and get an idea  of you know how many grades and rights are there in Wales which you know you just couldn't  do in the paper form or even just taking the paper form in its current format and putting  it digitally. So if I come back up to this map here I'm not constrained again I can start  zooming around as well here if I want to get it slightly closer in and we can see some of  these purple polygons boundaries so those are deregistered areas so they were bits of  commonland and for one reason or another they're no longer commonland and the last bit I want  to share you on this public side is just this notion of what we've kind of termed as hopping  so at the moment obviously I'm centred on this register unit CL120 but if I want to change  my context and say actually for example this DT30 where else does that DT have rights which  other bits of commonland does it have rights and I can actually select that from here and  the context is updated the maps updated not sure how good the video quality is there in  terms of the frames per second but it's a smooth transition that's brought that map  out there and it's now showing me that actually this dominant tenement has rights also over  CL30 that we can see up in the corner here and the actual list of rights has been updated  to be specific to that dominant tenement and that I'm not stopped there I can carry on  doing this kind of hopping so I can say okay who's got rights over CL13 and I can select  that as well and we can see in this case quite a lot of people have rights over CL13 so it's  just a really good way of you know initially placing yourself somewhere looking at a certain  bit of commonland and I've just been able to kind of continue your investigation continue  your search into it and you know keep jumping and looking at rights and looking at different  bits of commonland. Okay so I'm just going to quickly go into the admin side of the portal  so this is the local authorities that would sign in to make changes to the register. I'm  just going to sign in with my government gateway account and this is a test government gateway  service so there's no need to try and write down my access code it's not a real access  code and it's definitely not my pin number so now I'm in we can see now that the screen  is talking about managing the register and what I can do from here is I can select update  the register so there's quite a few steps here different paths through I'm going to  kind of skip most of these because they're not really relevant to what we want to show  here and get to the bit where we can see some maps so I'm going to select the register unit  that we saw previously and I'm going to select to create a new rights entry so I'm going  to keep these details the same as default and I'm going to start seeing this in here  so now that we've kind of normalized it standardize this data it's meant that we can select stuff  for drop down to just much easier to obviously create rights with so I'm just going to say  that they can graze 10 cattle you know I could keep adding rights of common there as well  I'll just add that one for now so now I need to put in the rights description so this is  a description of the dominant tenement so who are the rights attached to and what's  important here is I have to put it in English and in Welsh so obviously the system itself  labels the screen it's obviously available in Welsh but the actual register content as  well will also be available in Welsh and that's a big plus so the register as it is at the  moment it's paper format it's only available in English but the digital version will be  available in both English and Welsh so that's a massive bonus for Welsh speakers so I'm  just going to put in the dominant tenement name in this case the name of a farm and I'm  going to select to digitize this dominant tenement so here we've got a map again and  we can see my register unit if I zoom in a bit we can see we're getting master map data  back as well and I've got these extra controls on the side here that are going to allow me  to digitize this dominant tenement so the first thing I want to do is just turn on this  epoch 2 data so the dominant tenements a lot of the time their descriptions are just and  field numbers and obviously in modern versions of master map data for modern survey those  field numbers aren't included anymore and so what we've done to aid the digitization  processes we've processed the epoch 2 data and we've loaded that into into geoserver  and we're serving that up basically to allow them to overlay that over the up-to-date master  map data and to allow them to capture dominant tenements based on the field numbers so I'm  going to select to create here I'm just going to tweak transparency a little bit and I was  going to select to snap so this is snapping to the the underlying master map coming from  modern survey so not obviously to the epoch data because that's just raster and the master  map data here is actually a it's a vector feed WFS coming directly from via Europa which  is really nice so we don't have to host that or manage that it just you know comes to us  and that's because the Welsh government got a public sector mapping agreement that allows  them to have access to that which is very nice I want to quickly show you these other  tools that we've got here so we've got this reduced polygon and combined and basically  what these let us do in a lot of cases it's necessary to kind of cut holes out of the  geometry and if they're bits that aren't common land you know they've installed an electricity  pile on or or such and there's cases as well where they would need to kind of do the opposite  and fill them back in so they've got the tools to be able to do that and lastly I just wanted  to show very quickly if I kind of edit this now and move that boundary so it's over the  register unit which is the green boundary I get this warning come up telling me that  you know the dominant tenement boundary intersects a registered area and so what we do is we put  kind of validation rules in and there's lots of these for the other layers as well it kind  of help the user guide them to be like okay the update you're making might not be strictly  correct now in this case it's only a warning because the data that we're transforming into  the digital register some of that data is already going to have these inconsistencies  so we don't want to be too strict because the person making that update they might not  be able to you know make a decision as to which is the right way to go but we've got  a raft of those kind of updates those validation rules to help them you got about five minutes  Peter okay thanks and so that's showing me digitizing the dominant tenement so that's  fine I've done that I come back to here there's lots of extra information I could put in here  but it's not really relevant to what we want to show today so I'm just going to submit  that so that's gone in I can view that update and as we can see I'm back to this page again  and hopefully you know you can see that DT that I created is there and the right that  I put in is also there as well cool okay I'm just going to jump back into the presentation  quickly so in terms of what's what the benefits that open source geo has given to this project  that we've put together so obviously we've mentioned before we're using geo server that's  super fast WMS so one of the things that love that geo server is the performance of that  so we've kind of seen that on the big map there bring about those registered units bring  about the dominant tenements you know really performant the OTC standards that has out  of the box so we touched on it earlier as well it's critical that although this register  of common land it's in charge of managing that data within Wales but it needs to be  able to share that data with other systems and other departments and hopefully you caught  the data map Wales presentation from my colleague Pascal earlier today and that's a great example  of a system in Welsh government that will be able to harness this data and display within  their system as well so though they won't be the owner of it they won't be able to update  it but in a read-only capacity they'll be able to get you know common land geometry  and display it in their portal which is really critical and the last bit a little bit more  technical but because it's open source you can kind of get under the hood of it and diagnose  any issues you can get into the code you can debug it so we didn't have any issues on our  project which is great but just knowing that that option is there is a great place to be  so the second part is open layers that we're using so obviously that's a very powerful  comprehensive web mapping library and in our case you know we could tailor that to the  specific application needs so some of those geometry validations some of the geometry  alterations that we were making and the kind of zooming in and out changing the context  you know open layers allowed us to do all of that you know pretty seamlessly and the  last bit there we're using that in combination with JSTS which is the library that allows  us to perform those fairly complex geometry operations you know cutting holes out and  we could have used turf.js as well you know those two are pretty interchangeable.  So just want to highlight one of the challenges we had because you know we always have challenges  and we were using obviously we were using SQL Server as the database management system  that was the preferred option from Welsh Government mainly down to security but obviously  that isn't often the first choice of database when you're working with Spatial usually it's  Postgres, PostJS so that creates some issues just because it's more difficult to find to  help articles and documentation on that and we did have some bugs when it's put some work  around in you know just because working with Spatial SQL Server is just a little bit less  mature than some of the other alternatives but in the end the effort was justified obviously  we have it working now and what that's really given us is the ability to use SQL Server as  your SQL in particular which you know if you host your application in Azure you know SQL  Server has got the best support, the best level of functionality, the different services to  do with scaling, you know security, availability all of that stuff you know SQL Server within  Azure is the best at that so it's given the application access to that which is which  is fantastic. Okay the last bit one on just top Sean so we are deploying this application  into Azure you'll probably do a half hour talk just on that but I just wanted to pick  up two points about how we are doing that that is maybe slightly different to how other  people are so the first bit is how we are building this so the application is really  in two parts so it's the portal application so that's written in.NET Core and then obviously  we've got Geo Server that's serving up the GIS information that's obviously in Java you  know two different languages, two pretty different applications but in this case they're both  built with Docker and they're both deployed in containers using Ubuntu and that gives  us a bit of parity essentially in that build process even though they're both very different  and then how do we actually deploy that up into the cloud so we're using within Azure  there's an app service and that's their kind of PAS platform as a service and within there  there's the option to use web app for containers which is what we're using and that basically  allows you to take those containers and deploy them up into Azure as web apps and that is  obviously managed service from Microsoft from Azure and that helps with things like availability  scaling and security particularly on the SSL side where I don't know if anybody's tried  to do SSL and Geo Server before sometimes that isn't straightforward can be a bit of  a nightmare so anything that helps with that you know is really good.  Okay so that was it really I know that was a little bit of a whistle stop tour going  through it but I'm hoping I covered you know most of the stuff and certainly how the open  source Geo elements have really enhanced the solution.  Thanks very much Peter that's great yeah we've got a few questions that have come in  so I'm going to go straight on to those and run through those we've got eight to nine  minutes I guess.  So first of all from Victor about base mapping saying asking what base map did you use and  was can open street map attribute data be included in the database is there any element  of crowd sourcing in this project?  So yeah good question so there's no element of crowd sourcing so in terms of the base  map that we're using so it's the standard audit survey master map and the base map as  well so that all comes from this far Europa service that we have access to so that provides  us with a base map but also provides us with the vector geometry for master map which is  what we start to on the admins range.  Cool thanks and the next one is from David says hi Pete fantastic project and presentation  the question is have you used these technologies in other projects and if so what are the main  issues you've faced with the technologies I know you mentioned some issues with SQL server  for example.  Yeah so that was certainly one of the bigger ones and just because a lot of that was unfamiliarity  so yeah a lot of our other projects that we've run is usually post you know Postgres  or PostGIS if you're working in the spatial domain so SQL server did provide a few issues  there probably the other big change for this project is it's all it's all in Azure and  it was using web apps so that's slightly different perhaps different to what's normally done  and especially in terms of Geo server so taking Geo server and hosting that in a web app there  was some stuff that we had to do configuration wise to get that and to work if you've got  a little bit less control when you deploy it into a web app so although obviously Microsoft  takes care of some of that scaling and you know availability but in terms of getting  getting behind the scenes looking at what Geo servers doing there's a little bit more  configuration you have to do there and so I think those are probably the two main points  especially on the SQL server side the geometry so in native SQL we take that and we obviously  transform it back into Geo JSON WKT so we can work with it in the browser with in open  layers and that kind of transformation back and forwards there was a couple of that there's  a couple of cultures there that we had to be careful of.  Cool the next one is a comment really from Dave who says he had no idea that common land  wasn't digitized in Wales so it's back in the day in 2004 in natural England so I'm  not sure I've lost the end of my natural map to all common land in England so it's getting  one up for England there I think.  Yes so I mean the thing to mention there as well is that as far as we're aware certainly  in the UK maybe even worldwide certainly in terms of taking common land and building  a digital register Wales seemed to be the first in the UK at least to do this so we  know in England obviously they don't have a digital register and the same with Scotland  so they might have the common land geometries digitized but this is kind of a full service  that allows them to completely manage their registers and basically get rid of the paper-based  ones put them in the National Archive but they don't need to rely on those it's the  legal thing anymore it's all totally digital so I think Wales are leading the way.  So we've got one up for Wales in fact.  And the next one so Tony asks aren't vector tiles generally simplified and so how does  the SNAP2 vector data work with respect to that?  So yes so at the moment the way that's working is the actual the raster imagery that comes  back that still comes back as raster we still get that master map back as raster and what  we're doing is we're loading WFS service for the vector data and we're kind of loading  that in but we just don't show it so it kind of sits underneath underneath the raster and  then with an open layers you can you can basically give it that kind of vector vector layer and  say you know SNAP2 this and so obviously in this case because both of those sets of data  are coming from the same provider they kept in sync so as far as their users concerned  there's no you know there's no difference there the raster always represents the it's  always in line with the vector but what it meant was we didn't need to we didn't need  to manage that so we wanted to avoid having to take the master map data and you know put  it into Geo server and do all the styling and that kind of stuff which is quite a lot.  So it's not simplified at all the vector data?  No.  No okay cool.  A question from Akin has Wales got a digitized brownfield map do you know?  I do not no sorry.  Okay somebody else might know.  Okay a couple more questions they keep coming in this one from Ulrich not really a Geo question  but I'm going to ask it anyway how was the textual content captured and how was the transition  to the legal acceptance of the digital data from the paper maps managed?  Do you want to answer that one?  Yeah so essentially each page and each map was scanned in so we've got some scanners  going around Wales literally scanning in the pages and then they use OCR so Optical Character  Recognition to take the data they can and handwritten data is actually manually typed  so a very very long long process which we're still undertaking to be honest and then basically  they're linked to the maps with sort of you know unique keys which are always put into  the database so that's done.  That'll be going on for another year.  So is it legally accepted now that the digital data is?  So not yet it will go through the local authorities consultation which will happen over the next  12 months and then it will go to public consultation which is spring 2022 and once that's happened  then it will become the legal register.  Okay cool so I've just got one more comment here which is batting back for England so  Sarah says rural payments agency England maintains the digital record of the common  land register for the purposes of inspections which can be accessed by the public on request.  So who knew?  Just don't tell Wales.  They're very proud of thinking they've beaten England.
FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_0_107.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_108_887.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_888_3299.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_3300_8699.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_8700_10391.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_10392_11603.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_11604_12155.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_12156_12995.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_12996_13607.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_13608_14099.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_14100_16907.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_16908_17963.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_17964_18443.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_18444_18827.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_18828_18947.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_18948_19091.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_19092_19307.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_19308_19451.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_19452_19895.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_19896_20735.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_20736_20999.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_21000_22871.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_22872_25175.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_25176_25511.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_25512_25607.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_25608_25727.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_25728_25835.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_25836_26279.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_26280_26411.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_26412_26531.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_26532_26927.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_26928_29459.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_29460_31307.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_31308_31451.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_31452_31511.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_31512_44615.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_44616_44747.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_44748_44819.png FOSS4GUKOnline_2020_Mercator_Marlow_H8UWboFZFY0_44820_44842.png
Hello, I'm Peter Marlowe, I work for CGI, I've been the technical lead on this project that's been running for just over a year now. I'm joined me as well as Catherine Ratcliffe, who's the project manager, so she's also from CGI as well. We've been helping create a digital register of common land in Wales for the Welsh Government. Quite a bit to get through, so let's crack on. So the first bit is what is common land. So common land is land subject to rights enjoyed by one or more persons to take or use part of a piece of land or of the produce of a piece of land which is owned by someone else, so straight forward. What I want to touch on here is just a few of the kind of references and key terms that you're going to hear me use in this presentation that's related to common land and try and explain those. So you know what I'm talking about when I come to mention them. So the first one is register unit. So register unit is the official term for a piece of common land, so within Wales there's lots of common land and the term to refer to an individual piece of that is a register unit and that usually has a reference as well, so for example CL120 that would be the unique reference for a piece of common land within Wales. Second point here is dominant tenement, so not to be mistaken for the famous actor who played Doctor Who and so dominant tenements or usually refer to them just as DTs. So these are the people that own the rights, so they own the specific rights over common land and they typically tend to be farmers. And then lastly we've got the rights of common, so these are these detail the specifics of the rights, so for example grazing 10 sheep. So kind of in conclusion, you have dominant tenements who have one or more rights of common and those rights of common apply to a register unit which is a piece of common land. So in terms of the background around the project itself, so obviously the register of common land exists at the moment in its paper format. Each local authority in Wales has its own version of their register that they maintain and that's open to the public at the moment but obviously in order for people to come and see it they have to book in with their local authority a range of time, physically come to the offices and open up the register and find the bit that they want to take a look at and so ties in a little bit with point two there which is time consuming, so obviously that isn't a quick process, you know in this day and age people expect to see you know information after a couple of clicks on the internet. So part of this making it digital will allow the public for the first time to be able to access all the information to do with common land in Wales, you know online, you know no restrictions they don't need to go to their local authority and request time, they can see it all online which is fantastic and again on the second point there for the time consuming where there's disputes in common land where there's rights apply and the details of those rights but often they end up resulting in kind of legal cases and that takes a long time to resolve especially having to go back to look at the physical registers and trying to determine who changed them and what change they made and that feeds into the third point there which is the register maintenance and so by making it digital it's not just allowing the public to see that information but actually all the local authorities will be able to maintain that register digitally going forward and that will be the official legal register and as part of that this system is recording all of the changes that are made to this full audit history so we're recording who made the change, why they made the change and what they actually changed and that's really important in these legal cases to determine what action should be taken. So the fourth point on there which is about the physical state so as I mentioned they are paper based at the moment, they're quite old and so you know they're starting to deteriorate and we'll have a look at a minute that's come to the state of the maps and the kind of damage that's happened to them obviously by digitising it we can make it last forever essentially, we don't have those issues of deteriorating over time and the last bit on there is the data sharing so it's really critical that when we digitise this data we can make it available to other departments within Welsh Government or other organisations outside of Welsh Government and obviously at the moment that's almost impossible to do because each local authority has their own register, you know it's paper based if you wanted to get a Wales century view of common land in Wales, it's really trickle to do at the moment because you have to call off to lots of different local authorities. So just a quick look at some of the numbers here, so there's 22 local authorities in Wales as I mentioned each one's got their own register, most of these registers are kind of more than 50 years old so they're old documents in terms of the quantity of maps, you know we're looking at about 12,000 map pages, 25,000 register pages around 70,000 land and right entry pages so that's a lot but I think the point I want to make here is it isn't really a big data project we're not talking about millions of records that we need to process here but the challenge is more in the quality of the data that we're trying to transform so on this next slide here hopefully you can see that in some detail so this is some example of register maps as they are now in the paper based register and you can kind of see hopefully from this that some of the quality is not fantastic and obviously these maps have been drawn on, hand drawn to trying to note the different boundaries and where edits have been made sometimes they're kind of on top of each other, pretty difficult to determine where the boundary starts and where the boundary ends and the same is true of the register pages so here we've got a couple of examples so this is the textual content of the register so the rights we can see here that they are handwritten if they get voided then obviously they just get crossed out and in a lot of cases normally in pencil where they've done an update kind of in line is quite tricky to decipher the kind of changes that have been made here and what's the kind of real legal version of it and the hope is that by digitizing this we can standardize that so there's just one way to create a right one way to record common land in Wales. So without further ado let's jump into the application so I'll come out of here. So hopefully you can see this on my screen okay so what we've got here is the application is in two parts so this part I'm going to show you first is search the register there's also manage the register so this is the screen that the public will be able to see so anybody can come on and start using this service and we can see here there's a couple of options that we've got to find this common land data because obviously we're phosphogy and we all have maps we're going to use the by location search so I'm just going to search for my favorite village in Wales or favorite after starting this project and what we can see here is that's brought me up brought up a map for me and I sent it on that location in Wales and we can start to see some of the register units that are in green and the dominant tenements that are in red and we've got some of the textual information around those on the side here so I'm just going to go into full screen because bigger maps are always better and at this point obviously in Wales and I can just kind of zoom around and I'm free to go and look around and look at common land and dominant tenements kind of anywhere in Wales at this point I'm not restricted by that initial search and this is test data so this is on our test system but a lot of it's indicative of real data so I'm going to come back here now and just come back to CL120 and if I hover over this I get this pop up and it allows me to drill in and see more details for this register unit so if I select that I'm now on the second map page now this is a page that's kind of centered on this specific register unit so we can still see that register unit in the bottom right here and now we can see a couple of dominant tenements and the important thing about this page is it's focused on this register unit and just showing who has rights over this register unit so it's only showing the dominant tenements that have rights so if I scroll down a little bit you can see some of the tabs here and you can see there an example right of common that exists so this is in the kind of formatted standardized version of this so we can see they've got the right to graze as the type and we've got the subjects that they can graze. We've also got just as a temporary measure here we can see the original text as it was in the register before it was transformed so these are the original five columns from the register with their original data so I think the bit that's interesting here is column four which contains the details of the rights so we can see here that's basically just a big kind of block of text you know paragraph of text is difficult to query that it's difficult to break it down and do a lot with that whereas in the new format you know it's split out we spit out the type of right which is graze and we spit out the numbers and the subjects of that right which means you know going forward it's much easier to query that and get an idea of you know how many grades and rights are there in Wales which you know you just couldn't do in the paper form or even just taking the paper form in its current format and putting it digitally. So if I come back up to this map here I'm not constrained again I can start zooming around as well here if I want to get it slightly closer in and we can see some of these purple polygons boundaries so those are deregistered areas so they were bits of commonland and for one reason or another they're no longer commonland and the last bit I want to share you on this public side is just this notion of what we've kind of termed as hopping so at the moment obviously I'm centred on this register unit CL120 but if I want to change my context and say actually for example this DT30 where else does that DT have rights which other bits of commonland does it have rights and I can actually select that from here and the context is updated the maps updated not sure how good the video quality is there in terms of the frames per second but it's a smooth transition that's brought that map out there and it's now showing me that actually this dominant tenement has rights also over CL30 that we can see up in the corner here and the actual list of rights has been updated to be specific to that dominant tenement and that I'm not stopped there I can carry on doing this kind of hopping so I can say okay who's got rights over CL13 and I can select that as well and we can see in this case quite a lot of people have rights over CL13 so it's just a really good way of you know initially placing yourself somewhere looking at a certain bit of commonland and I've just been able to kind of continue your investigation continue your search into it and you know keep jumping and looking at rights and looking at different bits of commonland. Okay so I'm just going to quickly go into the admin side of the portal so this is the local authorities that would sign in to make changes to the register. I'm just going to sign in with my government gateway account and this is a test government gateway service so there's no need to try and write down my access code it's not a real access code and it's definitely not my pin number so now I'm in we can see now that the screen is talking about managing the register and what I can do from here is I can select update the register so there's quite a few steps here different paths through I'm going to kind of skip most of these because they're not really relevant to what we want to show here and get to the bit where we can see some maps so I'm going to select the register unit that we saw previously and I'm going to select to create a new rights entry so I'm going to keep these details the same as default and I'm going to start seeing this in here so now that we've kind of normalized it standardize this data it's meant that we can select stuff for drop down to just much easier to obviously create rights with so I'm just going to say that they can graze 10 cattle you know I could keep adding rights of common there as well I'll just add that one for now so now I need to put in the rights description so this is a description of the dominant tenement so who are the rights attached to and what's important here is I have to put it in English and in Welsh so obviously the system itself labels the screen it's obviously available in Welsh but the actual register content as well will also be available in Welsh and that's a big plus so the register as it is at the moment it's paper format it's only available in English but the digital version will be available in both English and Welsh so that's a massive bonus for Welsh speakers so I'm just going to put in the dominant tenement name in this case the name of a farm and I'm going to select to digitize this dominant tenement so here we've got a map again and we can see my register unit if I zoom in a bit we can see we're getting master map data back as well and I've got these extra controls on the side here that are going to allow me to digitize this dominant tenement so the first thing I want to do is just turn on this epoch 2 data so the dominant tenements a lot of the time their descriptions are just and field numbers and obviously in modern versions of master map data for modern survey those field numbers aren't included anymore and so what we've done to aid the digitization processes we've processed the epoch 2 data and we've loaded that into into geoserver and we're serving that up basically to allow them to overlay that over the up-to-date master map data and to allow them to capture dominant tenements based on the field numbers so I'm going to select to create here I'm just going to tweak transparency a little bit and I was going to select to snap so this is snapping to the the underlying master map coming from modern survey so not obviously to the epoch data because that's just raster and the master map data here is actually a it's a vector feed WFS coming directly from via Europa which is really nice so we don't have to host that or manage that it just you know comes to us and that's because the Welsh government got a public sector mapping agreement that allows them to have access to that which is very nice I want to quickly show you these other tools that we've got here so we've got this reduced polygon and combined and basically what these let us do in a lot of cases it's necessary to kind of cut holes out of the geometry and if they're bits that aren't common land you know they've installed an electricity pile on or or such and there's cases as well where they would need to kind of do the opposite and fill them back in so they've got the tools to be able to do that and lastly I just wanted to show very quickly if I kind of edit this now and move that boundary so it's over the register unit which is the green boundary I get this warning come up telling me that you know the dominant tenement boundary intersects a registered area and so what we do is we put kind of validation rules in and there's lots of these for the other layers as well it kind of help the user guide them to be like okay the update you're making might not be strictly correct now in this case it's only a warning because the data that we're transforming into the digital register some of that data is already going to have these inconsistencies so we don't want to be too strict because the person making that update they might not be able to you know make a decision as to which is the right way to go but we've got a raft of those kind of updates those validation rules to help them you got about five minutes Peter okay thanks and so that's showing me digitizing the dominant tenement so that's fine I've done that I come back to here there's lots of extra information I could put in here but it's not really relevant to what we want to show today so I'm just going to submit that so that's gone in I can view that update and as we can see I'm back to this page again and hopefully you know you can see that DT that I created is there and the right that I put in is also there as well cool okay I'm just going to jump back into the presentation quickly so in terms of what's what the benefits that open source geo has given to this project that we've put together so obviously we've mentioned before we're using geo server that's super fast WMS so one of the things that love that geo server is the performance of that so we've kind of seen that on the big map there bring about those registered units bring about the dominant tenements you know really performant the OTC standards that has out of the box so we touched on it earlier as well it's critical that although this register of common land it's in charge of managing that data within Wales but it needs to be able to share that data with other systems and other departments and hopefully you caught the data map Wales presentation from my colleague Pascal earlier today and that's a great example of a system in Welsh government that will be able to harness this data and display within their system as well so though they won't be the owner of it they won't be able to update it but in a read-only capacity they'll be able to get you know common land geometry and display it in their portal which is really critical and the last bit a little bit more technical but because it's open source you can kind of get under the hood of it and diagnose any issues you can get into the code you can debug it so we didn't have any issues on our project which is great but just knowing that that option is there is a great place to be so the second part is open layers that we're using so obviously that's a very powerful comprehensive web mapping library and in our case you know we could tailor that to the specific application needs so some of those geometry validations some of the geometry alterations that we were making and the kind of zooming in and out changing the context you know open layers allowed us to do all of that you know pretty seamlessly and the last bit there we're using that in combination with JSTS which is the library that allows us to perform those fairly complex geometry operations you know cutting holes out and we could have used turf.js as well you know those two are pretty interchangeable. So just want to highlight one of the challenges we had because you know we always have challenges and we were using obviously we were using SQL Server as the database management system that was the preferred option from Welsh Government mainly down to security but obviously that isn't often the first choice of database when you're working with Spatial usually it's Postgres, PostJS so that creates some issues just because it's more difficult to find to help articles and documentation on that and we did have some bugs when it's put some work around in you know just because working with Spatial SQL Server is just a little bit less mature than some of the other alternatives but in the end the effort was justified obviously we have it working now and what that's really given us is the ability to use SQL Server as your SQL in particular which you know if you host your application in Azure you know SQL Server has got the best support, the best level of functionality, the different services to do with scaling, you know security, availability all of that stuff you know SQL Server within Azure is the best at that so it's given the application access to that which is which is fantastic. Okay the last bit one on just top Sean so we are deploying this application into Azure you'll probably do a half hour talk just on that but I just wanted to pick up two points about how we are doing that that is maybe slightly different to how other people are so the first bit is how we are building this so the application is really in two parts so it's the portal application so that's written in.NET Core and then obviously we've got Geo Server that's serving up the GIS information that's obviously in Java you know two different languages, two pretty different applications but in this case they're both built with Docker and they're both deployed in containers using Ubuntu and that gives us a bit of parity essentially in that build process even though they're both very different and then how do we actually deploy that up into the cloud so we're using within Azure there's an app service and that's their kind of PAS platform as a service and within there there's the option to use web app for containers which is what we're using and that basically allows you to take those containers and deploy them up into Azure as web apps and that is obviously managed service from Microsoft from Azure and that helps with things like availability scaling and security particularly on the SSL side where I don't know if anybody's tried to do SSL and Geo Server before sometimes that isn't straightforward can be a bit of a nightmare so anything that helps with that you know is really good. Okay so that was it really I know that was a little bit of a whistle stop tour going through it but I'm hoping I covered you know most of the stuff and certainly how the open source Geo elements have really enhanced the solution. Thanks very much Peter that's great yeah we've got a few questions that have come in so I'm going to go straight on to those and run through those we've got eight to nine minutes I guess. So first of all from Victor about base mapping saying asking what base map did you use and was can open street map attribute data be included in the database is there any element of crowd sourcing in this project? So yeah good question so there's no element of crowd sourcing so in terms of the base map that we're using so it's the standard audit survey master map and the base map as well so that all comes from this far Europa service that we have access to so that provides us with a base map but also provides us with the vector geometry for master map which is what we start to on the admins range. Cool thanks and the next one is from David says hi Pete fantastic project and presentation the question is have you used these technologies in other projects and if so what are the main issues you've faced with the technologies I know you mentioned some issues with SQL server for example. Yeah so that was certainly one of the bigger ones and just because a lot of that was unfamiliarity so yeah a lot of our other projects that we've run is usually post you know Postgres or PostGIS if you're working in the spatial domain so SQL server did provide a few issues there probably the other big change for this project is it's all it's all in Azure and it was using web apps so that's slightly different perhaps different to what's normally done and especially in terms of Geo server so taking Geo server and hosting that in a web app there was some stuff that we had to do configuration wise to get that and to work if you've got a little bit less control when you deploy it into a web app so although obviously Microsoft takes care of some of that scaling and you know availability but in terms of getting getting behind the scenes looking at what Geo servers doing there's a little bit more configuration you have to do there and so I think those are probably the two main points especially on the SQL server side the geometry so in native SQL we take that and we obviously transform it back into Geo JSON WKT so we can work with it in the browser with in open layers and that kind of transformation back and forwards there was a couple of that there's a couple of cultures there that we had to be careful of. Cool the next one is a comment really from Dave who says he had no idea that common land wasn't digitized in Wales so it's back in the day in 2004 in natural England so I'm not sure I've lost the end of my natural map to all common land in England so it's getting one up for England there I think. Yes so I mean the thing to mention there as well is that as far as we're aware certainly in the UK maybe even worldwide certainly in terms of taking common land and building a digital register Wales seemed to be the first in the UK at least to do this so we know in England obviously they don't have a digital register and the same with Scotland so they might have the common land geometries digitized but this is kind of a full service that allows them to completely manage their registers and basically get rid of the paper-based ones put them in the National Archive but they don't need to rely on those it's the legal thing anymore it's all totally digital so I think Wales are leading the way. So we've got one up for Wales in fact. And the next one so Tony asks aren't vector tiles generally simplified and so how does the SNAP2 vector data work with respect to that? So yes so at the moment the way that's working is the actual the raster imagery that comes back that still comes back as raster we still get that master map back as raster and what we're doing is we're loading WFS service for the vector data and we're kind of loading that in but we just don't show it so it kind of sits underneath underneath the raster and then with an open layers you can you can basically give it that kind of vector vector layer and say you know SNAP2 this and so obviously in this case because both of those sets of data are coming from the same provider they kept in sync so as far as their users concerned there's no you know there's no difference there the raster always represents the it's always in line with the vector but what it meant was we didn't need to we didn't need to manage that so we wanted to avoid having to take the master map data and you know put it into Geo server and do all the styling and that kind of stuff which is quite a lot. So it's not simplified at all the vector data? No. No okay cool. A question from Akin has Wales got a digitized brownfield map do you know? I do not no sorry. Okay somebody else might know. Okay a couple more questions they keep coming in this one from Ulrich not really a Geo question but I'm going to ask it anyway how was the textual content captured and how was the transition to the legal acceptance of the digital data from the paper maps managed? Do you want to answer that one? Yeah so essentially each page and each map was scanned in so we've got some scanners going around Wales literally scanning in the pages and then they use OCR so Optical Character Recognition to take the data they can and handwritten data is actually manually typed so a very very long long process which we're still undertaking to be honest and then basically they're linked to the maps with sort of you know unique keys which are always put into the database so that's done. That'll be going on for another year. So is it legally accepted now that the digital data is? So not yet it will go through the local authorities consultation which will happen over the next 12 months and then it will go to public consultation which is spring 2022 and once that's happened then it will become the legal register. Okay cool so I've just got one more comment here which is batting back for England so Sarah says rural payments agency England maintains the digital record of the common land register for the purposes of inspections which can be accessed by the public on request. So who knew? Just don't tell Wales. They're very proud of thinking they've beaten England.
in with their local authority a range of time, for the first time to be able to access all the to look at the physical registers and trying to the local authorities will be able to maintain there's 22 local authorities in Wales as I here but the challenge is more in the quality of register pages so here we've got a couple of be available in both English and Welsh so that's quickly so in terms of what's what the benefits that open source geo has given to this project because working with Spatial SQL Server is just a portal application so that's written in.NET Core which is what we're using and that basically allows you to take those containers and deploy Azure and that helps with things like