Creating a New River Network for Ireland
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68447 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Europe 2024 Tartu29 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
Streaming mediaImage resolutionSoftwarePresentation of a groupAxonometric projectionSoftwarePhysical systemLine (geometry)Prime idealScaling (geometry)Integrated development environmentMappingPresentation of a groupEndliche ModelltheorieTerm (mathematics)Streaming mediaSlide ruleFirst-order logicDataflowSet (mathematics)BitLevel of measurementArchaeological field surveyVector spaceTexture mappingMetreVector graphicsLecture/ConferenceComputer animation
02:18
Image resolutionProgrammschleifeConnectivity (graph theory)DataflowSoftwareCASE <Informatik>BitCombinational logicDirection (geometry)outputCausalitySkeleton (computer programming)Proper mapConnected spaceReference dataVector graphicsLengthMappingPolygonSet (mathematics)Web 2.0CalculationLine (geometry)AreaGreen's functionStructural loadStreaming mediaFlagError messageTexture mappingMathematical morphologyWater vaporOrder (biology)Loop (music)BraidContext awarenessSynchronizationOpen sourceRight angleRandomizationArrow of timeScripting languagePoint (geometry)ProgrammschleifeMereologyDirected graphArchaeological field surveyDependent and independent variablesIntegrated development environmentTrailControl flowText editorMetreRoundness (object)Computer animationLecture/Conference
09:03
Connectivity (graph theory)DatabasePairwise comparisonComputer networkWorkstation <Musikinstrument>Computer wormPrime idealDefault (computer science)PolygonSocket-SchnittstelleProcess (computing)World Wide Web ConsortiumScripting languageDifferent (Kate Ryan album)Software development kitSkeleton (computer programming)PolygonoutputProcess (computing)Point (geometry)Graph (mathematics)SequelServer (computing)Text editorMultiplicationSet (mathematics)First-order logicLevel (video gaming)Sheaf (mathematics)Texture mappingMultiplication signConnected spaceStructural loadFeedbackLoop (music)Control flowGraph coloringSimilarity (geometry)Visualization (computer graphics)SoftwareBitWeb 2.0Library (computing)GeometryWeb pageSingle-precision floating-point formatMedical imagingAreaDirected graphCartesian coordinate systemHeegaard splittingOpen setMetreMereologyQuicksortOpen sourceTerm (mathematics)Sampling (statistics)Factory (trading post)File viewerOrder (biology)Price indexMappingLine (geometry)Heat transferSelectivity (electronic)Mathematical analysis1 (number)Streaming mediaDataflowProgrammschleifeKey (cryptography)Workstation <Musikinstrument>Water vaporFront and back endsSocket-SchnittstelleWeb applicationDebuggerDatabaseInteractive televisionLink (knot theory)Shared memoryInterface (computing)SynchronizationLecture/ConferenceComputer animation
15:47
World Wide Web ConsortiumSocket-SchnittstelleProcess (computing)Raw image formatLibrary (computing)Computer networkNeuroinformatikCompass (drafting)Component-based software engineeringTexture mappingMathematical morphologyCalculationExecution unitData managementSoftware developerLengthBitArithmetic progressionSoftwareMapping1 (number)Term (mathematics)Set (mathematics)Direction (geometry)Software frameworkTraffic reportingStreaming mediaOrder (biology)MereologyLibrary (computing)Process (computing)Projective planeSocket-SchnittstelleWeb 2.0Function (mathematics)Scripting languageGraph (mathematics)Self-organizationNumberFacebookTexture mappingOpen sourceView (database)Flow separationCodePresentation of a groupLengthFront and back endsFirst-order logicPairwise comparisonDebuggerCalculationSQL Server 7.0Water vaporComponent-based software engineeringField (computer science)Different (Kate Ryan album)Temporal logicStrategy gameAlgorithmMultiplication signMathematicsPoint (geometry)Physical systemStructural loadSlide ruleAreaAuthorizationRoundness (object)Open setComputer animationLecture/ConferenceMeeting/Interview
22:31
Least squaresRoundness (object)Computer animation
Transcript: English(auto-generated)
00:00
Hello everyone, so yeah my name is Seth and I've been spent the last six to eight months working on creating new river networks for for island So in this presentation, I'm just going to go through a bit of the background the problems We came across the software. We used how we validated the river networks and then a summary So back in 2005 we created
00:22
I'm old enough that I was helped to create the first set of river networks for island in Yeah, 2005 for the Environmental Protection Agency So they manage all the the environment so they have an interest in the river network for things like runoff and pollution So we created this back in 2005 using vector mapping from the National Mapping Agency the Ordnance Survey islands
00:45
And that data set was was made in 1992 so it was based on 50,000 scale mapping so that's an example in in the background and And then in 2014 the National Mapping Agency came out with a new vector data set called prime 2
01:01
So this covered the whole of islands or Republic of Ireland so everything had a feature It was a lot more accurate They changed the the projection to a new more accurate projection for island So the EPA in the last couple of years Started doing projects where things like sinuosity and accuracy of flow became more important
01:25
So they decided to switch from the current networks that are in use in about 10 active systems that are continually running So they want to kind of update all these systems to use a more accurate river network for more accurate modeling So in this example here, you can see the the old river network
01:41
It's kind of the jaggedy light blue line And then the new network is the the green line so you can see there you've got the the more accurate sinuosity it lines up best with the aerial photography and In some places kind of the the old mapping was 15 to 20 meters out in terms of accuracy So in this slide you can kind of see that there's a lot more river channels are mapped
02:03
So the light blue is is the current EPA River networks, and then you can see here that There's a lot more kind of first-order streams So the purple mapping is is what we're going to use to to create the new river networks So a nice thing about the the new data sets is that it's not just line work
02:22
They also have polygons for the water So you get widths so this was a key thing for the environmental protection agency is to have the river widths Because then you can do more calculations on load and flow and again sinuosity and morphology So the new data sets didn't just have lines and nodes, but it also had
02:42
The polygons so each polygon had a center line. So you had an association between between the two features So that was kind of the background as into why we were going to switch to new river networks So then the next step was to actually turn them into proper networks where we could calculate flow and direction
03:00
So we ran into running into because this is ongoing a few problems So I'm just going to run through them through them here So the first one is that The the data is highly accurate, but it's it's not connected So a lot of the digitizing is based on aerial photography So sometimes the river would go under a road and there'll be no river line there. Then the river network would break
03:25
So in this example, this is one of the hydrometric areas the bits in red disconnected from the bits in green So the first step to turn these into proper connected river networks was to turn the red bits green So basically we had to go in find all the gaps and connect them together
03:42
So We started creating a kind of a toolkit Using the web using open layers so that we could quickly come in and connect and the broken bits So in this example, just connecting underneath the road This is a nice simple case where it's kind of obvious where the flow goes But there were a lot more complicated things. So we had we have about 60 reference data sets
04:02
So we have mapping from the early 1900s Which was highly accurate kind of black and white old mapping that would show kind of the original ditches and flows And then we have a DM so we get the kind of the the generated flow directions So we use the combination of those to try and find out where whether if there were gaps how we would connect the data together
04:24
Another cause of gaps was was when the the water flowed through a lake or a water polygon So we had to connect these so none of the lines would be joined together So again, we created some tools so you could just click on the polygon It would find all the the inputs to the water polygon
04:41
It would connect them together and create a skeleton and then you could manually make the skeleton look a bit a bit nicer But the idea is always to make sure that you have the fully connected flow from all the the sources down to the sinks One of the other things with the complicated matters was there there are lots of underground connections water connections in Ireland
05:03
So the water would disappear underground and the geological survey in Ireland They're responsible for kind of mapping these these areas, but they have theoretical collect connections So these bits in in kind of pink they show where theoretically two water bodies might or two rivers might connect
05:20
So for big chunks, we try to work out or just add in theoretical lines Just so that we have the connection between the two So again, this would all then lead into things like you could work out if if pollution is coming in at one point in The river network, what will affect downstream or what will happen upstream? Then the other major thing we had to fix was the the flow direction
05:42
So the the original data set it's nice and highly detailed Everything's fully connected at nodes where it is connected, but the the line direction is is random So it just follows the the digitized direction So the data would come in like this and one of the things we had to do would be to correct the flow
06:01
So all the arrows are pointing pointing downstream So there's a set of Python scripts that would would kind of go from every source down to the sink and then flip all the Lines around so that everything was pointing in in the right direction And then yet once we managed to connect stuff up and we got the flow together then we could start
06:21
working out things like Stral order and the connectivity and then you can Do the calculations like if you click anywhere on the river network You can get all the connected segments upstream or you can get all the connected segments downstream One of the things that we we didn't count on so we did a pilot hydrometric area
06:41
But then when we came to the national rollouts We found out that there's actually too much data So we thought we would be connecting all the gaps. So we're aware of those but what we weren't aware of was loops So basically there were connections where there shouldn't have been connections and there were things like 10 kilometer loops going round
07:00
Which Hydrologically doesn't sound correct. So there was a huge exercise. You can see here This is just one hydrometric area and all those red bits are where there's a loop so in some cases like a braided river a loop might be might be valid but As far as I'm aware loops like this shouldn't really be we shouldn't be there unless there's there's flooding
07:23
So in these cases we we manually found like the best place to break to break these loops for the bigger loops So we could use that based on the current river networks So it was handy to have a reference data set that's already been validated for for 15 years So we knew where to break them But there were so many that we had an automated approach to break the the smaller loops. So it was based on
07:45
The stream order or the associated polygon width so we'd remove the smallest bit of the the network that we could We don't actually remove anything we just have a flag so we can bring these all back So when people go out in the field and say that the networks are wrong We can we can bring these back and then remove different bits of the network
08:05
So again, we had to create a whole tool set That bits in the background is probably the worst bit of the river network It looks like an error, but these were all designated as streams. I think they're meant to be tracks But 99% of the data was was very high quality, but when we ran into this
08:21
Yeah, it caused a million loops to appear in that in that channel and kind of crashed all the tools So we had some manual tools so you could deactivate the edges Again through a web UI so an editor could come in and and break those And rather than removing whole edges There was also a tool that you could split edges so you could remove chunks
08:41
So you could put in you want to break 50 meters here and then just click on the map and it removed that Part of the network. So again, this was all to to break the the loops And one of the other issues was was around the hydrometric area borders So we we have the country split up into 40 hydrometric areas
09:01
But again, these hydrometric areas are based on the original river network So we assign them we assign the edges to each hydrometric area, but obviously there's a lot of edges that cross borders So there was a whole process of trying to find out Which edges should flow into to which hydrometric area and breaking things along the border? Fortunately, there'd been another mapping exercise where they created kind of newer catchments
09:24
So like the jaggedy black line was a new catchments and this actually mapped up with the the more accurate data So again, there was a tool set around transferring edges either dynamically based on how many edges they were connected to Or we also had tools where you could just select edges and transfer from one hydrometric area to to another and
09:47
Then obviously there's another bigger border on the island of islands between between the north north and the Republic so Basically the the National mapping agencies water stops at the border So they stop mapping at that point, but obviously if you want to do any analysis on those hydrometric areas
10:02
So the ones kind of with two colors You're going to be missing a huge chunk of the river network. So I guess in Europe, it's probably a bigger problem. But um, There was just one border to be able to deal with here so we're either going to try and get some data from Northern Ireland that said the same accuracy and then try and stitch it together or
10:20
Use the current river networks, so There'll be a lot less data there. But at least there'll be some indication of Strala order and things will be vaguely correct around in those catchments So once we we can connect everything for a hydrometric area that we have to validate it So there's a few different techniques we had to to validate the the networks
10:44
the first one is Monitoring stations, so the the EPA the Environmental Protection Agency has 10,000 monitoring stations around the country So these are places where people go out and and take a sample in the water or they could be automated sampling points
11:02
So these monitor kind of key areas of the network. So let's see if there's a factory or something They might put a monitoring station downstream of it So with this point data set we could find out which bits to our new network were missing So that they were in the old network, but they're not in the new one So for example, if they were 15 meters from the the new network, we'd go and investigate and we might bring in the current network
11:25
So there might be bits missing from the new network. So we'd have to add those in So these are kind of key parts of the network that that they were interested in monitoring So so we wanted to make sure that they were available in the the new networks After we completed each hydrometric area we kind of got all the the summary stats
11:43
We could generate kind of automated images using WMS And then we can do things like compare the catchments from the current network with the new network So we get all the edges that connect down to a sink so a sink is where the rivers may meet the sea and we could just compare the colors on the
12:01
current network with the new one and Sometimes that would highlight that we we'd had some connection that should have been broken and everything be the wrong color So there's a kind of a quick high-level visualization of the the new networks And Similar with the flow again if we'd connected to bits of the the new networking correctly You'd see that there's a massive river that in the current network was a first-order stream
12:22
so obviously there was an issue and then we'd have to go in and investigate and Yeah, as I say the loops was probably the most problematic issue. So there was a whole tool set of them All run through the web so you could you could check for loops. You could check they were removed And you would highlight them on on the map and then obviously at the database level
12:44
We have a load of checks around making sure that a line has a node at the start and the end There's not overlapping geometries Every sink has some sort of connection Every edge flows into a sink every edge has an upstream source So there's a lot of stuff you can you can pick up at the database level
13:04
Okay, so in in terms of the software I Get a map server mentioned in here So at the back end we're using map server to serve out about 80 or 90 WMS and WFS layers So these could go into the into a web viewer We had a sequel server back end for the database and then in the middle
13:24
All the geo processing and things was done using Python so there lots of different Python libraries and then these are wrapped up with a Python web API and Then the front end is a JavaScript single page application. So based on open layers and xjx and geoext
13:42
So kind of that's That's most of the tech stack so it's split into the front end The middle section is all the Python libraries and then the back end is map server and sequel server So we put it in a web editor so that we could have multiple people editing
14:01
River networks at the same time And it was also it was also useful because we could have permalinks So any section of the network we could zoom in we could get the permalink So if there's an issue we could then send the permalink to the EPA or there would be other stakeholders Who are interested in the the river networks? So they could then have the permalink and then they could give their feedback on is there an underground connection here?
14:23
Does it look right that we're breaking this loop here? so having it all as a web application made it a lot easier for the sharing and And showing issues between different stakeholders as I say yet that the Python way web API. So it's a fast Fast API interface. So there's a lot of Python scripts in the background and then it's all wrapped up in the in a nice API
14:48
One of the more interesting things on the back end was creating these skeletons for the for the water polygons So this was the side kit has a nice feature where you can create skeletons for for a polygon
15:01
Initially, it didn't work because the polygons were too detailed. So there was a process of making sure the input points were kept but then getting rid of points simplifying the points in between and Then you could generate these skeletons and then using a graphing library You could get the shortest path between all the points and that would connect everything up
15:20
So this was probably one of the bigger locks in western islands and one of the more complicated Features then if you if you had the time you could then go in and modify those lines So it'd avoid islands and things But yet the key thing here was just to make sure that all those bits were then connected to to downstream rivers All the Python scripts would were running through web sockets so that you could get interaction between the the geoprocessing and the web
15:47
Otherwise you'd be sat there stuff would be happening for five minutes in the background you wouldn't really know so the the use of web sockets made it so that you could basically have the outputs of the Python scripts coming into the web front end so, you know when things were had finished processing and
16:03
There was a couple of Libraries Python libraries that were open sourced as part of this project So there's a library called cascade that we released that can create the Straler or Shreve stream order Using pure Python so it quickly create the the stream orders for the networks
16:20
And the second library that we've all but also been using for four roads is wayfarer So if you've heard of the network X library, it's a graphing library It's used to connect nodes and edges. So it's used for things like seeing who's connected to on Facebook But the wayfarer library takes that and turns it so that you can use it on geospatial data sets
16:41
And that gave us all the things like can it getting all the paths from source to sinks? And running it all through quickly through the through the Python back end And then we also had some additional components on the JavaScript front end that were released as a map view and Then we also had to export to our map and ArcGIS Pro because a lot of the EPA
17:02
Users what use want this data set on their desktops as well. So there was some proprietary stuff at the end Okay, so just briefly in summary We're about halfway through the project. So that's the progress the bits the bits where the rivers are done The two big ones the the Shannon
17:21
They were quite nasty, but they're they're completed now. So hopefully it's going to be smooth sailing from from now on to complete the other ones But yeah, one thing that's become apparent is that the the new data set there's a lot more data So for the channel through the upper Shannon, so that's half of the Shannon There's a five thousand edges in the current network and now that's gone up to fifty fifty five thousand and the length of the river
17:46
Network's gone from five and a half thousand kilometers to eleven eleven and a half thousand so in terms of reporting to Europe and the water framework directive there might be an exercise to remove some of the first-order streams because Otherwise the EU reporting people are going to get a shock when they see they see the changes in the network
18:06
So yeah The network's going to take a few more months to complete to move all the systems across is going to take probably a decade because there's a lot of systems built on this but It'll be used not just by the EPA but by other agencies in Ireland. So Irish water the inland fisheries. So things like
18:25
Finding salmon breeding grounds what's upstream from certain points and it can be used for that Nutrient load calculations. Yeah, there's a lot of plans for the new networks from from different organizations So, yeah, I just like to thank the rest of the team
18:40
I think some of them are watching so I'll give a shout out to them And yeah, as I say it's an ongoing project, so hopefully we'll get this done in the next few months And that's everything right? Thank you Thank you very much. Thank you very much this are there any questions?
19:08
So I have a question with the second last night where you know the the comparison what was there and what is now here? that one What are the authorities saying to that, you know, it's
19:23
It you know, is that a big surprise or did you expect something like that we expected more data Not every hydrometric area had that much of an increase on average those about three times more length So this is this is an exceptional one where it's ten times more streams There's a lot of things like ditches around fields that have been marked to streams
19:43
But whether they're a stream the whole of the year is is doubtful So there might be a follow-up project to work out. What's a ditch and what's a stream? Then you get into the The difficult world of temporal river networks like a river network in in January is probably not going to be the same in July But that's a different problem. Yeah, and I think about the implications with heavy rains, but
20:07
Good any other question Hey Great presentation. I thank you. And actually I wondering what was the motivation to choose the Backend based on Python not for example on post.js because when you're explaining your algorithms are always thinking about how
20:27
implementing post.js the same thing, but this is just you have some Strategy for using Python or just you are good at Python and that's why you use it
20:41
Yes, I like I like Python a lot. So that's that's probably number one The second thing you have things like PG routing are great if you got post just Postgres But the EPA is very much a kind of Microsoft SQL Server backend so it's nice to have a separate Python library and I guess the second thing is we we did a Similar project based on islands road network where we took the prime to data for the roads and we connected
21:04
Well, we didn't connect them. They were connected. We had lots of tools around routing and things already written in Python So we're able to use a lot of the experience from from that project on the roads and take it to rivers So we had quite a bit of code that was nice to tidy up and open source
21:20
But code that's been around for a number of years Thank you for your presentation. I
21:40
Might have missed it, but it's a data. Is that going to be available publicly available or is it internal only? Yeah, it's a very good question. So the The OSI they recently changed the organization name. So the National mapping agency is now Talsha Aaron For the original river networks. There was a problem around licensing, but then they allowed it to be released as open data
22:02
So there's discussions ongoing that hopefully the same will be done with this because it's based on their original data But there's lots of value added but as it's going to be a national asset It should be available for downloads. So people can download the whole network and we be used for reporting around Europe Thank you
22:30
Then if not another round of applause for our three speakers