Lidar classification, accuracy and change detection using the Norwegian open lidar data archive.
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 351 | |
Author | ||
Contributors | ||
License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/69110 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Production Year | 2022 |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
| |
Keywords |
FOSS4G Firenze 2022134 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
FingerprintSet (mathematics)Open setGeomaticsFile archiverBasis <Mathematik>Computer animation
00:35
Personal digital assistantSet (mathematics)CASE <Informatik>
01:01
BuildingEntire functionData modelOpen setPoint (geometry)Characteristic polynomialSingle-precision floating-point formatPoint cloudDigital signalSurfaceMetadataPolygonLine (geometry)SoftwareOpen sourcePairwise comparisonTask (computing)SurfaceWebsiteResultantEndliche ModelltheorieProjective planePoint cloudTraffic reportingOpen sourcePolygonDifferent (Kate Ryan album)Square number1 (number)Raw image formatType theoryPopulation densityDocument management systemLine (geometry)Internet service providerTesselationSoftware testingCharacteristic polynomialPoint (geometry)Data compressionBitHistogramElectric power transmissionNoise (electronics)Pairwise comparisonComputer fileCuboidPhysical systemProcess (computing)PlotterAreaParameter (computer programming)File archiverMathematicsService (economics)Doubling the cubeMappingDigitizingAuthorizationComputer animation
05:54
Point (geometry)UsabilitySet (mathematics)ForestPatch (Unix)LaserHistogramDatabase normalizationSupport vector machineAutocovarianceVirtual machinePlanar graphPoint cloudAnisotropyScatteringSurfaceBounded variationVertical directionPlug-in (computing)Error messageSet (mathematics)BitPoint cloudLevel (video gaming)Point (geometry)Line (geometry)Virtual machineSupport vector machineAreaLatent heatPatch (Unix)Product (business)QuicksortForestHydraulic jumpBlock (periodic table)HistogramSquare numberDifferent (Kate Ryan album)MereologyPattern languageAutocovarianceComputer animationProgram flowchart
08:36
BuildingObject (grammar)Electric power transmissionObject (grammar)Network topologyBuildingPoint cloudIdeal (ethics)Type theoryComputer animation
09:11
Population densityArea
09:33
Database normalizationPoint (geometry)Numerical analysisPixelGroup actionBitPopulation densityLevel (video gaming)Computer animation
09:48
Radiology information systemDatabase normalizationType theoryPopulation densityComputer animation
10:05
Entire functionDiscrete element methodPoint (geometry)Point cloudProcess (computing)Graphics processing unitCoroutineScale (map)ImplementationBitProcess (computing)Computer fileProjective planeCoroutineServer (computing)Point cloudObservational studyScaling (geometry)CASE <Informatik>DataflowTensorComputer animation
11:04
Right angleSlide ruleObservational studyAuthorizationCASE <Informatik>AreaComputer animation
11:30
Advanced Encryption StandardHazard (2005 film)Natural numberDifferent (Kate Ryan album)AuthorizationMassGraph coloringRight angleProfil (magazine)Hill differential equationMappingOrthogonalityDigital photographyEvent horizonPort scannerComputer animation
12:31
Natural numberHazard (2005 film)MathematicsSet (mathematics)Event horizonDisplacement MappingComputer animation
13:21
Sweep line algorithmFlash memoryMetreMappingDenial-of-service attackComputer animation
13:41
Denial-of-service attackRemote Access ServiceAreaScale (map)Process (computing)Open sourceCovering spaceOpen setPersonal digital assistantVariable (mathematics)LaserHazard (2005 film)Natural numberMathematicsTexture mappingMappingEndliche ModelltheorieEvent horizonDenial-of-service attackStudent's t-testType theorySquare numberEntire functionMultiplicationImage resolutionOpen sourceProcess (computing)Extension (kinesiology)Point (geometry)ConsistencyCASE <Informatik>Time zoneScaling (geometry)Open setComputer animation
Transcript: English(auto-generated)
00:00
Hello, I'm so impressed so many people made it to the last talk So my name is Christopher Knuth I'm a geoscientist. I consider myself I'm interested in geomatics remote sensing and together with my colleague Sonia Greve We have been looking at the open light art archive in the Norwegian government shall I say and
00:24
I guess what inspired this talk for me was meeting the people developing open topography if anybody knows what that is Where you can access and so the basis of this talk is basically to showcase that data set I'm gonna go through a little history and then I'm gonna talk about the data and how you get it
00:43
I'm gonna discuss shortly some of the methods but very Not detailed and I'm gonna discuss some of the accuracy of the data sets that are on there And then I'm gonna talk briefly again Classification and issues we've had actually scaling which I've heard many good talks about so far and then end up with use cases
01:01
So in 2010 the Norwegian government passed the spatial data act basically what this did is this like paved the way for opening up a national infrastructure for housing and Actually distributing geospatial data. The other thing this did was that insured anything publicly funded Acquisitions would be given to the public to be able to download
01:24
so in 2016 the government Started a a project called the national detail terrain model project and what their mission was to create a one-meter DM over the entire country and basically using laser scanning All points or all ground coverage greater than two points per square meter
01:44
How are they're gonna accomplish this was using only private companies and then all data was made publicly available Now you can go onto these websites. There's two different Front ends or websites now and you can actually if you I think you have to register
02:01
But maybe not even not you can search the archive and you can just download laser data Currently as of last week, this was the coverage and The majority of the coverage is light are but there's areas where they only do photogrammetry because there's not much Vegetation so then they choose to only use photogrammetry and air flights on this
02:24
Status as today is 230,000 square kilometers or laser scanned I write 12 terabytes of compressed data, but this is like double compression So it's not just las it's then all the last files compressed again 13,000 individual projects which are individual acquisitions from one flight and
02:45
Then what they're doing is using private companies. So the private companies are Acquiring this stuff processing the data and submitting it back to the mapping authority Some of this work is maybe a year too old
03:01
So these are a bit outdated But it gives you an idea of like the ground point coverage over in the entire country and the most the highest acquisitions are usually where the populated places are and along corridors we have roads and rivers and Landslides and other things The other thing you'll notice is that there's a temporal Portion to this of course the project didn't start until 2016, but you see data from back in 2003 years now available
03:27
this is quite interesting because then you can start thinking about doing change detection you can start comparing and Looking at looking just at the histogram of the projects you can see there's quite a data from before 2016 here and then on this plot you can see anything that's darker has more temporal coverage
03:46
and so it's a majority where the people live Basic characteristics of the data is that all the data is chopped up into a national grid and it's actually a standard grid and they're about one to two kilometer tiles and
04:05
What the companies have to do they have to provide original point clouds and they have to be Classified some basics some are more advanced so the basic ones have ground noise and everything else Some of the more advanced ones have power lines and everything else so you can do any type of classifications and test them
04:26
The company's also had to provide digital terrain models digital service models You get metadata which includes flight lines border polygons point density and a project quality report which In my opinion is quite limited the reports of these don't really show exactly how they tie their data to the surface of the system
04:48
but you can check it out and So the methods that we used are mainly open source of Python PDAU or poodle or whatever
05:02
QGIS um we also looked a bit at cloud compare white box and last tools I'm not really showing much of this, but I'm just gonna show some results or talk about some results and One of the basic even though all these DMS are created already we decided let's do it ourselves So we take our raw point cloud through the Smurf filter And then we ended up getting ground classified and we get four products out of this a surface model a terrain model
05:27
height over ground model and a density We compared and we tried also the last tools the p-tin and the white box tools And I think the results we have is that the Smurf filter is the fastest most robust and you can actually
05:40
pre-parameterize it based on your topography and That makes it really automated if you know which parameters you want for the type of topography that you have so this is really awesome awesome job PDAU and Here's some of them comparisons. I just grabbed a region This is actually not far from my house and showing one area
06:00
that's got three acquisitions one in 2013 one in 2015 and one in 2017 and looking at the ground point coverage is exactly almost like you can see it on the map here on the histogram the 2013's is less than two points per square meter 2015 you're getting different lasers, and you're getting above two and then in 2017 you have a really great coverage with up to four almost ten
06:24
This is just a forest patch here because I talked a little bit about vegetation as well So this just shows you some of the data some of the qualities of the data now. I'm just going to compare What we did when we do to a DTM from these two products just to see what's the accuracy of?
06:41
the raw data that we're getting and so Generally the accuracy is quite good Plus minus 25 centimeters that's within the specifications in the Norwegian government when you look at the data You'll see the flight lines here in purple and so I guess this is something very common Is that I think it's problems of the the gyre and lack of maybe correction of it
07:04
So and some of these jumps will be up to about half a meter, but that's within the original data Maybe there's some corrections can be made Now choosing five or three other places here just a spot check We see the same pattern with these flight lines
07:24
Here is a valley and so up on the steeper parts of the valley you see that the errors get a little bit more and That just depending on where that light are actually reflects from how big the footprints are that sort of thing but up here Here's where you see a difference between the light are and then a photogrammetric DM
07:43
and so what you're seeing here is I think errors of the block adjustments and You see that in green down here that their accuracy is actually maybe about a meter or 50 centimeters So just some qualities of the data sets still phenomenal Once we get our ground out. We also have now then a non ground point cloud, and we're like okay
08:04
What methods can we use to begin segmenting this I've saw some great talks yesterday using machine learning We also tried, but we used cloud compare Because they have canoe po which is using support vector machines but before I did that or we looked at the dome and
08:24
Covariance features and We extracted all these in Python just using scipy simple clustering unsupervised and We can get out vegetation And just to show you a bit of an example there here is a colored point cloud
08:41
You have buildings trees roads and some other things and then when you pull out the ground you get everything else And then you throw it through a simple k-means it goes really fast in Python, and you get man-made objects I'm not going to go so much more in a detail here because that's a kind of ideal situation of course you have other
09:01
things power lines and stuff like this But still this type of stepwise approach down to classification segmentation is is what we want to do so we tried out also Canupa, and I'm going to go into some of the details there in a second because I had major problems scaling it But here's just a vegetative area and we wanted to calculate a vegetation density
09:21
But a vegetation density is very dependent on what kind of coverage you got what kind of laser, so how are you going to? compare So I normalize it and so When we count the number of points within pixels and normalize it by the ground coverage or an action off the ground covers a total
09:40
Coverage we can get like a density map, and you'll see like here. It's a little bit Darker here, it's more dense, and I think That this is actually related to topography as well steeper slope here So on these types of things you're not getting actually it's not really a vegetation density It's actually something that needs to be corrected
10:02
But this is what we're working towards Just to discuss a little bit We had major issues scaling when you're trying to run this on 13,000 project or 1300 projects which probably consists of about 20 I don't know 50,000 files I Have big server lots of RAM still took six months to process at least
10:26
so I Also with the point cloud classification unsupervised was really cool and fast, but we wanted something supervised but we couldn't scale up what was built in cloud compare and I don't know I started I've been seeing some great talks your talk yesterday Brad
10:42
And I'm wondering if I should I'm a little bit inexperienced in this so I wonder if I should be using Postgres Postgres and twine And also Developing routines I think for GPU processing tensorflow pytorch, and I think that would maybe speed up the issue any solutions available I saw some great talks yesterday, so hopefully soon
11:02
So just to show some case studies of this data, and what it's being used for so in 2020 right before to 2021 there was a quick clay slide, and this is not far from where we work in In Oslo, but an area with a whole bunch of houses suddenly gave way
11:26
And you see this car almost drove right into it The authorities have done quite a few scans, and so this is just to show you some of the You'll notice right up here. This is an ortho photo this is then a hill shade from the year before and
11:44
then after And so The data is quite accurate, so you're getting quite an amazing precision of detail in this I'm going to zoom in in a second, but what you mainly see is that the mass here Slid down and filled up inside these small valleys
12:00
You can then difference these And you get two profiles. This is a cross profile here, and you see about a 20 meter difference and a Longitudinal profile where you see actually the the mass flowing down This was a quite a tragic event. I can show you what the Mapping authorities put together because this is pretty amazing when they showed
12:23
this because you're actually seeing the color of the houses that fell down And so they were using this to try and locate people another example this is quite a funny event they call a piece of rock up here the man and
12:41
For a period of about three to four months we had slow TV in Norway Where they had a camera focused up at this rock because they were afraid it was going to move down, and it never happened It kept going and then what they use Lidar and found displacements this is actually going downhill this way and they found that the rock was separating here moving up here this is from Lidar data and
13:05
Eventually it actually did come down not as big as they wanted it to but then Lena Christensen and them could do a nice study and measure the change in the amount of rock coming down so for landslide purposes for monitoring from measuring these these data sets are absolutely crucial and
13:22
Then the last example I want to show flood risk and flood mapping Was a 2018 in October there was a lot of snow in the mountains And it got very warm very fast and the rivers weren't ready for this and this swamped an entire village shall I say and a
13:43
Master student took the Lidar data and started using Hecaros which is a hydraulic model for modeling the flung and he was able to reproduce quite well the flood extent so for for Preparing for these types of events this data is absolutely you couldn't you couldn't run this model without this Lidar data
14:06
And then just to summarize High resolution Lidar data is available over in the entire country of Norway It's open Download it use it try to train some of its classified zone. It's not so well classified The quality it's variable just like if anybody saw Howard stock the early data is very low
14:25
Quality but the later data sometimes you're getting up to 25 meters per 25 points per square meter So you can find some treasures in here. It's multiple use cases change detection vegetation modeling and monitoring risk mapping
14:42
The open source pipe Processing is amazing. I'm just having trouble scaling. How can we process all this data in a consistent matter? If anybody has any suggestions let me know and that's it. Thank you